Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The incorporation of essential material in the specification by reference to an unpublished U.S. application, foreign application or patent, or to a publication is improper. Applicant is required to amend the disclosure to include the material incorporated by reference, if the material is relied upon to overcome any objection, rejection, or other requirement imposed by the Office. The amendment must be accompanied by a statement executed by the applicant, or a practitioner representing the applicant, stating that the material being inserted is the material previously incorporated by reference and that the amendment contains no new matter. 37 CFR 1.57(g).
Response to Amendment
Claims 14-18 remain pending in the application in response to the applicant’s amendments to the rejections previously set forth in the Non-Final Office Action mailed 06/30/2026.
Response to Arguments
Applicant’s arguments filed 08/17/2026 with respect to claim(s) 14 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Given the amendments to claim 14, reference to Xie is being relied upon to teach dependent claim 18 more-consistently with the instant claim language, as shown below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 14-18 are rejected under 35 U.S.C. 103 as being unpatentable over B. Schmauch et al, "Diagnosis of focal liver lesions from ultrasound using deep learning", Diagnostic and Interventional Imaging, vol. 100, pp. 227-233, Feb. 2019 in view of Song et al. (CN 109886285 A, June 14, 2019), Lee et al. (US 20190325275 A1, published October 24, 2019), and Xie et al. (US 20210177373 A1, published June 17, 2021 with a priority date of July 30, 2018), hereinafter referred to as Schmauch, Song, Lee, and Xie, respectively.
Regarding claim 14, Schmauch teaches a method of analyzing a liver tumor comprising:
obtaining, by an analysis module, a plurality of existing ultrasonic reference images of benign and of malignant liver tumors, wherein each existing reference image includes a clinician's marker of tumor pixel areas and liver tumor category for the reference image (see Fig. 2 "Ultrasound images of focal liver lesions from the training set. Bounding boxes annotations are superimposed on the original image for an angioma (a) and multiple metastasis(b). Those annotations were generated a radiologist using dedicated tool." Marked reference images as annotated images from the training set);
defining a result of a liver tumor category for one of the existing ultrasonic reference images based on shading and shadowing areas of the existing ultrasonic reference image and marking a plurality of tumor pixel areas in the existing ultrasonic reference image automatically with the analysis module according to one or more parameters derived from empirical data locating and marking a plurality of tumor image pixel areas in the ultrasonic reference liver tumor image, wherein the liver tumor categories include benign liver tumors and malignant liver tumors (see Fig. 2 "Ultrasound images of focal liver lesions from the training set [reference images]. Bounding boxes annotations [marked tumor pixel areas] are superimposed on the original image for an angioma [benign] (a) and multiple metastasis [malignant] (b). Those annotations were generated a radiologist using dedicated tool." Defining result of reference images as annotated images from the training set);
training a categorizer model by comparing the automatically defined result of liver tumor category and marked tumor pixel areas in the ultrasonic reference image with the clinician's liver tumor category and marker of tumor pixel areas for the ultrasonic reference image with coordination of a learning algorithm and adjusting the one or more parameters of the analysis module based on the comparison; iterating the defining and training for a next ultrasonic reference image (see Abstract "The purpose of this study was to create an algorithm that simultaneously detects and characterizes (benign vs. malignant) focal liver lesion (FLL) using deep learning."; see Fig. 2 "Ultrasound images of focal liver lesions from the training set. Bounding boxes annotations are superimposed on the original image for an angioma (a) and multiple metastasis(b). Those annotations were generated a radiologist using dedicated tool."; see pg. 230, col. 1, para. 3 "We thus performed three-fold cross validation, which we repeated for three different splits of the data, to account for this variability. We repeated nine experiences during which we selected randomly 245 images (out of the 367 images of the training set) to train our neural network [training categorizer model], and estimated its performance by computing an AUC [parameter of analysis module] over the 122 images left.");
ultrasonically scanning an area of a liver of an examinee from an external position and absent added contrast agent to obtain an ultrasonic image of a target liver tumor of the examinee (see Abstract "The purpose of this study was to create an algorithm that simultaneously detects and characterizes (benign VS. malignant) focal liver lesion (FLL) using deep learning. The dataset was composed of 367 two dimensional ultrasound images from 367 individual livers, captured at various institutions. The algorithm was then tested on a new data set from 177 patients." Where scanning and obtaining an ultrasound image of a liver tumor is inherent and known in the art, and reference Schmauch does not mention adding contrast agent before obtaining image); and
automatically analyzing the ultrasonic image of the target liver tumor of the examinee with the categorizer model and providing a liver tumor category and a risk probability of malignance of said target liver tumor (see Fig. 3 "Architecture of the neural network [categorizer model]... Finally, the 2048 features are fitted to a logistic regression that outputs a score ranging from 0 to 1 [risk probability] for each category of breast [liver] lesion [liver tumor category]. This score can be interpreted as the probability of presence of such lesion in the image." where neural networks inherently perform automatically).
Schmauch teaches marking a plurality of tumor pixel areas in the existing ultrasonic reference image with an analysis module, but does not explicitly teach where the analysis module is a you-only learn-one-representation (YOLOR) module.
Whereas, Song, in an analogous field of endeavor, teaches marking a plurality of tumor pixel areas in the existing ultrasonic reference image automatically with a you-only-learn-one-representation (YOLOR) module (see Abstract "The invention claims a YOLO convolutional neural network of gall stone disease rapidly CT medical image identification method...";see pg. 3, para. 5 and 11 "Further, the image the gall stone disease CT medical image training set for analysis, comprising the following steps in any one or more of the following:.. 6) step for lesion area marking the image of the training set [reference images].").
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified marking a plurality of tumor pixel areas in the existing ultrasonic reference image with an analysis module, as disclosed in Schmauch, by having the analysis module as a you-only-learn-one-representation (YOLOR) module, as disclosed in Song. One of ordinary skill in the art would have been motivated to make this modification in order to avoid the present in-depth study of medical image data set redundancy problem and realize quick identification of medical images, quick identification speed, as taught in Song (see Abstract).
Schmauch in view of Song teaches iterating the defining and training for a next ultrasonic reference image, but does not explicitly teach calculating a mean Average Precision (mAP) score until the mAP score meets or exceeds a mAP threshold.
Whereas, Lee, in an analogous field of endeavor, teaches calculating a mean Average Precision (mAP) score until the mAP score meets or exceeds a mAP threshold (Fig. 7; see para. 0057 "After the base performance of the localization model (more specifically, the attention module hτi) is determined, samples in sample set Uτi can be individually selected in each iteration j…The mean Average Precision (mAP) achieved by attention module hτij test dataset Dtest is determined and compared with the base performance value ατi to determine the performance improvement στi of attention module over attention module hτi . Again, true positives can be assigned to proposed temporal segments that have a temporal Intersection over Union (tloU) with ground-truth instances greater than, for example, 0.7 [mAP threshold]."; see para. 0061 "At block 740, the performance (e.g., mAP value) of the retrained model on the set of test samples is determined. At block750, the performance improvement of the retrained model over the initial model is determined based on the performance of the initial model and the performance of the retrained model.").
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified iterating the defining and training for a next ultrasonic reference image, as disclosed in Schmauch in view of Song, by also calculating a mean Average Precision (mAP) score until the mAP score meets or exceeds a mAP threshold, as disclosed in Lee. One of ordinary skill in the art would have been motivated to make this modification in order to select one or more unlabeled videos to be annotated for training from a set of unlabeled videos such that the performance of the temporal action localization model retrained using the selected one or more unlabeled videos (after annotation) can improve the most, and in this way, the number of labeled training videos used to train an accurate and robust temporal action localization model can be reduced or minimized, as taught in Lee (see para. 0019).
Schmauch in view of Song and Lee teaches automatically analyzing the ultrasonic image of the target liver tumor of the examinee with the categorizer model and providing a liver tumor category and a risk probability of malignance of said target, but does not explicitly teach automatically analyzing and providing determinations of the ultrasound liver image in real-time during ultrasound scanning.
Whereas, Xie, in an analogous field of endeavor, teaches automatically analyzing and providing determinations of the ultrasound liver image in real-time during ultrasound scanning (see para. 0028 – “FIG. 1 shows a flow diagram 100 of an example process for H/R quantification in accordance with the principles of the present disclosure. The H/R quantification processes may be performed in real-time [automatic determinations of the ultrasound liver image], e.g., while the sonographer is acquiring the ultrasound image data [during ultrasound scanning].”).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified automatically analyzing the ultrasound liver image to provide a liver tumor category and a risk probability of malignance, as disclosed in Schmauch in view of Song and Lee, by automatically analyzing and providing determinations of the ultrasound liver image in real-time during ultrasound scanning, as disclosed in Xie. One of ordinary skill in the art would have been motivated to make this modification in order to receive real-time information about the liver currently being imaged.
Furthermore, regarding claim 15, Schmauch further teaches wherein the liver tumor category and risk probability of malignance of target liver tumor, as determined and predicted by the analysis module respectively, are directly displayed on a screen or outputted via a built-in communication interface to an electronic device for remote display thereon (Figs. 1-2 and 4, ultrasound images of liver tumor; see Fig. 3 "Finally, the 2048 features are fitted to a logistic regression that outputs a score ranging from 0 to 1 for each category of breast [liver] lesion. This score can be interpreted as the probability of presence of such lesion in the image." probability scores of types of lesion inherently output to a display for interpretation; see Abstract "The purpose of this study was to create an algorithm that simultaneously detects and characterizes (benign vs. malignant) focal liver lesion (FLL) using deep learning. The dataset was composed of 367 two-dimensional ultrasound images from 367 individual livers, captured at various institutions The algorithm was then tested on a new data set from 177 patients." Where displaying information is inherent and known in the art).
Furthermore, regarding claim 16, Schmauch further teaches wherein an area under a liver tumor differentiation curve of the analysis module reaches 0.9 (Table 3, Area under the ROC curve (AUC) scores by lesion type, where HCC (hepatocellular carcinoma), Cyst, HNF (focal nodular hyperplasia), and Average reaches (at or above) 0.9).
Furthermore, regarding claim 17, Lee further teaches wherein the analysis module attains a mAP score of 0.628 for tumors at least 5 cm in size (Fig. 7; see para. 0057 "After the base performance of the localization model (more specifically, the attention module hτi) is determined, samples in sample set Uτi can be individually selected in each iteration j…The mean Average Precision (mAP) achieved by attention module hτij test dataset Dtest is determined and compared with the base performance value ατi to determine the performance improvement στi of attention module over attention module hτi . Again, true positives can be assigned to proposed temporal segments that have a temporal Intersection over Union (tloU) with ground-truth instances greater than, for example, 0.7 [mAP threshold]."; see para. 0061 "At block 740, the performance (e.g., mAP value) of the retrained model on the set of test samples is determined. At block750, the performance improvement of the retrained model over the initial model is determined based on the performance of the initial model and the performance of the retrained model.").
Furthermore, regarding claim 18, Schmauch further teaches wherein the analysis module automatically analyzes the ultrasonic image of the target liver tumor of the examinee (see Fig. 3 "Architecture of the neural network [categorizer model]... Finally, the 2048 features are fitted to a logistic regression that outputs a score ranging from 0 to 1 [risk probability] for each category of breast [liver] lesion [liver tumor category]. This score can be interpreted as the probability of presence of such lesion in the image.") and
Xie further teaches providing determinations of the ultrasound liver image within a time period of 10 frame delays ± 20% (real-time; see para. 0028 – “FIG. 1 shows a flow diagram 100 of an example process for H/R quantification in accordance with the principles of the present disclosure. The H/R quantification processes may be performed in real-time [automatic determinations of the ultrasound liver image], e.g., while the sonographer is acquiring the ultrasound image data [during ultrasound scanning].”).
The motivation for claims 17-18 was shown previously in claim 14.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Luo et al. (US 20060120608 A1, published June 8, 2006) discloses determining a candidate lesion region within an ultrasound image in real time to provide the operator with instant warning or feedback.
Huepf et al. (US 20200113544 A1, published April 16, 2020 with a priority date of October 15, 2018) discloses acquired ultrasound scan data may be processed in real-time during a scanning session as the echo signals are received.
Ishrak et al. (US 20200397511 A1, published December 24, 2020 with a priority date of June 18, 2019) discloses using machine learning techniques, the system may identify features in the live image to quickly identify particular spatial orientations of the live images, and retrieve reference images corresponding to those spatial orientations.
Perrey et al. (US 20180125460 A1, published May 10, 2018) discloses acquired ultrasound data may be processed in real-time by the controller circuit during a scanning or therapy session as the echo signals are received.
Sudhakar et al. (US 20190200964 A1, published July 4, 2019) discloses acquired ultrasound scan data may be processed in real-time during a scanning session as the echo signals are received.
Gerard et al. (US 20180322627 A1, published November 8, 2018) discloses acquired ultrasound data may be processed in real-time by the controller circuit during a scanning or therapy session of the ultrasound exam as the echo signals are received.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Nyrobi Celestine whose telephone number is 571-272-0129. The examiner can normally be reached on Monday - Thursday, 7:00AM - 5:00PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pascal Bui-Pho can be reached on 571-272-2714. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.C./Examiner, Art Unit 3798
/PASCAL M BUI PHO/Supervisory Patent Examiner, Art Unit 3798
