Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant's claim for foreign priority based on an application filed in Taiwan on 11/21/2019. It is noted, however, that applicant has not filed a certified copy of the TW 108142298 application as required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 08/28/2025 has been considered by the examiner.
Specification
The incorporation of essential material in the specification by reference to an unpublished U.S. application, foreign application or patent, or to a publication is improper. Applicant is required to amend the disclosure to include the material incorporated by reference, if the material is relied upon to overcome any objection, rejection, or other requirement imposed by the Office. The amendment must be accompanied by a statement executed by the applicant, or a practitioner representing the applicant, stating that the material being inserted is the material previously incorporated by reference and that the amendment contains no new matter. 37 CFR 1.57(g).
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “analysis module” and “YOLOR module” in claim 14.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof. The modules are programs stored in a hard disk or a memory of a computer (see para. 0023-0024).
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 14-18 are rejected under 35 U.S.C. 103 as being unpatentable over B. Schmauch et al, “Diagnosis of focal liver lesions from ultrasound using deep learning”, Diagnostic and Interventional Imaging, vol. 100, pp. 227-233, Feb. 2019 in view of Song et al. (CN 109886285 A, June 14, 2019) and Lee et al. (US 20190325275 A1, published October 24, 2019), hereinafter referred to as Schmauch, Song, and Lee, respectively.
Regarding claim 14, Schmauch teaches a method of analyzing a liver tumor comprising:
ultrasonically scanning an area of a liver of an examinee from an external position and absent added contrast agent to obtain an ultrasonic image of a target liver tumor of said examinee (see Abstract – “The purpose of this study was to create an algorithm that simultaneously detects and characterizes (benign vs. malignant) focal liver lesion (FLL) using deep learning…The dataset was composed of 367 two-dimensional ultrasound images from 367 individual livers, captured at various institutions…The algorithm was then tested on a new data set from 177 patients.” Where scanning and obtaining an ultrasound image of a liver tumor is inherent and known in the art, and reference Schmauch does not mention adding contrast agent before obtaining image);
obtaining, by an analysis module, a plurality of existing ultrasonic reference images of benign and of malignant liver tumors, wherein each existing reference image includes a clinician's marker of tumor pixel areas and liver tumor category for the reference image (see Fig. 2 – “Ultrasound images of focal liver lesions from the training set. Bounding boxes annotations are superimposed on the original image for an angioma (a) and multiple metastasis (b). Those annotations were generated a radiologist using dedicated tool.” Marked reference images as annotated images from the training set);
defining a result of a liver tumor category for one of the existing ultrasonic reference images based on shading and shadowing areas of the existing ultrasonic reference image and marking a plurality of tumor pixel areas in the existing ultrasonic reference image automatically with the analysis module according to a one or more parameters derived from empirical data locating and marking a plurality of tumor image pixel areas in said ultrasonic reference liver tumor image, wherein the liver tumor categories include benign liver tumors and malignant liver tumors (see Fig. 2 – “Ultrasound images of focal liver lesions from the training set [reference images]. Bounding boxes annotations [marked tumor pixel areas] are superimposed on the original image for an angioma [benign] (a) and multiple metastasis [malignant] (b). Those annotations were generated a radiologist using dedicated tool." Defining result of reference images as annotated images from the training set);
training a categorizer model by comparing the automatically defined result of liver tumor category and marked tumor pixel areas in said ultrasonic reference image with the clinician's liver tumor category and marker of tumor pixel areas for the reference image with coordination of a learning algorithm and adjusting the one or more parameters of the analysis module based on the comparison; iterating the defining and training for a next ultrasonic reference image (see Abstract – “The purpose of this study was to create an algorithm that simultaneously detects and characterizes (benign vs. malignant) focal liver lesion (FLL) using deep learning.”; see Fig. 2 – “Ultrasound images of focal liver lesions from the training set. Bounding boxes annotations are superimposed on the original image for an angioma (a) and multiple metastasis (b). Those annotations were generated a radiologist using dedicated tool.”; see pg. 230, col. 1, para. 3 – “We thus performed three-fold cross validation, which we repeated for three different splits of the data, to account for this variability. We repeated nine experiences during which we selected randomly 245 images (out of the 367 images of the training set) to train our neural network [training categorizer model], and estimated its performance by computing an AUC [parameter of analysis module] over the 122 images left.”); and
automatically analyzing said ultrasonic image of said target liver tumor of said examinee with said categorizer model and providing, in real time, a liver tumor category and a risk probability of malignance of said target liver tumor (see Fig. 3 – “Architecture of the neural network [categorizer model]… Finally, the 2048 features are fitted to a logistic regression that outputs a score ranging from 0 to 1 [risk probability] for each category of breast [liver] lesion [liver tumor category]. This score can be interpreted as the probability of presence of such lesion in the image.” where neural networks inherently perform in real time and automatically).
Schmauch teaches marking a plurality of tumor pixel areas in the existing ultrasonic reference image with an analysis module, but does not explicitly teach where the analysis module is a you-only-learn-one-representation (YOLOR) module.
Whereas, Song, in an analogous field of endeavor, teaches marking a plurality of tumor pixel areas in the existing ultrasonic reference image automatically with a you-only-learn-one-representation (YOLOR) module (see Abstract – “The invention claims a YOLO convolutional neural network of gall stone disease rapidly CT medical image identification method…”; see pg. 3, para. 5 and 11 – “Further, the image the gall stone disease CT medical image training set for analysis, comprising the following steps in any one or more of the following:… 6) step for lesion area marking the image of the training set [reference images].”).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified marking a plurality of tumor pixel areas in the existing ultrasonic reference image with an analysis module, as disclosed in Schmauch, by having the analysis module as a you-only-learn-one-representation (YOLOR) module, as disclosed in Song. One of ordinary skill in the art would have been motivated to make this modification in order to avoid the present in-depth study of medical image data set redundancy problem and realize quick identification of medical images, quick identification speed, as taught in Song (see Abstract).
Schmauch in view of Song teaches iterating the defining and training for a next ultrasonic reference image, but does not explicitly teach calculating a mean Average Precision (mAP) score until the mAP score meets or exceeds a mAP threshold.
Whereas, Lee, in an analogous field of endeavor, teaches calculating a mean Average Precision (mAP) score until the mAP score meets or exceeds a mAP threshold (Fig. 7; see para. 0057 – “After the base performance of the localization model (more specifically, the attention module hτi) is determined, samples in sample set Uτi can be individually selected in each iteration j…The mean Average Precision (mAP) achieved by attention module hτij on test dataset Dtest is determined and compared with the base performance value ατi to determine the performance improvement δτij of attention module hτij over attention module hτi. Again, true positives can be assigned to proposed temporal segments that have a temporal Intersection over Union (tIoU) with ground-truth instances greater than, for example, 0.7 [mAP threshold].”; see para. 0061 – “At block 740, the performance (e.g., mAP value) of the retrained model on the set of test samples is determined. At block 750, the performance improvement of the retrained model over the initial model is determined based on the performance of the initial model and the performance of the retrained model.”).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified iterating the defining and training for a next ultrasonic reference image, as disclosed in Schmauch in view of Song, by also calculating a mean Average Precision (mAP) score until the mAP score meets or exceeds a mAP threshold, as disclosed in Lee. One of ordinary skill in the art would have been motivated to make this modification in order to select one or more unlabeled videos to be annotated for training from a set of unlabeled videos such that the performance of the temporal action localization model retrained using the selected one or more unlabeled videos (after annotation) can improve the most, and in this way, the number of labeled training videos used to train an accurate and robust temporal action localization model can be reduced or minimized, as taught in Lee (see para. 0019).
Furthermore, regarding claim 15, Schmauch further teaches wherein the liver tumor category and risk probability of malignance of said target liver tumor, as determined and predicted by said analysis module respectively, are directly displayed on a screen or outputted via a built-in communication interface to an electronic device for remote display thereon (Figs. 1-2 and 4, ultrasound images of liver tumor; see Fig. 3 – “Finally, the 2048 features are fitted to a logistic regression that outputs a score ranging from 0 to 1 for each category of breast [liver] lesion. This score can be interpreted as the probability of presence of such lesion in the image.” probability scores of types of lesion inherently output to a display for interpretation; see Abstract – “The purpose of this study was to create an algorithm that simultaneously detects and characterizes (benign vs. malignant) focal liver lesion (FLL) using deep learning…The dataset was composed of 367 two-dimensional ultrasound images from 367 individual livers, captured at various institutions…The algorithm was then tested on a new data set from 177 patients.” Where displaying information is inherent and known in the art).
Furthermore, regarding claim 16, Schmauch further teaches wherein an area under a liver tumor differentiation curve of said analysis module reaches 0.9 (Table 3, Area under the ROC curve (AUC) scores by lesion type, where HCC (hepatocellular carcinoma), Cyst, HNF (focal nodular hyperplasia), and Average reaches (at or above) 0.9).
Furthermore, regarding claim 17, Lee further teaches wherein said analysis module attains a mAP score of 0.628 for tumors at least 5 cm in size (Fig. 7; see para. 0057 – “After the base performance of the localization model (more specifically, the attention module hτi) is determined, samples in sample set Uτi can be individually selected in each iteration j…The mean Average Precision (mAP) achieved by attention module hτij on test dataset Dtest is determined and compared with the base performance value ατi to determine the performance improvement δτij of attention module hτij over attention module hτi. Again, true positives can be assigned to proposed temporal segments that have a temporal Intersection over Union (tIoU) with ground-truth instances greater than, for example, 0.7 [mAP threshold].”; see para. 0061 – “At block 740, the performance (e.g., mAP value) of the retrained model on the set of test samples is determined. At block 750, the performance improvement of the retrained model over the initial model is determined based on the performance of the initial model and the performance of the retrained model.”).
Furthermore, regarding claim 18, Schmauch further teaches wherein said analysis module automatically analyzes the ultrasonic image of the target liver tumor of the examinee and provides the liver tumor category and risk probability of malignance of the target liver tumor within a time period of 10 frame delays±20% (see Fig. 3 – “Architecture of the neural network [categorizer model]… Finally, the 2048 features are fitted to a logistic regression that outputs a score ranging from 0 to 1 [risk probability] for each category of breast [liver] lesion [liver tumor category]. This score can be interpreted as the probability of presence of such lesion in the image.” where “frame delays ± 20%” is equated to “real-time”, and neural networks inherently perform in real time and automatically).
The motivation for claim 17 was shown previously in claim 14.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Y. Wang et al, “Multi-class Object Detection Algorithm Based on Convolutional Neural Network”, 2019 IEEE International Conference of Intelligent Applied Systems on Engineering (IEEE ICIASE 2019), pp. 286-289, April 2019 discloses in order to improve the accurate recognition rate and localization rate of multi-class object detection, a new network structure, Res-YOLO-R., based on the combination of Residual Network (ResNet) and You Only Look Once (YOLO) detection network, is proposed.
J. Virmani et al, “SVM-Based Characterization of Liver Ultrasound Images Using Wavelet Packet Texture Descriptors”, Journal of Digital Imaging, vol. 26, pp. 530-543, Oct. 2012 discloses feature selection with GA (genetic algorithm) as search procedure and classification accuracy of the SVM classifier as a fitness function is used to remove noise, non-informative and redundant features.
D. Mittal et al, “Neural network based focal liver lesion diagnosis using ultrasound images”, Computerized Medical Imaging and Graphics, vol. 35, pp. 315-323, 2011 discloses training set, consisting of 250 SROIs having 50 SROIs (segmented ROIs) of each five classes of lesions, is used to select the sets of parameters for NN (neural network) architecture and NN weights. A set of 50 SROIs having 10 SROIs from each liver image class was used for the validation of the trained NN and to identify the NN architecture with the best performance. Once the architecture is selected using the validation set, the parameters of the detection program are fixed. The performance of the program is then evaluated with an independent test set.
H. Zhou et al, “CAD: Scale Invariant Framework for Real-Time Object Detection”, 2017 IEEE International Conference on Computer Vision Workshops, pp. 760-768, 2017 discloses in YOLO, input images are divided into several square grids and delivered to a convolution network. Each grid is required to detect the objects that their center points fall into its grid region. Convolution network outputs a cube that indicates the predictions of these grids.
Z. Zhang et al, “Ultrasonic Diagnosis of Breast Nodules Using Modified Faster R-CNN”, Ultrasonic Imaging, vol. 41, no. 6, pp. 353-367, Oct. 2019 discloses for multi-target detection using YOLO, mean average precision (mAP) is usually used to measure its detection precision. In the multi-target detection, curves can be drawn for each category according to recall and precision, then AP is the area under the corresponding curve, and mAP is the average AP of multiple targets. The mAP is between 0 and 1, and higher value indicates higher precision.
K. Xia et al, “Liver Detection Algorithm Based on an Improved Deep Network Combined With Edge Perception”, IEEE Access, vol. 7, pp. 175135-175142, Oct. 2019 discloses qualitatively and quantitatively analyze the accuracy of liver detection in images using YOLO.
Y. Wei et al, “A Hybrid Multi-atrous and Multi-scale Network for Liver Lesion Detection”, MLMI 2019, pp. 365-372, Oct. 2019 discloses defining liver lesion detection task as finding a 3D cubic bounding box for each liver lesion.
M. Al-masni et al, “Simultaneous detection and classification of breast masses in digital mammograms via a deep learning YOLO-based CAD system”, Computer Methods and Programs in Biomedicine, vol. 157, pp. 85-94, Aug. 2017 discloses preprocessing of mammograms, feature extraction utilizing deep convolutional networks, mass detection with confidence, and finally mass classification using Fully Connected Neural Networks (FC-NNs).
Tran (US 20200405148 A1, published December 31, 2020 with a priority date of June 27, 2019) discloses YOLO is a real time system built on deep learning for solving image detection problems.
Kushida et al. (US 20210224997 A1, published July 22, 2021 with a priority date of October 8, 2019) discloses a machine learning model (i.e., YOLO) for performing object recognition in region units.
Kreiger et al. (US 20200194117 A1, published June 18, 2020 with a priority date of December 13, 2018) the Faster R-CNN model was trained, and the Mean Average Precision (“mAP”) was calculated on the test data at an Intersection Over Union (“IOU”) of 0.5.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Nyrobi Celestine whose telephone number is 571-272-0129. The examiner can normally be reached on Monday - Thursday, 7:00AM - 5:00PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pascal Bui-Pho can be reached on 571-272-2714. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.C./Examiner, Art Unit 3798
Prosecution Insights