DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed on 04/01/2026 has been entered. Claims 1-12,14-15 and 17-18 have been amended. Claims 1-18 remain pending.
The previously raised objections for Claims 1, 4, 8, 9 and 15 are withdrawn because the issues have been properly corrected.
The previously raised rejections under 35 U.S.C. 112(b) for Claims 3-4, 7-13 and 15 are withdrawn because the issues have been properly corrected.
The previously raised rejections under 35 U.S.C. 101 for Claims 1-18 are withdrawn because the issues have been properly corrected.
Response to Arguments
On Pages 11-12 of Remarks, Applicant argues that, regarding the amended Claims 1 and 15-18, reference Abraham does not disclose (a) “a radiograph cut-out model that is a trained cut-out model specifically designed to locate the crop the region of interest before analysis is carried out”, also does not disclose (b) “a refinement model architecture” as claimed, does not disclose (c) “a structured data augmentation pipeline” as claimed, and does not disclose (d) a method that “includes steps of combining and error reporting that include …, and cephalometric tracing and analysis”. These arguments are moot in view of the new grounds of rejection which relies on the combination of references Abraham, Wozniak and Tuzoff to disclose these limitations in the claims.
Claim Objections
Claim 1 is objected to because of the following informalities:
Claim 1, Lines 1-2, recites “data from of a digital radiographic image”, which should be changed to “data [[from]] of a digital radiographic image”.
Claim 1, Lines 18-19, recites “from a learning radiographic image”, which should be changed to “based on [[from]] a learning radiographic image” or “using [[from]] a learning radiographic image”.
Claim 1, Line 32, recites “Principal Component Analysis (PCN)”, which should be changed to “Principal Component Analysis [[(PCN)]] (PCA)”.
Claim 1, Line 35, recites “the group”, which should be changed to “[[the]] a group”.
Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
“a receiver … to receive …” in Claim 1, Lines 7-8.
A review of the Specification does not identify any detail for providing structural support for the claimed “receiver”.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-18 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 1, Lines 9-10, recites “determine whether the image meets a predetermined criterion of image quality”. Specification does not provide any detail on how the determining is performed or what the predetermined criterion is.
Claim 1, Lines 55 and 60, recite “prediction confidences” and “based on prediction-confidence thresholds” respectively. Specification does not provide any detail on determining or using the prediction confidences or prediction-confidence thresholds.
Claim 2, Lines 2-5, recites “performing … a radiograph cutout model learning procedure …, prior to the general model learning procedure”. Specification, for example Fig. 1, presents the radiograph cutout model learning procedure and the general model learning procedure as independent procedures, and does not specify which one of the two is performed prior to the other.
Claims 3-18 are also rejected under 35 U.S.C. 112(a) because they inherit the deficiencies of the claim(s) they respectively depend upon.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-18 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1-3, 5-6, 8-9, 12 and 14 recite more than one images in different steps of the claimed method, such as “digital radiographic image (R)” (Claims 1 and 5-6), “learning radiographic image (R)” (Claims 1 and 9), “digital analysis radiographic image (R’)” (Claim 1), “cropped analysis radiographic image (R’’)” (Claims 1 and 3), “original analysis radiographic image (R’)” (Claim 1), “analysis radiographic image (R’)” (Claims 2-3 and 14), “radiographic images (R)” (Claim 8), and “new image” (Claim 12). Naming of the images, and their denoting (by R, R’ or R’’), are inconsistent and confusing. For present purposes of examination, the image to be analyzed (or “providing cephalographic data” as recited in Claim 1, Lines 1-2) is consistently interpreted as “analysis radiographic image”, the image used for learning models as “learning radiographic image”, and the image cropped to contain only skull-relevant region as “cropped radiographic image”. Furthermore, the plurality of image cutouts (i.e. input for the refinement model) are interpreted as “cutouts”, to differentiate from the “cropped radiographic image” above.
Claims 1-5 and 9 recite multiple terms such as “cutting” (Claim 1), “cutouts” (Claims 1 and 9), “cutout model” (Claims 1-4), “crop” and “cropped” (Claims 1-2), “cutting out” and “cut out” (Claims 5). According to the claims and the specification, the terms may be grouped into 2 different types of processing: (a) cutting an image into a plurality of smaller images, wherein each smaller image comprises a respective group of anatomical points of interest and is analyzed by a different refinement model; (b) cropping an image to crop a skull-relevant region. Current usages of cutting- or cropping-related terminology is confusing. For example, Claim 1, Line 27 “image cutouts (R1, R2, … RN)” and Line 40 “radiograph cutout model” are seemingly related, but the former “cutouts” refer to a plurality of smaller images obtained by cutting, and the latter “cutout” is the more conventionally termed “image cropping”. For present purposes of examination, the recited terms that relate to the above group (a) are interpreted as “cut” or “cutout”, and those of the group (b) as “crop” or “cropping”.
Claim 2, Line 4 recites “crop a skull-relevant region”, but Claim 1, Lines 40-41 recites “crop a cephalometric-relevant region”. It is unclear whether the recited “skull-relevant” and “cephalometric relevant” refer to the same, or different things. For present purposes of examination, the recited “skull-relevant” and “cephalometric relevant” are interpreted to refer to the same.
Claim 3, Line 3 recites “the inference step based on …”. It is unclear which the recited “inference step” refers to, either “reference step” in Claim 3 Line 2, or Claim 1 Line 37, or Claim 1 Line 39, or Claim 1 Lines 39-40, or Claim 1 Line 45, or Claim 1 Line 53, or a reference different from all the listed reference steps. For present purposes of examination, the recited “reference step” is interpreted to refer to the “reference” in Claim 1 Lines 39-40.
Claim 1 Lines 7-8 recites “a receiver … to receive …”, which invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function. Specification does not contain any information on the structure of the claimed “receiver” in Claim 1. Therefore, the claim is indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
Applicant may:
(a) Amend the claim so that the claim limitation will no longer be interpreted as a limitation under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph;
(b) Amend the written description of the specification such that it expressly recites what structure, material, or acts perform the entire claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(c) Amend the written description of the specification such that it clearly links the structure, material, or acts disclosed therein to the function recited in the claim, without introducing any new matter (35 U.S.C. 132(a)).
If applicant is of the opinion that the written description of the specification already implicitly or inherently discloses the corresponding structure, material, or acts and clearly links them to the function so that one of ordinary skill in the art would recognize what structure, material, or acts perform the claimed function, applicant should clarify the record by either:
(a) Amending the written description of the specification such that it expressly recites the corresponding structure, material, or acts for performing the claimed function and clearly links or associates the structure, material, or acts to the claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(b) Stating on the record what the corresponding structure, material, or acts, which are implicitly or inherently set forth in the written description of the specification, perform the claimed function. For more information, see 37 CFR 1.75(d) and MPEP §§ 608.01(o) and 2181.
Claims 7, 10-11, 13 and 15-18 are also rejected under 35 U.S.C. 112(b) because they inherit the indefiniteness of the claim(s) they respectively depend upon.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 9-13 and 15-18 are rejected under 35 U.S.C. 103 as being unpatentable over Abraham et al (US 20180061054 A1; hereafter Abraham), in view of Wozniak et al (Information fusion 16(2014) 3-17; hereafter Wozniak) and Tuzoff et al (US 20200146646 A1; hereafter Tuzoff).
With regard to Claim 1, Abraham discloses a method of providing cephalographic data from of a digital radiographic image (R) (Abraham, Abstract; “A system and method are described for automating the analysis of cephalometric x-rays. Included in the analysis is a method for automatic anatomical landmark localization …”) using a radiographic system (4), wherein said radiographic system (4) comprises:
a display unit (43) (Abraham, Fig. 3 shows a computer with display monitor.); and
processing means (41), connected to said display unit (43) (Abraham, Fig. 3 shows the display unit (1208) being connected to a processing means (1200).),
said method comprising the steps of:
causing a receiver of the radiographic system to receive the digital radiographic image (R) comprising a skull region (Abraham, Para 0038; “…the cephalometric image or images required. The image or images are provided … over a cloud interface 220, to the server described below.”);
pre-processing the digital radiographic image (R) to determine whether the image meets a predetermined criterion of image quality (Abraham, Para 0041; “At step 230 The image quality is estimated and pre-processing success is determined. If the detected image quality does not meet a specified quality level or threshold …”);
performing, with said processing means (41), a learning step (1) (Abraham, Para 0044; “a CNN Training Module 1202 can be employed to train the CNN system using pre-existing rules and prior store image data.”) comprising:
executing (13), from a learning radiographic image (R) (Abraham, Para 0041; “At step 230 The image quality is estimated and pre-processing success is determined. If the detected image quality does not meet a specified quality level or threshold …”), a general model learning procedure for learning a general model configured to detect one or more anatomic points of interest (Abraham, Para 0046; “These stored images, and optionally the stored landmarks and analytic information associated therewith, are used to train the CNN system using the CNN Training Module discussed.”); and
performing a refinement model learning procedure (14) (Abraham, Para 0063; “In an attempt to maximize accuracy, the algorithm then attempts to fine-tune the points location as follows.”), comprising:
cutting (143) the digital radiographic image into a plurality of image cutouts (R1, R2, ..., RN), each comprising a respective group of anatomical points of interest (Abraham, Para 0064; “A complete image may be divided into separate regions, for example defined by groups or sub-groups of landmark points of interest …”); and
training (151, ..., 15N) a refinement model (2) for each image cutout (R1, R2, ..., RN) (Abraham, Para 0064; “… a separate CNN-based model for each group in steps 630”) (Abraham, Para 0067; “each refinement model is trained according to the training process of FIG. 5, with the exception that only the corresponding subset of points is used and input images are cropped to contain only the relevant part of the skull.”), wherein each refinement model comprises:
a feature-engineering stage using a Histogram of Oriented Gradients (HOG), Haar-like features, or Convolutional Neural Network (CNN)-based descriptors (Abraham, Para 0060; “The convolutional layers thus extract features and the dense layers use said features to perform a regression or classification thereon.” This disclosure discloses the feature engineering step in the disclosed CNN).
carrying out an inference step (3) on a digital analysis radiographic image (R’) (Abraham, Para 0063 “FIG. 6 illustrates an outline of a detection process 601 as carried out in Landmark Detection Module 1204 in an exemplary embodiment. … for example an x-ray image of a patient’s head and jaw.”), comprising:
performing (33), before performing an interference based on the general model, detecting and cropping a cephalometric-relevant region of the radiographic image, thereby generating a cropped analysis radiographic image (R’’) (Abraham, Para 0039-0040; “Other pre-processing steps may include skull detection, skull region extraction and others. Detection of the patient's skull, the skull's position and rotation can be determined using an automated process.”);
performing, on the cropped analysis radiographic image (R’’), an inference step based on the general model to obtain initial geometric coordinates of anatomical points of interest (Abraham, Para 0063; “In the first phase of the detection, at step 610, a CNN detection process is carried out on the whole cephalometric image of interest …”);
cutting (34) the cropped analysis radiographic image (R’) into a plurality of image cutouts (R’1, R’2, ..., R’N), wherein each image cutout (R’1, R’2, ..., R’N) corresponds to a respective group of anatomical points of interest (Abraham, Para 0064; “These 90 landmarks can be divided into 12 separate groups of landmark points, each in different area of the skull.”); and
performing (361 ...36N), on each cutout, an inference through said refinement model to obtain refined coordinates and associated predication confidences (Abraham, Para 0064; “The points in each group are refined with a separate CNN-based model for each group in steps 630.”; Para 0068; “During the refinement stage each refinement model may be used to obtain predictions for a corresponding group of point and to estimate the prediction confidence.”); and
combining (37) the anatomical points of interest obtained from the general model and from the refined models by aggregating and repositioning sets of the refined coordinates into the coordinate system of the original analysis radiographic image (R’) (Abraham, Para 0064; “Thereafter, all the points are mapped back to the original image at step 640.”), including determining missing anatomical points based on the prediction-confidence thresholds (Abraham, Para 0068; “If the point confidence is above or equal to a controllable threshold then …, otherwise the initial prediction is kept for that point”); and
displaying said final geometric coordinates of the anatomical points of interest (Abraham, Fig. 2 displays an example of whole image with the detected landmarks.).
Abraham does not clearly and explicitly disclose:
a dimensionality-reduction stage using Principal Component Analysis (PCN) or Partial Least Squares (PLS);
a multi-regressor stacking stage comprising at least two regressors selected from the group consisting of Support Vector Machines (SVM), Random Forest, Extra Trees, and Gradient Boosting, and a linear metamodel;
using a trained radiograph cutout model for detecting and cropping a region in an image; and
reporting missing anatomical objects.
Wozniak in the same field of endeavor discloses:
a dimensionality-reduction stage using Principal Component Analysis (PCN) or Partial Least Squares (PLS) (Wozniak, Page 12, Para 2; “The effect of PCA initial dimensionality reduction is also tested …”); and
a multi-regressor stacking stage comprising at least two regressors selected from the group consisting of Support Vector Machines (SVM), Random Forest, Extra Trees, and Gradient Boosting, and a linear metamodel (Wozniak, Fig. 3 shows a classifier system that contains 2 levels, where the first level contains multiple classifiers and the second level “fuser” combines outputs of the multiple classifiers. Page 9, final para; “Homogeneous MCS, such as Random Forest (RF), are composed of classifiers of the same kind. In the works revised below, basic classifiers are Multi-Layer Perceptron (MLP), k-Nearest Neighbor (kNN), Radial Basis Function (RBF), Support Vector Machines (SVM), Probabilistic Neural Networks (PNNs), and Maximum Likelihood (ML) classifiers.” Here “random forest” and “support vector machines” are disclosed to be basic classifiers. Page 8, Para 6; “Tresp and Taniguchi [148] proposed a linear function for this fuser model …”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham, as suggested by Wozniak, in order to use a structure that contains multiple classifiers. One of ordinary skill in the art would have been motivated to make the modification for the benefit of improved classification accuracy by combining advantage of multiple individual classifiers (Wozniak, Page 3, Para 1; “… there is not a single classifier modeling approach which is optimal for all pattern recognition tasks, since each has its own domain of competence. For a given classification task, we expect the MCS to exploit the strengths of the individual classifier models at our disposal to produce the high quality compound recognition system overcoming the performance of individual classifiers.”).
Abraham and Wozniak as discussed above do not clearly and explicitly disclose:
using a trained radiograph cutout model for detecting and cropping a region in an image; and
reporting missing anatomical objects.
Tuzoff in the same field of endeavor discloses:
using a trained radiograph cutout model for detecting and cropping a region in an image (Tuzoff, Para 0024; “the classification module initially crops the image based on the predicted bounding boxes to produce cropped images 400 …”; Para 0022; “A VGG-16 Net … can be used as a base CNN for both RPN and object detection.”); and
reporting missing anatomical objects (Tuzoff, Para 0069; “This report includes … a listing, by tooth number (American notation is employed in FIG. 19) of teeth that were determined to be … missing, together with a listing of the findings as confirmed by the practitioner 1124.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham and Wozniak, as suggested by Tuzoff, in order to use a trained model for cropping images and to report missing object. One of ordinary skill in the art would have been motivated to make the modification of using a trained model for the benefit of trained models such as VGG-16 Net demonstrating robust capability in object detection, and make the modification of reporting missing object for the benefit of notifying the practitioner of either the real status of a patient or the quality of images so as to better plan for following steps of treatment.
With regard to Claim 2, Abraham, Wozniak and Tuzoff disclose all the limitations of Claim 1 as discussed above, but do not clearly and explicitly disclose further comprising performing (12), with said processing means (41), a radiograph cutout model learning procedure (12) for learning a radiograph cutout model configured to detect and crop a skull-relevant region of the digital radiographic image (R), prior to the general model learning procedure.
Tuzoff further discloses further comprising performing (12), with said processing means (41), a radiograph cutout model learning procedure (12) for learning a radiograph cutout model configured to detect and crop a skull-relevant region of the digital radiographic image (R), prior to the general model learning procedure (Tuzoff, Para 0034 and 0036; “During training for teeth detection, model weights pretrained on the ImageNet dataset were used for the basic CNN … As with teeth detection, for teeth classification the model weights pretrained on the ImageNet dataset were used to initialize the CNN in the classification module.”. Here the disclosed “training for teeth detection” is for training a model that determines a bounding box, so corresponds to the claimed radiograph cutout model learning procedure. It is also note that in the description of the training process, the training for teeth detection is prior to the training for teeth classification). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham, Wozniak and Tuzoff, as further suggested by Tuzoff, in order to train a model for cropping a relevant region from an image. One of ordinary skill in the art would have been motivated to make the modification for the benefit of improving the prediction accuracy of the landmarks by learning features in training images (Tuzoff, Para 0020; “Both the RPN and object detector modules share the convolution layers 210 of the base CNN that provides a compact representation of the source image, known as a feature map 220. The features are learned during a training phase …”).
With regard to Claim 3, Abraham, Wozniak and Tuzoff disclose all the limitations of Claim 2 as discussed above, but do not clearly and explicitly disclose wherein carrying out said inference step (3) comprises a sub-step of performing (31), on said analysis radiographic image (R’), the inference step based on said radiograph cutout model learned in said radiograph cutout model learning procedure (12), so as to obtain the cropped analysis radiographic image.
Tuzoff further discloses wherein carrying out said inference step (3) comprises a sub-step of performing (31), on said analysis radiographic image (R’), the inference step based on said radiograph cutout model learned in said radiograph cutout model learning procedure (12), so as to obtain the cropped analysis radiographic image (Tuzoff, Para 0024; “… initially crops the image based on the predicted bounding boxes to produce cropped images 400, which are provided as input to convolutional layers 510.”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham, Wozniak and Tuzoff, as further suggested by Tuzoff, in order to perform image cropping. One of ordinary skill in the art would have been motivated to make the modification for the benefit of improving the prediction accuracy of the landmarks by focusing on the most relevant part of image.
With regard to Claim 4, Abraham, Wozniak and Tuzoff disclose all the limitations of Claim 3. Abraham further discloses performing (31) the inference step based on said radiograph cutout model before the inference step based on said general model learned in said general model learning procedure (13) (Abraham, Fig. 2 shows that the step of “skull region extraction” 225 is before the step of “automated landmark detection” 240).
With regard to Claim 9, Abraham, Wozniak and Tuzoff disclose all the limitations of Claim 1 as discussed above. Abraham further discloses wherein said step of training (151, ..., 15N) a refinement model for each image cutout comprises:
resizing (1511, ..., 15N1) each cutout of said radiographic image (Abraham, Para 0067; “each refinement model is trained according to the training process of FIG. 5, with the exception that only the corresponding subset of points is used and input images are cropped to contain only the relevant part of the skull.” Paras 0070, 0088, 0105, 0123 and 0190 disclose that the refinement models for the different sub-regions accept input image of different sizes.), and
carrying out a feature engineering and refinement model learning procedure (2) (Abraham, Para 0014 discloses a random-tree approach for feature extraction; “a method has been proposed for automatically annotating objects in radiographic images by using predictive modeling approaches based on decision trees”) (Abraham, Para 0060 discloses a deep learning approach; “The convolutional layers thus extract features and the dense layers use said features to perform a regression or classification thereon.” More details in Fig. 7).
With regard to Claim 10, Abraham, Wozniak and Tuzoff disclose all the limitations of Claim 9 as discussed above. Abraham further discloses wherein said feature engineering and refinement model learning procedure (2) is based on computer vision algorithms (Abraham, Para 0014 discloses a random-tree approach for feature extraction; “a method has been proposed for automatically annotating objects in radiographic images by using predictive modeling approaches based on decision trees”), or on deep learning procedures (Abraham, Para 0060 discloses a deep learning approach; “The convolutional layers thus extract features and the dense layers use said features to perform a regression or classification thereon.” More details in Fig. 7).
With regard to Claim 11, Abraham, Wozniak and Tuzoff disclose all the limitations of Claim 9 as discussed above, including a step of carrying out a dimensionality reduction model learning procedure, and carrying out the refinement model learning,
wherein said step of carrying out a dimensionality reduction model learning procedure comprises Principal Component Analysis (PCA) or Partial Least Squares (PLS) (see discussion in Claim 1).
With regard to Claim 12, Abraham, Wozniak, and Tuzoff disclose all the limitations of Claim 11 as discussed above, including a feature engineering model (Abraham; see discussion in Claim 1), a set of regression models (22) with a two-level stacking technique, comprising a first level (221) of one or more models, and a second level (222) metamodel (2221) configured to output refined coordinates produced from new images after the learning step (Wozniak; see discussion in Claim 1).
With regard to Claim 13, Abraham, Wozniak, and Tuzoff disclose all the limitations of Claim 12 as discussed above, including wherein said one or more models of said set of regression models (22) comprise at least one of the following models: support vector machine (2211); and/or decision trees (2212); random forest (2213); extra tree (2214); or gradient boosting (2215) (Wozniak; see discussion in Claim 1).
With regard to Claim 15, Abraham, Wozniak and Tuzoff disclose the method according to Claim 1, wherein combining (37) the anatomical points of interest comprises:
aggregating and repositioning (371) the anatomical points of interest (Abraham, Para 0064; “Thereafter, all the points are mapped back to the original image at step 640.”);
reporting (372) missing anatomical points (In detecting points with supervised learning models like the application, all target points should be detected, albeit with variable level of confidence or error. Abraham discloses providing prediction confidence for every predicted point. Para 0068; “During the refinement stage each refinement model may be used to obtain predictions for a corresponding group of point and to estimate the prediction confidence.”);
carrying out a cephalometric tracing (373) (Abraham, Para 0008; “Cephalometric landmarks are used as reference points for the construction of various cephalometric lines or planes …”); and
performing a cephalometric analysis (374) (Abraham, Paras 0010-0011; “The resulting cephalometric tracings outline the particular measurements, landmarks, and angles that medical professionals need for treatment. … One example of a result typically generated in cephalometric analysis is a Jarabak analysis, developed by Joseph Jarabak in 1972.”).
With regard to Claim 16, Abraham, Wozniak and Tuzoff disclose the method according to Claim 1. Abraham further discloses a system for analyzing digital radiographic images (Abraham, Abstract; “A system and method are described for automating the analysis of cephalometric x-rays.”), comprising
a display unit (43) (Abraham, Fig. 3 shows a computer with display monitor.); and
processing means (41), connected to said display unit (43) (Abraham, Fig. 3 shows the display unit (1208) being connected to a processing means (1200)), configured to carry out the method according to claim 1 (see the discussion above for Claim 1).
With regard to Claim 17, Abraham, Wozniak and Tuzoff disclose the method according to Claim 1. Abraham further discloses a computer program comprising instructions (Abraham, Para 0042; “A method for automated landmark detection can be executed in a server having a processing circuit executing programmed instructions to act on said instructions and image data”) which, when the computer program is executed by a computer, cause the computer to execute the steps of the method according to claim 1 (see the discussion above for Claim 1), wherein the computer program is stored in a non-transitory computer-readable storage medium (Abraham, Para 0042; “… a server having a processing circuit …”).
With regard to Claim 18, Abraham, Wozniak and Tuzoff disclose the method according to Claim 1. Abraham further discloses a non-transitory computer readable storage medium (Abraham, Para 0042; “… a server having a processing circuit …”) comprising instructions (Abraham, Para 0044; “a simplified machine learning system (e.g., a CNN Engine 1200) according to the present disclosure, which includes a plurality of modules to carry out the CNN method described herein and in this context”) which, when executed by a computer, cause the computer to execute the steps of the method according to claim 1 (see the discussion above for Claim 1).
Claims 5-6 are rejected under 35 U.S.C. 103 as being unpatentable over Abraham, Wozniak and Tuzoff, in view of Kim (US 20200035351 A1; hereafter Kim) and Clymer et al (US 20210327061 A1; hereafter Clymer).
With regard to Claim 5, Abraham, Wozniak and Tuzoff disclose all the limitations of Claim 1 as discussed above, but do not clearly and explicitly disclose wherein said general model learning procedure (13) comprises a first data augmentation step (131) comprising:
random rotation (1311),
random horizontal flip (1312),
random contrast adjustment (1313),
random brightness adjustment (1314), and
random resizing and cutting out (1315, 1316) of the digital radiographic image.
Kim in the same field of endeavor discloses a first data augmentation step comprising:
random contrast adjustment (Kim, Para 0107; “the brightness and contrast of a medical lateral head image for learning a landmark prediction model may also be randomly determined from the same medical lateral head image”);
random brightness adjustment (Kim, Para 0107; “the brightness and contrast of a medical lateral head image for learning a landmark prediction model may also be randomly determined from the same medical lateral head image”); and
random resizing and cutting out of the digital radiographic iamge (Kim, Para 0105; “a medical lateral head image for learning can be an image of which the size of the image including a lateral facial region including landmarks is randomly determined”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham, Wozniak and Tuzoff, as suggested by Kim, in order to apply the data augmentation method. One of ordinary skill in the art would have been motivated to make the modification for the benefit of increasing the number of training data so as to increase the performance of the trained model (Kim, Para 0106; “the prediction level can be improved with an increase in the amount of learning in the landmark prediction model.”).
Abraham, Wozniak, Tuzoff and Kim do not clearly and explicitly disclose comprising the following data augmentation sub-step:
random rotation, and
random horizontal flip.
Clymer in the same field of endeavor discloses the following data augmentation sub-step:
random rotation (Clymer, Para 0038; “During training, the images were augmented with random flips, rotations, and translations to increase the volume of the training data.”) (Clymer, Para 0045; “… rotation was applied between −45 and 45 degrees …”), and
random horizontal flip (Clymer, Para 0038; “During training, the images were augmented with random flips, rotations, and translations to increase the volume of the training data.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham, Wozniak, Tuzoff and Kim, as suggested by Clymer, in order to apply the data augmentation method of image rotation and flip. One of ordinary skill in the art would have been motivated to make the modification for the benefit of increasing the number of training data so as to increase the performance of the trained model (Kim, Para 0106; “the prediction level can be improved with an increase in the amount of learning in the landmark prediction model.”).
With regard to Claim 6, Abraham, Wozniak, Tuzoff, Kim and Clymer disclose all the limitations of Claim 5 as discussed above, but do not clearly and explicitly disclose wherein said general model learning procedure (13) comprises resizing (132) the digital radiographic image (R) to a predetermined size.
Kim further discloses wherein said general model learning procedure comprises resizing the digital radiographic image (R) to a predetermined size (Kim, Para 0080; “a pre-process for the medical lateral head image 212 that adjusts the size to provide predetermined pixel units”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham, Wozniak, Tuzoff, Kim and Clymer, as further suggested by Kim, in order to change an input image to a predetermined size. One of ordinary skill in the art would have been motivated to make the modification for the benefit of making a model simpler by fixing its input layer’s size to a predetermined size, and increasing efficiency of both model training and testing (Kim, Para 0081; “the resolution or the size can be smaller than those of the original medical lateral head image, so the processing speed of the prediction model can be improved.”).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Abraham, Wozniak and Tuzoff, in view of Clymer and Tan et al (A Cascade Regression Model for Anatomical Landmark Detection. STACOM 2019, LNCS 12009, pp. 43-51, 2020; hereafter Tan).
With regard to Claim 7, Abraham, Wozniak and Tuzoff disclose all the limitations of Claim 1 as discussed above, but do not clearly and explicitly disclose wherein said refinement model learning procedure (14) comprises:
performing a second data augmentation step (141); and
executing said general model (13) learned in said general model learning procedure.
Clymer in the same field of endeavor discloses wherein said refinement model learning procedure comprises:
performing a second data augmentation step (141) (Clymer, Paras 0038 and 0045 show a first data augmentation being performed in the first stage (Para 0038), and then in the second stage for each patch, a second data augmentation (Para 0045); “Standard data augmentation of flips, shear, rotation, and translation were used.”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham, Wozniak and Tuzoff, as suggested by Clymer, in order to apply data augmentation for the learning data of the refinement model. One of ordinary skill in the art would have been motivated to make the modification for the benefit of increasing the number of training data so as to increase the performance of the trained model.
Abraham, Wozniak, Tuzoff and Clymer do not clearly and explicitly disclose wherein said refinement model learning procedure (14) comprises the sub-steps of executing said general model (13) learned in said general model learning procedure.
Tan in the same field of endeavor discloses wherein said refinement model learning procedure comprises the sub-steps of executing said general model learned in said general model learning procedure (Tan, Page 47,Section 2.2; “In the second stage, we propose a CNN model to refine the primary prediction. … The CNN is trained to predict the displacement vector ΔS from the primary prediction S0 to the true landmark position SGT. … The ground truth displace vector ΔSGT is given by ΔSGT = SGT − S0.”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham, Wozniak, Tuzoff and Clymer, as suggested by Tan, in order to execute the general model in training the refinement model. One of ordinary skill in the art would have been motivated to make the modification for the benefit of obtaining an initial estimation of the landmark location for utilizing a displacement-regression strategy (Tan, Page 44, Para 1; “For landmark detection, an intuitive patch-based approach is to regress displacements from patches center to the target landmark [3]. Then the landmark position is calculated by these displacements following a majority/average voting strategy.”).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Abraham, Wozniak, Tuzoff, Clymer and Tan, further in view of Kim.
With regard to Claim 8, Abraham, Wozniak, Tuzoff, Clymer and Tan disclose all the limitations of Claim 7 as discussed above, but do not explicitly and clearly disclose wherein said second data augmentation step (141) comprises:
random rotation (1411),
random horizontal flip (1412),
adjusting contrast (1413) of said radiographic images (R) based on a random factor, and
adjusting brightness (1414) based on a predefined random factor.
Clymer further discloses wherein said second data augmentation step comprises:
random rotation (Clymer, Para 0038; “During training, the images were augmented with random flips, rotations, and translations to increase the volume of the training data.”) (Clymer, Para 0045; “… rotation was applied between −45 and 45 degrees …”), and
random horizontal flip (Clymer, Para 0045; “Flips were performed with 50% likelihood …”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham, Wozniak, Tuzoff, Clymer and Tan, as further suggested by Clymer, in order to apply the data augmentation steps. One of ordinary skill in the art would have been motivated to make the modification for the benefit of increasing the number of training data so as to increase the performance of the trained model.
Abraham, Wozniak, Tuzoff, Clymer and Tan as discussed above do not clearly and explicitly disclose:
adjusting contrast (1413) of said radiographic images (R) based on a random factor, and
adjusting brightness (1414) based on a predefined random factor.
Kim in the same field of endeavor discloses:
adjusting contrast of said radiographic images (R) based on a random factor (Kim, Para 0107; “the brightness and contrast of a medical lateral head image for learning a landmark prediction model may also be randomly determined from the same medical lateral head image”); and
adjusting brightness based on a predefined random factor (Kim, Para 0107; “the brightness and contrast of a medical lateral head image for learning a landmark prediction model may also be randomly determined from the same medical lateral head image”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham, Wozniak, Tuzoff, Clymer and Tan, as suggested by Kim, in order to apply the data augmentation steps. One of ordinary skill in the art would have been motivated to make the modification for the benefit of increasing the number of training data so as to increase the performance of the trained model.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Abraham, Wozniak and Tuzoff, further in view of Kim and Yum et al (US 20200013162 A1; hereafter Yum).
With regard to Claim 14, Abraham, Wozniak and Tuzoff disclose all the limitations of Claim 1 as discussed above, but do not clearly and explicitly disclose wherein a step (32) of pre-processing said analysis radiographic image (R’) comprises the following sub-steps:
performing an adaptive equalization of a Contrast Limited Adaptive Histogram Equalization (CLAHE) (321), wherein the image is modified in contrast; and
resizing (322) the analysis radiographic image (R’) to a predetermined size.
Kim in the same field of endeavor discloses wherein a step of pre-processing said analysis radiographic image (R’) (Kim, Para 0080; “a pre-process for the medical lateral head image 212 that adjusts the size to provide predetermined pixel units or adjusts contrast, resolution, brightness, and left-right symmetry can be further performed on the received medical lateral head image 212 after the receiving of a medical lateral head image (S210)”) comprises the following sub-steps:
modifying the contrast of the image (Kim, Para 0080; “adjusts contrast”); and
resizing the analysis radiographic image (R’) to a predetermined size (Kim, Para 0080; “adjusts the size to provide predetermined pixel units”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham, Wozniak and Tuzoff, as suggested by Kim, in order to pre-process the analysis image by adjusting its contrast and size. One of ordinary skill in the art would have been motivated to make the modification for the benefit of normalizing the image to be analyzed to a specific size and a proper contrast so that it can be accepted and properly analyzed by the trained model (Kim, Para 0080; “to have predetermined pixels, to be able to quickly analyze the medical lateral head image 212.”).
Abraham, Wozniak, Tuzoff and Kim do not explicitly and clearly disclose modifying image contrast by an adaptive equalization of a Contrast Limited Adaptive Histogram Equalization (CLAHE).
Yum in the same field of endeavor discloses modifying image contrast by an adaptive equalization of a Contrast Limited Adaptive Histogram Equalization (CLAHE) (Yum, Para 0131; “the apparatus 100 for analyzing a cephalometric image may preprocess the raw image by applying CLAHE parameters, a Gaussian blur or warping to the raw image”; CLAHE stands for Contrast Limited Adaptive Histogram Equalization.). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham, Wozniak, Tuzoff and Kim, as suggested by Yum, in order to use CLAHE for modifying contrast of an image. One of ordinary skill in the art would have been motivated to make the modification for the benefit of improving contrast in images without the noise-amplification problem.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure. Liu et al (US 20160328643 A1) discloses using PCA as a method for dimensionality reduction. Lindner et al (Scientific Reports, 6:33581 (2016)) discloses using a two-level structure comprising random forest as the first level and a linear model as the second level, for localization anatomical landmarks in cephalographic images.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LEI ZHANG whose telephone number is (571)272-7172. The examiner can normally be reached Monday-Friday 8am-5pm E.T..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pascal Bui-Pho can be reached at (571) 272-2714. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/L.Z./Examiner, Art Unit 3798
/PASCAL M BUI PHO/Supervisory Patent Examiner, Art Unit 3798
Prosecution Insights