DETAILED ACTION
This office action is responsive to original claims filed on 09/25/2024. Presently, Claims 1 - 18 remain pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 1, 4, 8, 9 and 15 are objected to because of the following informalities:
Claim 1, Lines 12-13: “each learning radiographic image (R) … plane of a learning radiographic image (R)” should be changed to “the learning radiographic image (R) … plane of the learning radiographic image (R)”.
Claim 1, Line 17: “one or more points of interest” should be changed to “one or more anatomical points of interest”.
Claim 1, Line 24: “training (151 … 15N)” should be changed to “training (151, …, 15N)”. The same change should be done for the same term in Claim 1, Line 37, and in Claim 9, Line 2.
Claim 1, Lines 32-33: “the cutting step (143) of the learning radiographic image” should be changed to “the cutting step (143) for the learning radiographic images”.
Claim 4, Lines 2-4: “comprising a step of performing …, which is carried out before …” should be changed to “comprising the step of performing … being carried out before …”.
Claim 5: “the radiographic image” in Line 4, “the acquired radiographic image” in Line 6, and “the radiographic image” in Line 12, should all be changed to “the learning radiographic image”.
Claim 8, Lines 9-12: the two “adjusting” contain almost the same content. Please delete one of them.
Claim 9, Line 3: “each image cutout (/?i,/?2> - > RN)” should be changed to “each image cutout (R1, R2, …, RN)”;
Claim 9, Line 4, “image (/?f)” should be changed to “image”.
Claim 15, Line 2: “said combining step (37) of said inference step (3)” should be changed to “said combining sub-step (37) of said inference step (3)”, so as to be consistent with the naming in Claim 1, Lines 26-27.
Claim 15, Line 7, “the original analysis radiographic image (R')” should be changed to “the analysis radiographic image (R’)”.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3-13 and 15 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 3, Line 6 recites “the cephalometric analysis (R”)”, and Claim 4, Line 4 recites “said analysis radiographic image (R”)”. It is unclear what the term R” denotes. For present purposes of examination, the Examiner neglects the term “(R”)” in the recited phrases.
Claim 5, Lines 12-13 recites “cutting out” and “cut out”. It is noted that the terms “cutting out” and “cutout” are recited in multiple other claims including Claims 1-4, and the specification provides multiple potential definitions of “cutting out” and cutout”: 1. cutting the analysis radiographic image into a plurality of image cutouts, in a similar way to said image cutting out step, wherein each image cutout comprises a respective group of anatomical points of interest (Page 3, Lines 16-18); 2. if part of the cutout does not fall within the image, it is filled with zeroes (Page 13, Line 14). For present purposes of examination, the recited terms in Claim 5 are interpreted as “cut out (if part of the cutout does not fall within the image, it is filled with zeroes)” (Specification, Page 13, Line 14).
Claim 7, Line 5 recites “executing said general model (13) as obtained from … procedure”. Here the number of 13 is used to denote “said general model”, but in other claims such as Line 2 of Claim 5 and Line 2 and Claim 6, the number of 13 is used to denote “said general model learning procedure”. For present purposes of examination, the part of “(13)” is not considered in the recited phrase above.
Claim 9, Line 4 recites “each cutout of said radiographic image”. It is unclear whether the “said radiographic image” is “said learning radiographic image” or “said analysis radiographic image”. For present purposes of examination, the recited “each cutout of said radiographic image” is interpreted as “each cutout of said learning radiographic image”.
Claim 12, Lines 8-10 recites “… at an output of … learning procedure, coordinates of … points … of each cutout are obtained”. In the learning process, coordinates of the anatomical points are supposed to be the input (or label) for training models, and the optimized parameter (e.g. weights) estimates of a model are typically the output. Hence, it is unclear why in the recited limitation the coordinates are obtained at an output of the model learning procedure. For present purposes of examination, the above recited limitation is interpreted as the coordinates being obtained in the inference procedure but not the learning procedure.
Claim 15, Line 5 recites “the annotations of the original model”. It is unclear what “the original model” is. For present purposes of examination, the phrase is interpreted as “the annotations returned by the general model”.
Claim 15 recites “said combining step (37) … comprises the steps of: … carrying out a cephalometric tracing (373) … performing a cephalometric analysis (374) …”. However, according to Claim 1, Lines 39-41, the combining step is “to obtain final geometric coordinates of points relative to the original analysis radiographic image (R’)”, which the steps of carrying out (373) and of performing (374) are not related to. For present purposes of examination, the steps of carrying out (373) and of performing (374) are interpreted as being separate from the combining step (37).
Claims 4, 6, 8, 10-11 and 13 are also rejected under 35 U.S.C. 112(b) because they inherit the indefiniteness of the claim(s) they respectively depend upon.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1 - 16 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
With regard to Claims 1-15:
Step 1: the claims are drawn to a method/process, one of the four statutory categories.
Step 2A, Prong One:
The claims recite the limitations of “performing … a learning step”, which comprises limitations of “executing … a general model learning procedure”, and “performing a refinement model learning procedure” that further comprises “cutting … into a plurality of image cutouts” and “training … a refinement model”, and “carrying out an inference step”, which comprises “performing … an inference step”, “cutting … into a plurality of image cutouts”, “performing … an inference” and “combining the anatomical points of interest” in Claim 1, “performing … a procedure for learning a radiograph cutout model” in Claim 2, “carrying out said inference step” and “performing … the inference step” in Claim 3, “performing … the inference step” and “performing … an inference step” in Claim 4, “a first data augmentation step comprising the following sub-steps” in Claim 5, “a resizing sub-step” in Claim 6, “performing a second data augmentation step” and “executing said general model” in Claim 7, “said second data augmentation step … comprises the following sub-steps” in Claim 8, “resizing each cutout”, “carrying out a feature engineering and refinement model learning procedure”, and “carrying out a dimensionality reduction model learning procedure and carrying out the refinement model learning” in Claim 9, “computer vision algorithms” and “deep learning procedures” in Claim 10, “PCA … or PLS” in Claim 11, “a feature engineering model or procedure” and “a set of regression models” in Claim 12, “support vector machine … gradient boosting” in Claim 13, “performing an adaptive equalization” and “resizing … image” in Claim 14, “aggregating and repositioning (371) the anatomical points of interest” and “carrying out a cephalometric tracing” in Claim 15. These limitations are, under their broadest reasonable interpretation, limitations that cover performance of the limitation in the mind, or mathematical calculations. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind or mathematical calculations but for the recitation of generic computer components, then it falls within the “Mental Processes” or “Mathematical Concepts” grouping of abstract ideas. Accordingly, the claims recite an abstract idea.
Step 2A, Prong Two:
This judicial exception is not integrated into a practical application. In particular, the claims recite the additional elements – using a display unit and processing means, receiving images, displaying coordinates of points in Claim 1, and reporting missing points and performing a cephalometric analysis in Claim 15. The display and processing means are recited at a high-level of generality (i.e., as generic computer components performing generic computer function of displaying and processing) such that it amounts no more than mere instructions to apply the exception using generic computer components. The additional elements of receiving images and reporting results are insignificant extra-solution activities. Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea.
Step 2B:
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements either amount to no more than mere instructions to apply the exception using generic computer components, or are insignificant extra-solution activities, so accordingly cannot provide an inventive concept.
For the reasons set forth above, Claims 1-15 are not patent eligible.
With regard to Claim 16:
Step 1: the claim is drawn to a system/apparatus, one of the four statutory categories.
Step 2A, Prong One:
The claim recites the same limitations as Claims 1-15 as discussed above. Accordingly, the claim recites an abstract idea.
Step 2A, Prong Two:
The claim recites the same additional elements as Claims 1-15 as discussed above. Accordingly, the claim is directed to an abstract idea.
Step 2B:
Same as Claims 1-15 discussed above, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
For the reasons set forth above, Claim 16 is not patent eligible.
Claim 17 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because the broadest reasonable interpretation of the “computer program” of claim 17 encompasses software per se.
Claim 18 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because the broadest reasonable interpretation of the “computer readable storage medium” of claim 18 encompasses signals per se. The specification does not specify what types of computer readable storage medium include. Hence, the broadest reasonable interpretation of claim 18 in view of the state of the art encompasses signals which are not within one of the four statutory categories of invention. It is suggested that Claim 18 be amended to recite a “non-transitory” computer readable medium to overcome this rejection.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1 and 15-18 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Abraham et al (US 20180061054 A1; hereafter Abraham).
With regard to Claim 1, Abraham discloses a method for a computer-implemented geometric analysis of digital radiographic images (R) (Abraham, Abstract; “A system and method are described for automating the analysis of cephalometric x-rays.”) using a radiographic system (4), wherein said radiographic system (4) comprises:
a display unit (43) (Abraham, Fig. 3 shows a computer with display monitor.); and
processing means (41), connected to said display unit (43) (Abraham, Fig. 3 shows the display unit (1208) being connected to a processing means (1200).),
said method comprising the steps of:
performing with said processing means (41) a learning step (1) (Abraham, Para 0044; “a CNN Training Module 1202 can be employed to train the CNN system using pre-existing rules and prior store image data.”) comprising the following sub-steps:
receiving (11) a plurality of digital learning radiographic images (Abraham, Para 0046; “CNN Training Module 1202 is coupled to a data store 1206 containing a large plurality of pre-stored images relevant to the cephalometric image of interest.”), each accompanied by annotations, wherein an annotation comprises a label identifying an anatomical point of interest of each learning radiographic image (R), and geometric coordinates of the anatomical point of interest (Abraham, Para 0046; “data store 1206 may contain analysis results or landmark data associated with the many stored images thereon, similar to those depicted in FIG. 1B, FIG. 1C, or other data indicative of landmarks, structure and analytical results of prior analysis by humans or machines, corresponding to the stored images.”) in a plane of a learning radiographic image (R);
executing (13), with said processing means (41) for each learning radiographic image (R), a general model learning procedure for learning a general model for detecting one or more points of interest from the learning radiographic image (R) (Abraham, Para 0046; “These stored images, and optionally the stored landmarks and analytic information associated therewith, are used to train the CNN system using the CNN Training Module discussed.”); and
performing a refinement model learning procedure (14) (Abraham, Para 0063; “In an attempt to maximize accuracy, the algorithm then attempts to fine-tune the points location as follows.”), comprising the sub-steps of:
cutting (143) the learning radiographic image into a plurality of image cutouts (R1, R2, ..., RN), each comprising a respective group of anatomical points of interest (Abraham, Para 0064; “A complete image may be divided into separate regions, for example defined by groups or sub-groups of landmark points of interest …”); and
training (151 ...15N) a refinement model (2) for each image cutout (RI, R2, ..., RN) (Abraham, Para 0064; “… a separate CNN-based model for each group in steps 630”) (Abraham, Para 0067; “The points refinement process is set up in a way that all refinement models are independent …”); and
carrying out an inference step (3) using said processing means (41) on a digital analysis radiographic image (R') (Abraham, Para 0063 “FIG. 6 illustrates an outline of a detection process 601 as carried out in Landmark Detection Module 1204 in an exemplary embodiment. … for example an x-ray image of a patient’s head and jaw.”), comprising the following sub-steps:
performing (33) on said analysis radiographic image (R') an inference step based on said general model learned in said general model learning procedure, so as to obtain geometric coordinates of a plurality of anatomical points of interest (Abraham, Para 0063; “In the first phase of the detection, at step 610, a CNN detection process is carried out on the whole cephalometric image of interest …”);
cutting (34) the analysis radiographic image (R') into a plurality of image cutouts (R'1, R'2, ..., R'N) as in the cutting step (143) of the learning radiographic image, wherein each image cutout (R'1, R'2, ..., R'N) of the analysis radiographic image (R') comprises a respective group of anatomical points of interest (Abraham, Para 0064; “These 90 landmarks can be divided into 12 separate groups of landmark points, each in different area of the skull.”); and
performing (361 ...36N) on each cutout of the analysis radiographic image (R') an inference through said refinement model obtained in said training step (151 ... 15N) of said refinement model learning procedure (14) (Abraham, Para 0064; “The points in each group are refined with a separate CNN-based model for each group in steps 630.”); and
combining (37) the anatomical points of interest of each image cutout (R'1, R'2, ..., R'N) of the analysis radiographic image (R') so as to obtain final geometric coordinates of points relative to the original analysis radiographic image (R') (Abraham, Para 0064; “Thereafter, all the points are mapped back to the original image at step 640.”); and
displaying said final geometric coordinates of the points relative to the original analysis radiographic image (R') with said display unit (43) (Abraham, Fig. 2 displays an example of whole image with the detected landmarks.).
With regard to Claim 15, Abraham discloses the method according to claim 1, wherein said combining step (37) of said inference step (3) comprises the steps of:
aggregating and repositioning (371) the anatomical points of interest, wherein the annotations returned by the refinement models are aggregated together with the annotations of the original model, in such a way that geometric coordinates of the anatomical points detected are relative to the original analysis radiographic image (R') (Abraham, Para 0064; “Thereafter, all the points are mapped back to the original image at step 640.”);
reporting (372) missing anatomical points of interest, wherein the reporting comprises reporting whether there are points that have not been detected (In detecting points with supervised learning models like the application, all target points should be detected, albeit with variable level of confidence or error. Abraham discloses providing prediction confidence for every predicted point. Para 0068; “During the refinement stage each refinement model may be used to obtain predictions for a corresponding group of point and to estimate the prediction confidence.”);
carrying out a cephalometric tracing (373), wherein, based on the detected points, tracing lines are defined (Abraham, Para 0008; “Cephalometric landmarks are used as reference points for the construction of various cephalometric lines or planes …”); and
performing a cephalometric analysis (374), wherein, based on the detected points, one or more cephalometric analyses among cephalometric analyses known in scientific literature are performed (Abraham, Paras 0010-0011; “The resulting cephalometric tracings outline the particular measurements, landmarks, and angles that medical professionals need for treatment. … One example of a result typically generated in cephalometric analysis is a Jarabak analysis, developed by Joseph Jarabak in 1972.”).
With regard to Claim 16, Abraham discloses a system for analyzing digital radiographic images (Abraham, Abstract; “A system and method are described for automating the analysis of cephalometric x-rays.”), comprising
a display unit (43) (Abraham, Fig. 3 shows a computer with display monitor.); and
processing means (41), connected to said display unit (43) (Abraham, Fig. 3 shows the display unit (1208) being connected to a processing means (1200)), configured to carry out the method according to claim 1 (see the discussion above for Claim 1).
With regard to Claim 17, Abraham discloses a computer program comprising instructions (Abraham, Para 0042; “A method for automated landmark detection can be executed in a server having a processing circuit executing programmed instructions to act on said instructions and image data”) which, when the computer program is executed by a computer, cause the computer to execute the steps of the method according to claim 1 (see the discussion above for Claim 1).
With regard to Claim 18, Abraham discloses a computer readable storage medium comprising instructions (Abraham, Para 0044; “a simplified machine learning system (e.g., a CNN Engine 1200) according to the present disclosure, which includes a plurality of modules to carry out the CNN method described herein and in this context”) which, when executed by a computer, cause the computer to execute the steps of the method according to claim 1 (see the discussion above for Claim 1).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2-4 are rejected under 35 U.S.C. 103 as being unpatentable over Abraham in view of Kim (US 20200035351 A1; hereafter Kim).
With regard to Claim 2, Abraham discloses all the limitations of Claim 1 as discussed above. Abraham does not clearly and explicitly disclose wherein said learning step (1) comprises a sub-step of performing (12), with said processing means (41), for each learning radiographic image (R), a procedure for learning a radiograph cutout model for cutting out a part of a lateral-lateral teleradiograph of a skull that is relevant for cephalometric analysis.
Kim in the same field of endeavor discloses wherein said learning step (1) comprises a sub-step of performing (12), with said processing means (41), for each learning radiographic image (R), a procedure for learning a radiograph cutout model for cutting out a part of a lateral-lateral teleradiograph of a skull that is relevant for cephalometric analysis (Kim, Para 0090; “A learning method of a lateral facial region prediction model … FIGS. 3A to 3C show a procedure of creating medical lateral head image data for learning a lateral facial region prediction model, the data being used in the method of predicting anatomical landmarks according to an embodiment of the present disclosure and the device for predicting anatomical landmarks using the method.”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham, as suggested by Kim, in order to train a model for cutting out a part of lateral-lateral teleradiograph of skull. One of ordinary skill in the art would have been motivated to make the modification for the benefit of improving the prediction accuracy of the landmarks (Kim, Para 0060; “the landmark prediction model can be a model learned to predict x-axial and y-axial coordinates of landmarks in the lateral facial region on the basis of a medical lateral head image and the coordinates of a lateral facial region. By this learning method, the landmark prediction model can predict landmarks with higher accuracy than when it is configured to predict landmarks in the entire medical lateral head image”).
With regard to Claim 3, Abraham and Kim disclose all the limitations of Claim 2 as discussed above, but do not clearly and explicitly disclose wherein carrying out said inference step (3) comprises a sub-step of performing (31), on said analysis radiographic image (R'), the inference step based on said radiograph cutout model learned in said radiograph cutout model learning procedure (12), so as to obtain a cutout of the part of the lateral-lateral teleradiograph of the skull relevant for the cephalometric analysis (R").
Kim further discloses wherein carrying out said inference step (3) comprises a sub-step of performing (31), on said analysis radiographic image (R'), the inference step based on said radiograph cutout model learned in said radiograph cutout model learning procedure (12), so as to obtain a cutout of the part of the lateral-lateral teleradiograph of the skull relevant for the cephalometric analysis (R") (Kim, Para 0078; “a lateral facial region using a lateral facial region prediction model configured to predict a lateral facial region in the medical lateral head image is predicted (S220).”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham and Kim, as further suggested by Kim, in order to obtain a cutout of the part of the lateral-lateral teleradiograph of the skull. One of ordinary skill in the art would have been motivated to make the modification for the benefit of improving the prediction accuracy of the landmarks (Kim, Para 0060; “the landmark prediction model can be a model learned to predict x-axial and y-axial coordinates of landmarks in the lateral facial region on the basis of a medical lateral head image and the coordinates of a lateral facial region. By this learning method, the landmark prediction model can predict landmarks with higher accuracy than when it is configured to predict landmarks in the entire medical lateral head image”).
With regard to Claim 4, Abraham and Kim disclose all the limitations of Claim 3 as discussed above, but do not clearly and explicitly disclose further comprising a step of performing (31), on said analysis radiographic image (R'), the inference step based on said radiograph cutout model, which is carried out before said step of performing (33) on said analysis radiographic image (R") an inference step based on said general model learned in said general model learning procedure (13).
Kim further discloses further comprising a step of performing (31), on said analysis radiographic image (R'), the inference step based on said radiograph cutout model, which is carried out before said step of performing (33) on said analysis radiographic image (R") an inference step based on said general model learned in said general model learning procedure (13) (Kim, Fig. 2B shows the procedure of first predicting a lateral facial region (212, 222, 224), and then predict anatomical landmarks in the cropped region (226, 232, 234).). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham and Kim, as further suggested by Kim, in order to crop the part of the lateral-lateral teleradiograph of the skull before performing landmark detection. One of ordinary skill in the art would have been motivated to make the modification for the benefit of improving the prediction accuracy of the landmarks (Kim, Para 0060; “the landmark prediction model can be a model learned to predict x-axial and y-axial coordinates of landmarks in the lateral facial region on the basis of a medical lateral head image and the coordinates of a lateral facial region. By this learning method, the landmark prediction model can predict landmarks with higher accuracy than when it is configured to predict landmarks in the entire medical lateral head image”).
Claims 5-6 are rejected under 35 U.S.C. 103 as being unpatentable over Abraham and Kim, further in view of Clymer et al (US 20210327061 A1; hereafter Clymer).
With regard to Claim 5, Abraham discloses all the limitations of Claim 1 as discussed above. Abraham does not clearly and explicitly disclose wherein said general model learning procedure (13) comprises a first data augmentation step (131) comprising the following sub-steps:
random rotation (1311) of the radiographic image (R) by a predefined range of angles with predefined probability;
random horizontal flip (1312), wherein the acquired radiographic images (R) with the annotations are randomly flipped horizontally with a predefined probability;
random contrast adjustment (1313), wherein image contrast is adjusted based on a predefined random factor;
random brightness adjustment (1314), wherein a brightness of images is adjusted based on a predefined random factor; and
random resizing and cutting out (1315, 1316), wherein the radiographic image (R) is resized with a random scale factor and cut out.
Kim in the same field of endeavor discloses wherein said general model learning procedure (13) comprises a first data augmentation step (131) comprising the following sub-steps:
random contrast adjustment (1313), wherein image contrast is adjusted based on a predefined random factor (Kim, Para 0107; “the brightness and contrast of a medical lateral head image for learning a landmark prediction model may also be randomly determined from the same medical lateral head image”);
random brightness adjustment (1314), wherein a brightness of images is adjusted based on a predefined random factor (Kim, Para 0107; “the brightness and contrast of a medical lateral head image for learning a landmark prediction model may also be randomly determined from the same medical lateral head image”); and
random resizing and cutting out (1315, 1316), wherein the radiographic image (R) is resized with a random scale factor and cut out (Kim, Para 0105; “a medical lateral head image for learning can be an image of which the size of the image including a lateral facial region including landmarks is randomly determined”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham, as suggested by Kim, in order to apply the data augmentation method. One of ordinary skill in the art would have been motivated to make the modification for the benefit of increasing the number of training data so as to increase the performance of the trained model (Kim, Para 0106; “the prediction level can be improved with an increase in the amount of learning in the landmark prediction model.”).
Abraham and Kim do not clearly and explicitly disclose comprising the following data augmentation sub-step:
random rotation (1311) of the radiographic image (R) by a predefined range of angles with predefined probability;
random horizontal flip (1312), wherein the acquired radiographic images (R) with the annotations are randomly flipped horizontally with a predefined probability.
Clymer in the same field of endeavor discloses comprising the following data augmentation sub-step:
random rotation (1311) of the radiographic image (R) by a predefined range of angles with predefined probability (Clymer, Para 0038; “During training, the images were augmented with random flips, rotations, and translations to increase the volume of the training data.”) (Clymer, Para 0045; “… rotation was applied between −45 and 45 degrees …”);
random horizontal flip (1312), wherein the acquired radiographic images (R) with the annotations are randomly flipped horizontally with a predefined probability (Clymer, Para 0038; “During training, the images were augmented with random flips, rotations, and translations to increase the volume of the training data.”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham and Kim, as suggested by Clymer, in order to apply the data augmentation method of image flip. One of ordinary skill in the art would have been motivated to make the modification for the benefit of increasing the number of training data so as to increase the performance of the trained model (Kim, Para 0106; “the prediction level can be improved with an increase in the amount of learning in the landmark prediction model.”).
With regard to Claim 6, Abraham, Kim and Clymer disclose all the limitations of Claim 5 as discussed above. As discussed above, Kim discloses wherein said general model learning procedure (13) comprises a resizing sub-step (132) (Kim, Para 0105; “a medical lateral head image for learning can be an image of which the size of the image including a lateral facial region including landmarks is randomly determined”).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Abraham and Clymer, further in view of Tan et al (A Cascade Regression Model for Anatomical Landmark Detection. STACOM 2019, LNCS 12009, pp. 43-51, 2020; hereafter Tan).
With regard to Claim 7, Abraham discloses all the limitations of Claim 1 as discussed above. Abraham does not clearly and explicitly disclose wherein said refinement model learning procedure (14) comprises the sub-steps of:
performing a second data augmentation step (141); and
executing said general model (13) as obtained from said general model learning procedure.
Clymer in the same field of endeavor discloses wherein said refinement model learning procedure (14) comprises the sub-steps of:
performing a second data augmentation step (141) (Clymer, Paras 0038 and 0045 show a first data augmentation being performed in the first stage (Para 0038), and then in the second stage for each patch, a second data augmentation (Para 0045); “Standard data augmentation of flips, shear, rotation, and translation were used.”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham, as suggested by Clymer, in order to apply data augmentation for the learning data of the refinement model. One of ordinary skill in the art would have been motivated to make the modification for the benefit of increasing the number of training data so as to increase the performance of the trained model.
Abraham and Clymer do not clearly and explicitly disclose wherein said refinement model learning procedure (14) comprises the sub-steps of executing said general model (13) as obtained from said general model learning procedure.
Tan in the same field of endeavor discloses wherein said refinement model learning procedure (14) comprises the sub-steps of executing said general model (13) as obtained from said general model learning procedure (Tan, Page 47,Section 2.2; “In the second stage, we propose a CNN model to refine the primary prediction. … The CNN is trained to predict the displacement vector ΔS from the primary prediction S0 to the true landmark position SGT. … The ground truth displace vector ΔSGT is given by ΔSGT = SGT − S0.”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham and Clymer, as suggested by Tan, in order to execute the general model in training the refinement model. One of ordinary skill in the art would have been motivated to make the modification for the benefit of obtaining an initial estimation of the landmark location for utilizing a displacement-regression strategy (Tan, Page 44, Para 1; “For landmark detection, an intuitive patch-based approach is to regress displacements from patches center to the target landmark [3]. Then the landmark position is calculated by these displacements following a majority/average voting strategy.”).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Abraham, Clymer and Tan, further in view of Kim.
With regard to Claim 8, Abraham, Clymer and Tan disclose all the limitations of Claim 7 as discussed above. Clymer further discloses wherein said second data augmentation step (141) of said refinement model learning procedure (14) comprises the following sub-steps:
random rotation (1411), wherein each radiographic image (R) and related annotations are rotated by a predefined range of angles and/or with a predefined probability, generating a plurality of rotated images (Clymer, Para 0038; “During training, the images were augmented with random flips, rotations, and translations to increase the volume of the training data.”) (Clymer, Para 0045; “… rotation was applied between −45 and 45 degrees …”);
horizontal flip (1412) of the radiographic images (R) randomly annotated with a predefined probability (Clymer, Para 0045; “Flips were performed with 50% likelihood …”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham, Clymer and Tan, as further suggested by Clymer, in order to apply data augmentation for the learning data of the refinement model. One of ordinary skill in the art would have been motivated to make the modification for the benefit of increasing the number of training data so as to increase the performance of the trained model.
Abraham, Clymer and Tan do not clearly and explicitly disclose wherein said second data augmentation step (141) of said refinement model learning procedure (14) comprises the following sub-steps:
adjusting contrast (1413) of said radiographic images (R) based on a predefined random factor; and
adjusting the contrast (1414) of said radiographic images based on a predefined random factor.
Kim in the same field of endeavor discloses wherein said second data augmentation step (141) of said refinement model learning procedure (14) comprises the following sub-steps:
adjusting contrast (1413) of said radiographic images (R) based on a predefined random factor (Kim, Para 0107; “the brightness and contrast of a medical lateral head image for learning a landmark prediction model may also be randomly determined from the same medical lateral head image”); and
adjusting the contrast (1414) of said radiographic images based on a predefined random factor (Kim, Para 0107; “the brightness and contrast of a medical lateral head image for learning a landmark prediction model may also be randomly determined from the same medical lateral head image”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham, Clymer and Tan, as suggested by Kim, in order to apply data augmentation for the learning data of the refinement model. One of ordinary skill in the art would have been motivated to make the modification for the benefit of increasing the number of training data so as to increase the performance of the trained model.
Claims 9-11 are rejected under 35 U.S.C. 103 as being unpatentable over Abraham in view of Liu et al (US 20160328643 A1; hereafter Liu).
With regard to Claim 9, Abraham discloses all the limitations of Claim 1 as discussed above. Abraham further discloses wherein said step of training (151 ... 15N) a refinement model for each image cutout (/?i,/?2> - > RN) comprises the following sub-steps:
resizing (1511 ... 15N1) each cutout of said radiographic image (/?E) (Abraham, Para 0067; “each refinement model is trained according to the training process of FIG. 5, with the exception that only the corresponding subset of points is used and input images are cropped to contain only the relevant part of the skull.” Paras 0070, 0088, 0105, 0123 and 0190 disclose that the refinement models for the different sub-regions accept input image of different sizes.), and carrying out a feature engineering and refinement model learning procedure (2) (Abraham, Para 0014 discloses a random-tree approach for feature extraction; “a method has been proposed for automatically annotating objects in radiographic images by using predictive modeling approaches based on decision trees”) (Abraham, Para 0060 discloses a deep learning approach; “The convolutional layers thus extract features and the dense layers use said features to perform a regression or classification thereon.” More details in Fig. 7).
Abraham does not clearly and explicitly disclose carrying out a dimensionality reduction model learning procedure, and carrying out the refinement model learning.
Liu in the same field of endeavor discloses carrying out a dimensionality reduction model learning procedure, and carrying out the refinement model learning (Liu, Para 0026; “… the approximation of the deep neural network may be calculated as the deep neural network is being trained or multiple iterations of training and approximating may be performed.”)(Liu, Para 0043; “Using PCA, … where K indicates the reduced dimensionality …”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham, as suggested by Liu, in order to reduce dimensionality in the process of model training. One of ordinary skill in the art would have been motivated to make the modification for the benefit of reducing the computational complexity thus achieving a higher speed for anatomical object detection (Liu, Para 0060; “As a result, a speed up of more 30 times the detection speed of the original trained deep neural network was achieved for LZ apex detection at the error rate of 4.54%”).
With regard to Claim 10, Abraham and Liu disclose all the limitations of Claim 9 as discussed above. Abraham further discloses wherein said step of carrying out a feature engineering and refinement model learning procedure (2) is based on computer vision algorithms (Abraham, Para 0014 discloses a random-tree approach for feature extraction; “a method has been proposed for automatically annotating objects in radiographic images by using predictive modeling approaches based on decision trees”), or on deep learning procedures (Abraham, Para 0060 discloses a deep learning approach; “The convolutional layers thus extract features and the dense layers use said features to perform a regression or classification thereon.” More details in Fig. 7).
With regard to Claim 11, Abraham and Liu disclose all the limitations of Claim 9 as discussed above, but do not explicitly and clearly disclose wherein said step of carrying out a dimensionality reduction model learning procedure comprises Principal Component Analysis (PCA) or Partial Least Squares regression (PLS).
Liu further discloses wherein said step of carrying out a dimensionality reduction model learning procedure comprises Principal Component Analysis (PCA) or Partial Least Squares regression (PLS) (Liu, Para 0043; “Using PCA, … where K indicates the reduced dimensionality …). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham and Liu, as further suggested by Liu, in order to comprise PCA in the dimensionality reduction model learning procedure. One of ordinary skill in the art would have been motivated to make the modification for the benefit of exploiting redundancy in the hidden layers of the model being trained (Liu, Para 0042; “… principal component analysis (PCA) can be used to reduce computational costs by exploiting redundancy over all of the hidden layer units (nodes)”).
Claims 12-13 are rejected under 35 U.S.C. 103 as being unpatentable over Abraham and Liu, further in view of Lindner et al (Fully Automatic System for Accurate Localisation and Analysis of Cephalometric Landmarks in Lateral Cephalograms. Scientific Reports, 6:33581 (2016); hereafter Lindner).
With regard to Claim 12, Abraham and Liu disclose all the limitations of Claim 11 as discussed above. Abraham further discloses wherein said step of carrying out a feature engineering and refinement model learning procedure (2) comprises:
a feature engineering model or procedure (21) (Abraham, Para 0060; “The convolutional layers thus extract features and the dense layers use said features to perform a regression or classification thereon.”).
Abraham and Liu do not explicitly and clearly disclose a set of regression models (22) with a two-level stacking technique, comprising,
a first level (221), comprising one or more models, and
a second level (222) comprising a metamodel (2221) and
wherein at an output of said refinement model learning procedure (2), coordinates of a group of anatomical points or points of interest of each cutout of said radiographic image (Ri) are obtained.
Lindner in the same field of endeavor discloses a set of regression models (22) with a two-level stacking technique (Lindner, Page 3, Para 3; “The FALA system follows a machine learning approach where Random Forest regression-voting is used both to detect the position, scale and orientation of the skull (similar to Hough Forests24) and then, in the Constrained Local Model framework (RFRV-CLM), to locate the individual landmarks25.”. A person having ordinary skill in the art would understand this to be a two-level stacking technique), comprising,
a first level (221), comprising one or more models (Lindner, Page 3, Para 4; “Random Forests (RFs)26 … During search, all trees in the RF will cast independent votes to make predictions on the landmark’s position.”), and
a second level (222) comprising a metamodel (2221) (Lindner, Page 3, Para 6; “Based on a number of landmark points in a set of images, a Statistical Shape Model (SSM) is trained by applying principal component analysis to the aligned shapes27. This yields a linear model of shape variation which represents the position of each landmark point l …”) and
wherein at an output of said refinement model learning procedure (2), coordinates of a group of anatomical points or points of interest of each cutout of said radiographic image (Ri) are obtained (Lindner, Fig. 2, top row shows the process of obtaining the location of the anatomical points).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham and Liu, as suggested by Lindner, in order to further refine the position of the detected landmarks using a two-level tacking technique involving one or more regression models. One of ordinary skill in the art would have been motivated to make the modification for the benefit of improving the detection accuracy by combining global shape constraints with local pattern of intensities (Lindner, Page 3, Para 6; “CLMs combine global shape constraints with local models of the pattern of intensities.”).
With regard to Claim 13, Abraham, Liu and Lindner disclose all the limitations of Claim 12 as discussed above, but do not disclose wherein said one or more models of said set of regression models (22) comprise at least one of the following models: support vector machine (2211); and/or decision trees (2212); random forest (2213); extra tree (2214); or gradient boosting (2215).
Lindner further discloses wherein said one or more models of said set of regression models (22) comprise at least one of the following models: support vector machine (2211); and/or decision trees (2212); random forest (2213) (Lindner, Page 3, Para 4; “Random Forests (RFs)26 … During search, all trees in the RF will cast independent votes to make predictions on the landmark’s position.”); extra tree (2214); or gradient boosting (2215). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abraham, Liu and Lindner, as further suggested by Lindner, in order to use random forest for the detection task. One of ordinary skill in the art would have been motivated to make the modification for the benefit of random forest being a popular regression method for its balance of accuracy and efficiency (Lindner, Page 9, Para 3; “All techniques submitted to the 2014 and 2015 ISBI Grand Challenges followed a supervised learning approach with RFs being the main method of choice13”).
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Abraham and Kim, further in view of Yum et al (US 20200013162 A1; hereafter Yum).
With regard to Claim 14, Abraham discloses all the limitations of Claim 1 as discussed above. Abraham does not clearly and explicitly disclose wherein a step (32) of pre-processing said analysis radiographic image (R') comprises the following sub-steps:
performing an adaptive equalization of a contrast-limited histogram (321), wherein the image is modified in contrast; and
resizing (322) the analysis radiographic image (R').
Kim in the same field of endeavor discloses wherein a step (32) of pre-processing said analysis radiographic image (R') (Kim, Para 0080; “a pre-process for the medical lateral head image 212 that adjusts the size to provide predetermined pixel units or adjusts contrast, resolution, brightness, and left-right symmetry can be further performed on the received m