DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments have been fully considered but are not persuasive. Applicant respectfully submits that the prior art fails to teach “generating an input to a second trained classification network based at least in part on the location predicted in (b)”. Applicant argues that the positional data in D1 is used as a routing signal to select an algorithm, but not an input generated for and processed by a second network. Applicant further notes that the D1’s pipeline proceeds from a metadata analyzer (104) to an algorithm selector (108), to an image analyzer (110). The positional data is an output of the metadata analyzer that feeds the algorithm selector, which is the component that decides which algorithm to invoke; it does not feed the image analyzer, which is the component that executes the algorithm.
Examiner respectfully disagrees. First, the claimed limitation does not stipulate that the positional data itself is used as an input to a second network. Rather, the input to the second network is based on the positional data.
In D1, a first trained algorithm is used to identify the anatomical structures and their position in the image. In particular, the initial algorithm “is capable to derive a segmentation mask indicating the area within the image for the respective identified structure. The segmentation mask itself or the respective coordinates of bounding boxes or other means allowing to describe the position of the anatomical structures within the image may be saved as input or metadata for subsequent processing” (see ¶ 52). The image may contain plurality of anatomical structures (see ¶ 51: a medical image of a patient's left knee joint, the derived image content metadata may include “left knee joint”, “left tibia”, “left femur”, and/or “left patella”. Or for an anteroposterior (AP) image of the lung, the visible image content may be “lung”, “thoracal spine”, “heart”, and/or “aorta”). The second or subsequent algorithm is selected based on the metadata, which includes the position information of the anatomical structures. The selected subsequent algorithm is specific to the anatomical structure that is segmented by the initial algorithm and performs processing on the segmented anatomical structure, which implies that the input to the second or subsequent algorithm is based on the positional information of the segmented structure. D1 describes the subsequent or second algorithms as such: “For example, a first algorithm may be suitable for segmenting the cervical spine, a second algorithm can be used to segment the lumbar spine, a third to recognize the heads of the femur, a fourth to place landmarks on the sacrum, a fifth to determine the anatomical designation of the segmented vertebrae, a sixth for determining bone density in a given image section, a seventh for classifying degenerative changes”. It is clear that the subsequent algorithms perform processing on the specific image section of the anatomical structure indicated by the initial algorithm, which is based on positional information. Stated differently, in order for the subsequent or second algorithm to perform processing on the specific anatomical structure among the plurality of anatomical structures segmented by the initial algorithm, either the segmentation mask output from the initial algorithm or the positional information of the anatomical structure must be input to the second or subsequent algorithm. In either case, the input to the subsequent or second algorithm is necessarily based on positional information of the segmentation mask or segmented anatomical structure. Otherwise, the second or subsequent algorithm would need to search the image again to identify where the specific anatomical structure is located, which is highly inefficient. Even if the Applicant is correct and the prior art does not anticipate “generating an input to a second trained classification network based on at least in part on the location predicted in (b)”, the claim maybe rejected under obviousness rejection: one skilled in the art would have found it obvious to input the location of the anatomical structure to the subsequent algorithm so that the subsequent or second algorithm can perform processing on the specific anatomical structure without having to search the entire image for the specific anatomical structure that was identified by the initial algorithm.
Moreover, Applicant’s argument that the positional data is only used to select a subsequent or second algorithm does not appear to be a convincing reading of D1. It is not clear how positional information alone provides any useful indication of what algorithm to select.
Applicant respectfully submits that the prior art does not teach “predicting a degenerative condition for the one or more structures by processing the input using the second [[deep]]trained classification network”. Applicant argues that
“D1 states that "a seventh [algorithm may be suitable] for classifying degenerative changes," listed among "a sixth for determining bone density in a given image section, . . . an eighth for determining the extent of carcinogenic tissue, a ninth for counting metastases, etc." See Dl at 57. Dl provides no description of the architecture of this seventh algorithm, no indication whether it is a trained model or a rule-based or geometric computation, and no description of any input it receives. Indeed, the list in which it appears includes methods that are not trained networks at all; D1 elsewhere identifies candidate algorithm types as including "trigonometry, vector algebra, optimization functions." A bare listing of a function ("classifying degenerative changes") does not disclose a network that processes a location-derived input to perform that function, as limitations (c) and (d) require.
To the extent the Office reads Dl's image analyzer (110) as executing the seventh algorithm using deep learning, D1 does not support that reading. D1's description of what its deep networks do is expressly limited: "deep learning or artificial neural network algorithms (e.g. convolutional neural networks (CNNs), recurrent neural networks (RNNs), long-short term memories (LSTMs)) may segment image areas or place landmarks or markers." See D1 at 60. Degenerative classification is absent from that enumeration.”
Examiner respectfully disagrees. It is clear that the algorithms for the listed tasks (classifying degenerative change, determining bone density, counting metastases etc.) can be performed by trained algorithms or rule based or geometric computation algorithms. A person having ordinary skill in the art would immediately envision a trained machine learning algorithm for predicting degenerative conditions. Under Federal Circuit precedent (such as In re Petering), when a reference discloses small, finite lists of interchangeable elements, the reference can anticipate specific combinations from those lists without inventive picking and choosing.
With regard to claims 5 and 16, which depend from claims 3 and 14 requiring a dual regulation module, Applicant argues that the rejection must be withdrawn because the prior art does not teach a dual regulation module and traverses the Official Notice taken with regard to the dual regulation module recited in claims 3-4 and 14-15. Examiner herewith provides documentary evidence. Arbel et al. (US Publication No. 2020/0327671) teach a dual regression deep neural network for segmentation (see ¶¶ 85, 109).
With regard to claims 6 and 17, Applicant respectfully submits that the prior art does not teach patches from the medical image based on the predicted location for the reasons set forth in part A regarding input to a second network. Examiner respectfully disagrees for the reasons set forth above in response to part A. Specifically, D1 explicitly teach detecting “segmentation mask itself or the respective coordinates of bounding boxes or other means allowing to describe the position of the anatomical structures within the image may be saved as input or metadata for subsequent processing”. The bounding boxes or segmentation masks may be read as patches, which serve as input for subsequent image processing, as explained above.
Applicant respectfully submits that D1 does not teach the features of claims 7-11 and 18-20 for the reasons set forth in part A, which is that D1 does not disclose the second network that receives as an input generated from the predicted location and predicts a degenerative condition. Examiner respectfully disagrees for the reasons set forth above.
With regard to claims 7 and 18, Applicant respectfully traverses the official notice that the claimed attention-map as input to a second network is well known in the art. Examiner herewith provides documentary evidence. Liu et al. (US Publication No. 2021/0089807) teach generating an attention map from a segmentation map (see ¶ 25).
Applicant respectfully traverses the Official Notice taken by the Examiner with regard to claims 8-11 and 19-20. Examiner herewith provides documentary evidence. Wang et al. (US Publication No. 2023/0177390) teach inputting features from multiple image views into a machine learning model for classification, as stipulated by claim 8. Liu et al. (US Publication No. 2021/0089807) teach a trained network comprising a plurality of branches (see figs. 6, 7, 9, ¶¶ 12, 93) as stipulated by claim 9. Liu et al. also teach input comprises wherein the input comprises patches of at least two different sizes and wherein at least two of the plurality of branches are configured to process the patches of at least two different sizes respectively (see figs. 5-7, ¶ 101: different window sizes) as stipulated by claim 10. Wang et al. (US Publication No. 2023/0177390) teach inputting features from multiple image views into a machine learning model for classification, as stipulated by claim 11.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-2, 6, 12-13, 17 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by D1.1
With regard to claim 1, D1 teach computer-implemented method for spine segmentation and classification, the method comprising:(a) receiving a medical image of a subject, wherein the medical image captures one or more structures of the subject (see ¶ 52, fig. 6, 8: receiving image); (b) applying a first deep network to the medical image and outputting a detection result, wherein the detection result comprises at least a segmentation map of the one or more structures and a location predicted for the one or more structures (see ¶¶ 52, 60: deep learning; segmentation mask and coordinates of structures); (c) generating an input to a second deep network based at least in part on the location predicted in (b) (see ¶ 52: position data input for subsequent processing; ¶¶ 54, 57, 60: deep neural network); and (d) predicting a degenerative condition for the one or more structures by processing the input using the second deep network (see ¶ 57: classifying degenerative changes).
With regard to claim 2, D1 teach computer-implemented method of claim 1, wherein the medical image includes a magnetic resonance image and the one or more structures comprise one or more spine structures (see ¶¶ 42, 57, 59, 65-66: vertebra).
With regard to claim 6, D1 teach computer-implemented method of claim 1, wherein the input to the second deep network comprises one or more patches generated from the medical image based at least in part on the location predicted in (b) (see ¶¶ 52, 54, 57, 60: location of anatomical structure or segmentation mask input for further processing, bounding box read as patch).
With regard to claim 12, see discussion of claim 1.
With regard to claim 13, see discussion of claim 2.
With regard to claim 17, see discussion of claim 6.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 3-5, 7-11, 14-20 are rejected under 35 U.S.C. 103 as being unpatentable over D1.
With regard to claim 3, D1 teach computer-implemented method of claim 1, but fails to explicitly teach wherein the first deep network comprises a segmentation model with a dual regulation module. However, Examiner takes Official Notice to the fact that dual regression regularization models are extremely well known in the art before the effective filing date and one skilled in the art would have been motivated to incorporate the dual regression model in the deep learning network for enhanced feature discrimination, stability and robustness with low computational overhead.
With regard to claim 4, D1 teach computer-implemented method of claim 3, wherein the see ¶ 52: predict coordinates of structures). Examiner takes Official Notice to the fact that dual regression regularization models are extremely well known in the art before the effective filing date and one skilled in the art would have been motivated to incorporate the dual regression model in the deep learning network for enhanced feature discrimination, stability and robustness with low computational overhead.
With regard to claim 7, D1 teach computer-implemented method of claim 1, wherein the input to the second deep network see ¶¶ 52, 54, 57, 60: segmentation mask). D1 teaches generating a segmentation mask, but fails to explicitly teach determining attention map. However, Examiner takes Official Notice to the fact that generating attention maps is extremely well known in the art before the effective filing date and that it would have been obvious to incorporate known teachings in to the configuration of D1 yielding predictable results. The motivation for calculating attention map would have been to differentiate between more important regions from less relevant regions.
With regard to claim 5, D1 teach computer-implemented method of claim 3, wherein the segmentation model is trained to predict the segmentation map of the one or more structures (see ¶¶ 52, 57: segmentation mask).
With regard to claim 8, D1 fails to explicitly teach computer-implemented method of claim 1, wherein the input to the second deep network comprises at least a second medical image of a view that is different from the first medical image. However, Examiner takes Official Notice to the fact that analyzing images with different views is extremely well known in the art before the effective filing date and one skilled in the art would have been motivated to incorporate known teachings into the configuration of D1 yielding predictable and enhanced results because images from different views provide additional information on the structures that are being analyzed.
With regard to claim 9, D11 fails to explicitly teach computer-implemented method of claim 1, wherein the second deep network comprises a plurality of branches. However, Examiner takes Official Notice to the fact that deep learning networks with plurality of branches or channels is extremely well known in the art before the effective filing date and that one skilled in the art would have been motivated to incorporate known teachings into the configuration of D1 yielding predictable and enhanced results by having different branches specialize in extracting different types of features, specialized functionality etc.
With regard to claim 10, D1 fails to explicitly teach computer-implemented method of claim 9, wherein the input comprises patches of at least two different sizes and wherein at least two of the plurality of branches are configured to process the patches of at least two different sizes respectively. However, Examiner takes Official Notice to the fact that multi-scale feature extraction via multiple channels or branches is extremely well known in the art before the effective filing date and would have been particularly obvious to incorporate known teachings into the configuration of D1 yielding predictable and enhanced multi-scale features for better discrimination of features.
With regard to claim 11, D1 fails to explicitly teach computer-implemented method of claim 9, wherein the input comprises patches of at least two different views and wherein at least two of the plurality of branches are configured to process the patches of at least two different views respectively. However, Examiner takes Official Notice to the fact that analyzing images with different views is extremely well known in the art before the effective filing date and one skilled in the art would have been motivated to incorporate known teachings into the configuration of D1 yielding predictable and enhanced results because images from different views provide additional information on the structures that are being analyzed.
With regard to claim 14, see discussion of claim 3.
With regard to claim 15, see discussion of claim 4.
With regard to claim 16, see discussion of claim 5.
With regard to claim 18, see discussion of claim 7.
With regard to claim 19, see discussion of claim 8.
With regard to claim 20, see discussion of claim 9.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AVINASH YENTRAPATI whose telephone number is (571)270-7982. The examiner can normally be reached on 8AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached on (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AVINASH YENTRAPATI/Primary Examiner, Art Unit 2672
1 US Publication No. 2021/0174503.
