DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 03/19/2026 has been entered.
Response to Arguments
Applicant’s arguments, see Remarks pages 12-13, filed 03/19/2026, with respect to the SuperGuide Interpretations of claims 18 and 20 have been fully considered and are persuasive. The interpretations of claims 18 and 20 have been withdrawn.
Applicant’s arguments, see Remarks page 13, filed 03/19/2026, with respect to the rejections of claim(s) amended claims 1 and 16-17 under 35 U.S.C. 102(a)(1) have been fully considered and are moot in view of the new grounds of rejection (detailed in the rejections below) necessitated by Applicant’s amendment to the claim(s).
Claim Objections
Claim 1 is objected to because of the following informalities:
Regarding claim 1, the amended limitation “acquire build a learning model” should be corrected to “build a learning model”.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 10, 12, 16-17, 19, and 21-22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wenzel et al. (US 2020/0279371 A1) hereinafter referenced as Wenzel, in view of Han et al. (WO 2016/036516 A1) hereinafter referenced as Han.
Regarding claim 1, Wenzel discloses: An image processing apparatus (Wenzel: Abstract) comprising: at least one memory storing a program; and at least one processor which, by executing the program, causes the image processing apparatus (Wenzel: 0031) to:
acquire data including a three-dimensional image obtained by capturing an image of a subject under examination (Wenzel: 0058-0059: “FIG. 2 is a flowchart of a method for locating anatomical landmarks of a predefined anatomy (or anatomical structure) using a scanning imaging system e.g. an MRI scanning imaging system…a current set of acquisition parameters may be determined. The current set of parameters may for example be determined by the control system 111. For example, the set of acquisition parameters may comprise at least one of: number of voxels in X- and Y-direction, the voxel size in mm, the center of voxel (0,0) in DICOM patient coordinates using for example DICOM Tag “ImagePositionPatient”, and 3D orientation of the slice in DICOM patient coordinates using for example DICOM Tag: “ImageOrientationPatient”. DICOM stands for Digital Imaging and Communications in Medicine.”), a standard plane parameter representing a predetermined standard plane in the three-dimensional image (Wenzel: 0024: “the execution of the learning algorithm comprises determining from the training set image data representing a slice corresponding to a given set of parameters, and executing the learning algorithm using the slice.”;
0059-0060: “In step 201, a current set of acquisition parameters may be determined. The current set of parameters may for example be determined by the control system 111. For example, the set of acquisition parameters may comprise at least one of: number of voxels in X- and Y-direction, the voxel size in mm, the center of voxel (0,0) in DICOM patient coordinates using for example DICOM Tag "ImagePositionPatient", and 3D orientation of the slice in DICOM patient coordinates using for example DICOM Tag: "ImageOrientationPatient”; Wherein the initial acquisition parameters for the survey image data represent the standard plane parameters), and an anatomical landmark position in the three-dimensional image (Wenzel: Abstract: “The present disclosure relates to a method for medical imaging method for locating anatomical landmarks of a predetermining defined anatomy. The method comprises…d) identifying anatomical landmarks in the received image data using the machine learning model; e) predicting another set of acquisition parameters using the machine learning model and repeating steps c )-e) for a predefined number of repetitions using the predicted set of 10 acquisition parameters as the current set of parameters; and f) providing the identified anatomical landmarks”);
acquire, on the basis of a vicinity cross section parameter representing a vicinity cross section obtained by moving the predetermined standard plane to the outside of the predetermined standard plane on the basis of the standard plane parameter and the three-dimensional image, a vicinity cross section image (Wenzel: 0025: “…reinforcement training of a deep learning system may be done by applying multi-planar reformatting (MPR) to the 3D brain scan in order to extract a slice matching desired scan parameters (e.g. size, orientation, origin etc.), simulating their acquisition on a scanner. Output variables of the deep learning network are the estimated positions of the anatomical landmarks as well as their confidence level, along with parameters for the next slice which may again be simulated by applying MPR on the dataset.”;
0062: “In case (inquiry step 207 ) a predefined number of repetitions is reached (or in case a stopping criteria is fulfilled) the identified anatomical landmarks may be provided in step 211; otherwise another set of acquisition parameters may be predicted in step 209 using the machine learning model; and steps 203 )- 211 ) may be repeated using the predicted set of acquisition parameters as the current set of parameters.”; Wherein the image slice parameters output by the deep learning model, used to extract subsequent image slices constitute vicinity cross section parameters.);
acquire the anatomical landmark position on the vicinity cross section image by projecting an anatomical landmark position in the predetermined standard plane onto the vicinity cross section on the basis of the anatomical landmark position and the relationship between the predetermined standard plane and the vicinity cross section (Wenzel: 0016: “According to the invention the (survey-) image from which the anatomical landmarks are derived is being built-up during the landmark identification. In particular, a survey slice image is received from which landmarks may be detected. In the next iteration acquisition parameters are updated to generate a next survey image from which further anatomical landmarks are identified. The update may be dependent on the anatomical landmarks already identified from previous iterations…The identification of (additional) anatomical landmarks from the current survey image slice is carried out by the machine learning model that is inspired to set the next survey slice image acquisition from anatomical landmarks already identified and uses trained rules that may predict further anatomical landmarks form the already identified ones.”; Wherein the landmark positions of the vicinity, or subsequent, cross-sections are acquired based on the identified landmarks on the survey image and its relationship to the survey images); and
acquire build a learning model, which is for estimating an anatomical landmark position on the cross section image by inputting the cross section image, wherein the learning model is built by using, as performing augmentation of learning data, information acquired by the image processing apparatus including a pair of the vicinity cross section image and the anatomical landmark position on the vicinity cross section image (Wenzel: 0024: “For example, arbitrary images for any set of parameters can be extracted from the training scans by planar reformatting with their corresponding ground truth information ( e.g. the position of anatomical landmarks, as manually marked by a technician, or as determined by automatic approaches such as SmartExam) …reinforcement training of a deep learning system may be done by applying multi-planar reformatting (MPR) to the 3D brain scan in order to extract a slice matching desired scan parameters (e.g. size, orientation, origin etc.), simulating their acquisition on a scanner. Output variables of the deep learning network are the estimated positions of the anatomical landmarks as well as their confidence level, along with parameters for the next slice which may again be simulated by applying MPR on the dataset.”; Wherein parameters for survey image slices, including initial acquisition parameters, are iteratively calculated to extract an image from a 3D scan to be input into a model to train it to estimate positions of landmarks on the slices, and wherein the extraction of subsequent image slices from a 3D scan by applying MPR constitutes learning data augmentation.).
Wenzel does not disclose expressly: wherein the learning model is built by using, as performing augmentation of learning data, information acquired by the image processing apparatus including a pair of the vicinity cross section image and the anatomical landmark position on the vicinity cross section image obtained by projecting the anatomical landmark position in the predetermined standard plane onto the vicinity cross section on the basis of the anatomical landmark position and the relationship between the predetermined standard plane and the vicinity cross section.
Han discloses: a learning model built by using information acquired by an image processing apparatus including a pair of a vicinity cross section image (Han: 042: “anatomical classifier training unit 102 may communicate with training image database 101 to receive one or more training images…the training images may include MRI images, 3D MRI, 2D streaming MRI, 4D volumetric MRI, CT images, CBCT images, Spiral CT images, PET images, functional MRI images (e.g., fMRI, DCE-MRI and diffusion MRI), X-ray images, fluoroscopic images, ultrasound images, radiotherapy portal images, SPECT images, and the like.”) and an anatomical landmark position on the vicinity cross section image obtained by projecting the anatomical landmark position in a predetermined standard plane onto the vicinity cross section on the basis of the anatomical landmark position and the relationship between the predetermined standard plane and the vicinity cross section (Han: 063-064: “In Step 420, landmark feature extraction module 121 may identify at least one landmark point in the training image.…These landmark points can either be manually picked by user 112 and provided to landmark feature extraction module 121, or automatically detected by landmark feature extraction module 121. In some embodiments, automatic detection may be performed through, e.g., image registration between the training image and a predetermined landmark map of the same anatomical region. For example, a predetermined landmark map may define a particular mapping or spacing of various landmark points for a particular anatomical region. If user 112 is processing medical images showing a particular anatomical region of a patient, then user 112 could thus select a predetermined landmark map that corresponds to that anatomical region. Upon selection, the landmark points defined by the predetermined map may then be provided to module 121.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the labeling of anatomical landmarks in a training dataset disclosed by Wenzel with the landmark feature extraction module for automatic landmark point determination taught by Han. The suggestion/motivation for doing so would have been “landmark points may be selected or identified by user 112. For example, user 112 may click on the training image displayed by an image display (not shown) to input the location of the landmark points or user may 1 12 may select a predetermined pattern, map, or configuration of landmark points.” (Han: 044; Wherein the landmark feature extraction module allows for the user to modify the labeling of the images based on desired landmark points.). Therefore, it would have been obvious to combine Wenzel with Han to obtain the invention as specified in claim 1.
Regarding claim 2, Wenzel in view of Han discloses: The image processing apparatus according to claim 1, wherein the at least one processor causes the image processing apparatus to: acquire, on the basis of the standard plane parameter, a plurality of the vicinity cross section images defining a plurality of the cross sections in the vicinity of the predetermined standard plane; and acquire the anatomical landmark position in each of the plurality of vicinity cross section images. (Wenzel: 0016: “a survey slice image is received from which landmarks may be detected. In the next iteration acquisition parameters are updated to generate a next survey image from which further anatomical landmarks are identified. The update may be dependent on the anatomical landmarks already identified from previous iterations. This process may continue for a pre-determined number of iterations, which may be defined by a pre-set maximum number of iterations, or the number of iterations that achieves to find a pre-set maximum number of landmarks identified.”; Wherein survey images determined within the same 3D volume constitute being in the same vicinity, wherein the initial survey image constitutes the standard plane image.).
Regarding claim 3, Wenzel in view of Han discloses: The image processing apparatus according to claim 1, wherein the vicinity cross section image is a cross section image defining a cross section outside the predetermined standard plane (Wenzel: 0059: “In step 201, a current set of acquisition parameters may be determined. The current set of parameters may for example be determined by the control system 111 . For example, the set of acquisition parameters may comprise at least one of: number of voxels in X- and Y-direction, the voxel size in mm, the center of voxel (0,0) in DICOM patient coordinates using for example DICOM Tag “ImagePositionPatient”, and 3D orientation of the slice in DICOM patient coordinates using for example DICOM Tag: “ImageOrientationPatient”. DICOM stands for Digital Imaging and Communications in Medicine.”;
0062-0063: “In case (inquiry step 207) a predefined number of repetitions is reached (or in case a stopping criteria is fulfilled) the identified anatomical landmarks may be provided in step 211; otherwise another set of acquisition parameters may be predicted in step 209 using the machine learning model.”).
Regarding claim 4, Wenzel in view of Han discloses: The image processing apparatus according to claim 1, wherein the at least one processor causes the image processing apparatus to: project the anatomical landmark position acquired by the image processing apparatus on the vicinity cross section image to acquire the anatomical landmark position on the vicinity cross section image (Wenzel: 0061: “In step 205 , anatomical landmarks may be identified in the acquired image data using the machine learning model. For example, a confidence level may be assigned to the identified anatomical landmarks. For example, the acquired image data may be input to the machine learning model. The identified landmarks may for example correspond to at least part of the set of landmarks that have been used for the generation of the machine learning model.”).
Regarding claim 10, Wenzel in view of Han discloses: The image processing apparatus according to claim 1, wherein the at least one processor further causes the image processing apparatus to: acquire an input three-dimensional image and an input standard plane parameter representing a predetermined standard plane on the input three-dimensional image (Wenzel: 0058-0059: “FIG. 2 is a flowchart of a method for locating anatomical landmarks of a predefined anatomy (or anatomical structure) using a scanning imaging system e.g. an MRI scanning imaging system. The anatomy may be part of a subject (e.g. 318 of FIG. 3) to be imaged. The anatomy may for example be a heart, brain, knee, spine, shoulder, breast, etc. The imaging of the anatomy may result in images that can be used to perform further actions such as treatment delivery. In step 201, a current set of acquisition parameters may be determined. The current set of parameters may for example be determined by the control system 111.”);
acquire an input cross section image based on each of the input three-dimensional image and the input standard plane parameter (Wenzel: 0059-0060: “”the set of acquisition parameters may comprise at least one of: number of voxels in X- and Y-direction, the voxel size in mm, the center of voxel (0,0) in DICOM patient coordinates using for example DICOM Tag "ImagePositionPatient", and 3D orientation of the slice in DICOM patient coordinates using for example DICOM Tag: "ImageOrientationPatient". DICOM stands for Digital Imaging and Communications in Medicine. In step 203, survey image data representing a slice of the anatomy may be received e.g. at the control system 111. For example, the receiving of the survey image data may automatically be performed.); and
estimate an anatomical landmark position on the input cross section image by using the learning model acquired by the image processing apparatus (Wenzel: 0061: “In step 205, anatomical landmarks may be identified in the acquired image data using the machine learning model. For example, a confidence level may be assigned to the identified anatomical landmarks.”).
Regarding claim 12, Wenzel in view of Han discloses: The image processing apparatus according to claim 19 (Disclosed in 35 U.S.C. 103 rejection below), wherein the at least one processor causes the image processing apparatus to: specify an anatomical landmark position on the vicinity cross section image corresponding to an anatomical landmark position on the standard plane image of the predetermined standard plane on the basis of a correspondence relationship between a pixel on the standard plane image of the predetermined standard plane and a pixel on the vicinity cross section image to acquire the anatomical landmark position on the vicinity cross section image (Wenzel: 0055: “The AI component 150 may be configured for a robust and fast detecting of anatomical landmarks by adaptive acquisition. The AI component 150 as further described herein may be configured to dynamically adapt/drive the acquisition process of the survey scan based on the partial anatomical information already available, and therefore reducing the number of required images and, consequently, overall scan time for localizing anatomical landmarks required for fully automated scan planning.”;
0057: “The AI component 150 may be configured to perform machine learning on training sets in order to generate one or more machine learning models for predicting anatomical landmarks in image data obtained using a set of acquisition parameters and for predicting a subsequent set of acquisition parameters of the set of acquisition parameters for subsequent acquiring of image data.”; Wherein the usage of the initial survey image and landmarks positions estimated by the model for the determination of subsequent, or vicinity cross section, images used constitutes on the basis of pixel relationships).
As per claim(s) 16, arguments made in rejecting claim(s) 1 are analogous.
As per claim(s) 17, arguments made in rejecting claim(s) 1 are analogous. In addition, 0070 of Wenzel discloses: A non-transitory computer readable medium that stores a program, wherein the program causes a computer to execute the steps.
Regarding claim 19, Wenzel in view of Han discloses: The image processing apparatus according to claim 1, wherein the at least one processor causes the image processing apparatus to: acquire, on the basis of the three-dimensional image and the standard plane parameter, a standard plane image (Wenzel: 0058-0060: “FIG. 2 is a flowchart of a method for locating anatomical landmarks of a predefined anatomy (or anatomical structure) using a scanning imaging system e.g. an MRI scanning imaging system. The anatomy may be part of a subject (e.g. 318 of FIG. 3) to be imaged. The anatomy may for example be a heart, brain, knee, spine, shoulder, breast, etc. The imaging of the anatomy may result in images that can be used to perform further actions such as treatment delivery. For example, the set of acquisition parameters may comprise at least one of: number of voxels in X- and Y-direction, the voxel size in mm, the center of voxel (0,0) in DICOM patient coordinates using for example DICOM Tag "ImagePositionPatient", and 3D orientation of the slice in DICOM patient coordinates using for example DICOM Tag: "ImageOrientationPatient”… In step 203, survey image data representing a slice of the anatomy may be received e.g. at the control system 111.; Wherein the survey image data for the anatomical structure is acquired based on the 3d acquisition parameters).
Regarding claim 21, Wenzel in view of Han discloses: The image processing apparatus according to claim 19, wherein the at least one processor causes the image processing apparatus to: acquire build the learning model built by using, as learning data, information including (i) a pair of the standard plane image and the anatomical landmark position in the predetermined standard plane (Han: 064: “a predetermined landmark map may define a particular mapping or spacing of various landmark points for a particular anatomical region. If user 112 is processing medical images showing a particular anatomical region of a patient, then user 112 could thus select a predetermined landmark map that corresponds to that anatomical region. Upon selection, the landmark points defined by the predetermined map may then be provided to module 121.”), and (ii) the pair of the vicinity cross section image and the anatomical landmark position on the vicinity cross section image (Han: 063-064: “In Step 420, landmark feature extraction module 121 may identify at least one landmark point in the training image.…These landmark points can either be manually picked by user 112 and provided to landmark feature extraction module 121, or automatically detected by landmark feature extraction module 121. In some embodiments, automatic detection may be performed through, e.g., image registration between the training image and a predetermined landmark map of the same anatomical region.”).
Regarding claim 22, Wenzel discloses: An image processing apparatus (Wenzel: Abstract) comprising: at least one memory storing a program; and at least one processor which, by executing the program, causes the image processing apparatus (Wenzel: 0031) to:
acquire an input three-dimensional image and an input standard plane parameter representing a predetermined standard plane in the input three-dimensional image (Wenzel: 0058-0059: “FIG. 2 is a flowchart of a method for locating anatomical landmarks of a predefined anatomy (or anatomical structure) using a scanning imaging system e.g. an MRI scanning imaging system…a current set of acquisition parameters may be determined. The current set of parameters may for example be determined by the control system 111 . For example, the set of acquisition parameters may comprise at least one of: number of voxels in X- and Y-direction, the voxel size in mm, the center of voxel ( 0 , 0 ) in DICOM patient coordinates using for example DICOM Tag “ImagePositionPatient”, and 3D orientation of the slice in DICOM patient coordinates using for example DICOM Tag: “ImageOrientationPatient”. DICOM stands for Digital Imaging and Communications in Medicine.”);
acquire an input cross-section image based on the input three-dimensional image and the
input standard plane parameter (Wenzel: 0024: “the execution of the learning algorithm comprises determining from the training set image data representing a slice corresponding to a given set of parameters, and executing the learning algorithm using the slice.”;
0059-0060: “In step 201, a current set of acquisition parameters may be determined. The current set of parameters may for example be determined by the control system 111. For example, the set of acquisition parameters may comprise at least one of: number of voxels in X- and Y-direction, the voxel size in mm, the center of voxel (0,0) in DICOM patient coordinates using for example DICOM Tag "ImagePositionPatient", and 3D orientation of the slice in DICOM patient coordinates using for example DICOM Tag: "ImageOrientationPatient”; Wherein the initial acquisition parameters for the survey image data represent the standard plane parameters);
acquire a trained learning model configured to estimate an anatomical landmark position
on a cross-section image from the cross-section image; and estimate an anatomical landmark position on the input cross-section image by inputting the input cross-section image to the trained learning model (Wenzel: 0016: “According to the invention the (survey-) image from which the anatomical landmarks are derived is being built-up during the landmark identification. In particular, a survey slice image is received from which landmarks may be detected. In the next iteration acquisition parameters are updated to generate a next survey image from which further anatomical landmarks are identified. The update may be dependent on the anatomical landmarks already identified from previous iterations…The identification of (additional) anatomical landmarks from the current survey image slice is carried out by the machine learning model that is inspired to set the next survey slice image acquisition from anatomical landmarks already identified and uses trained rules that may predict further anatomical landmarks form the already identified ones.”),
wherein the trained learning model is trained using, as training data, a plurality of pairs each including: (a) a vicinity cross-section image obtained based on a three-dimensional image and a vicinity cross-section parameter representing a vicinity cross-section obtained by moving the predetermined standard plane to the outside of the predetermined standard plane; and (b) an anatomical landmark position on the vicinity cross-section image (Wenzel: 0024: “For example, arbitrary images for any set of parameters can be extracted from the training scans by planar reformatting with their corresponding ground truth information ( e.g. the position of anatomical landmarks, as manually marked by a technician, or as determined by automatic approaches such as SmartExam) …reinforcement training of a deep learning system may be done by applying multi-planar reformatting (MPR) to the 3D brain scan in order to extract a slice matching desired scan parameters (e.g. size, orientation, origin etc.), simulating their acquisition on a scanner. Output variables of the deep learning network are the estimated positions of the anatomical landmarks as well as their confidence level, along with parameters for the next slice which may again be simulated by applying MPR on the dataset.”; Wherein parameters for survey image slices, including initial acquisition parameters, are iteratively calculated to extract an image from a 3D scan to be input into a model to train it to estimate positions of landmarks on the slices, and wherein the extraction of subsequent image slices from a 3D scan by applying MPR constitutes learning data augmentation.).
Wenzel does not disclose expressly: wherein the trained learning model is trained using, as training data, a plurality of pairs each including: (b) an anatomical landmark position on the vicinity cross-section image obtained by projecting an anatomical landmark position on an image of the predetermined standard plane onto the vicinity cross-section based on a relationship between the predetermined standard plane and the vicinity cross-section.
Han discloses: a learning model built by using information acquired by an image processing apparatus including a pair of a vicinity cross section image (Han: 042: “anatomical classifier training unit 102 may communicate with training image database 101 to receive one or more training images…the training images may include MRI images, 3D MRI, 2D streaming MRI, 4D volumetric MRI, CT images, CBCT images, Spiral CT images, PET images, functional MRI images (e.g., fMRI, DCE-MRI and diffusion MRI), X-ray images, fluoroscopic images, ultrasound images, radiotherapy portal images, SPECT images, and the like.”) and an anatomical landmark position on the vicinity cross section image obtained by projecting the anatomical landmark position in a predetermined standard plane onto the vicinity cross section on the basis of the anatomical landmark position and the relationship between the predetermined standard plane and the vicinity cross section (Han: 063-064: “In Step 420, landmark feature extraction module 121 may identify at least one landmark point in the training image.…These landmark points can either be manually picked by user 112 and provided to landmark feature extraction module 121, or automatically detected by landmark feature extraction module 121. In some embodiments, automatic detection may be performed through, e.g., image registration between the training image and a predetermined landmark map of the same anatomical region. For example, a predetermined landmark map may define a particular mapping or spacing of various landmark points for a particular anatomical region. If user 112 is processing medical images showing a particular anatomical region of a patient, then user 112 could thus select a predetermined landmark map that corresponds to that anatomical region. Upon selection, the landmark points defined by the predetermined map may then be provided to module 121.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the labeling of anatomical landmarks in a training dataset disclosed by Wenzel with the landmark feature extraction module for automatic landmark point determination taught by Han. The suggestion/motivation for doing so would have been “landmark points may be selected or identified by user 112. For example, user 112 may click on the training image displayed by an image display (not shown) to input the location of the landmark points or user may 1 12 may select a predetermined pattern, map, or configuration of landmark points.” (Han: 044; Wherein the landmark feature extraction module allows for the user to modify the labeling of the images based on desired landmark points.). Therefore, it would have been obvious to combine Wenzel with Han to obtain the invention as specified in claim 22.
Claim(s) 5-9, 11, 18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Wenzel in view of Han, and further in view of Liu et al. (WO2021239141A1) hereinafter referenced as Liu.
Regarding claim 5, Wenzel in view of Han discloses: The image processing apparatus according to claim 1, wherein the at least one processor causes the image processing apparatus to: acquire a plurality of the vicinity cross section images by causing at least one movement of the predetermined standard plane (Wenzel: 0025-0026: “reinforcement training of a deep learning system may be done by applying multi-planar reformatting (MPR) to the 3D brain scan in order to extract a slice matching desired scan parameters (e.g. size, orientation, origin etc.), simulating their acquisition on a scanner. Output variables of the deep learning network are the estimated positions of the anatomical landmarks as well as their confidence level, along with parameters for the next slice which may again be simulated by applying MPR on the dataset. According to one embodiment, the set of acquisition parameters comprises at least one of the following types: an indication of a slice of the anatomy; voxel size of the image data; number of voxels in the image data; the center of voxel in the image data; the 3D orientation of the slice.”; Wherein the calculation of a survey image based on the results of an input image constitutes the movement of the survey image to a different position); and
approximate the anatomical landmark position acquired by the image processing apparatus on the basis of the at least one movement to a position on the vicinity cross section image to thereby acquire the anatomical landmark position on each of the plurality of vicinity cross section images (Wenzel: 0055: “The AI component 150 as further described herein may be configured to dynamically adapt/drive the acquisition process of the survey scan based on the partial anatomical information already available, and therefore reducing the number of required images and, consequently, overall scan time for localizing anatomical”; Wherein the localization of anatomical landmarks based on subsequent survey images determined by the results of the previous images constitutes the landmark estimation based on movement).
Wenzel in view of Han does not disclose expressly: acquire a plurality of the vicinity cross section images by causing at least one movement which is either parallel movement or rotational movement of the predetermined standard plane.
Liu discloses: the acquisition of a plurality of images to be processed by causing parallel movement and rotational movement to an initial image (Liu: 0089-0091: “the plurality of images to be processed may include information of an initial image. For example, the multiple images to be processed may include an initial image or a portion of the initial image. For another example, the initial image may be directly used as one of the multiple images to be processed…In some embodiments, the plurality of images to be processed may include regions of interest at multiple angles. Accordingly, the plurality of images to be processed may also be understood as corresponding to a three-dimensional image… The processing device 120 may select a plurality of continuous images that are adjacent to the initial image, for example, in space and/or time, from the image sequence as the plurality of images to be processed.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the acquisition/selection of survey images using the modification of scan parameters disclosed by Wenzel in view of Han with the methods for the acquisition of images to be processed by using rotation and translation of an initial image disclosed by Liu. The suggestion/motivation for doing so would have been “For example, the regions of interest included in the at least two additional images may be the same as the region of interest included in the initial image” (Liu: 0079; Wherein by selecting images based on direct movement of the initial image within a 3d volume, the region of interest may be preserved). Therefore, it would have been obvious to combine Wenzel in view of Han with Liu to obtain the invention as specified in claim 5.
Regarding claim 6, Wenzel in view of Han and Liu discloses: The image processing apparatus according to claim 5, wherein the parallel movement is parallel movement to the outside of the predetermined standard plane (Liu: 0091: “The processing device 120 may select a plurality of continuous images that are adjacent to the initial image, for example, in space and/or time, from the image sequence as the plurality of images to be processed.”; 0135: “This embodiment selects multiple two-dimensional images adjacent to a two-dimensional image from a medical image sequence, and constructs a corresponding first image with three dimensions based on the multiple two-dimensional images to add additional feature information (such as texture information, gradient information, and grayscale information of the structure).”; Wherein the acquisition of multiple spatially adjacent images constitutes the parallel movement of the initial image), while the rotational movement is rotational movement around a predetermined axis extending over the predetermined standard plane (Liu: 0087: “The processing device 120 may select an image in the image sequence as the initial image…a region of interest (such as a lesion) of the object may be determined based on the image generated by the scan. If the region of interest is not at the reconstruction center, the reconstruction parameters can be adjusted to make it located at the center, and a new scan can be performed to obtain a new image”; 0090: “the plurality of images to be processed may include regions of interest at multiple angles. Accordingly, the plurality of images to be processed may also be understood as corresponding to a three-dimensional image.”; Wherein the initial image containing the region of interest at the center and selecting images based on the region of interest at multiple angles constitutes the rotational movement around an axis.).
Regarding claim 7, Wenzel in view of Han and Liu discloses: The image processing apparatus according to claim 6, wherein the predetermined axis is an axis passing through the at least one anatomical landmark in the predetermined standard plane (Liu: 0087: “The processing device 120 may select an image in the image sequence as the initial image…a region of interest (such as a lesion) of the object may be determined based on the image generated by the scan. If the region of interest is not at the reconstruction center, the reconstruction parameters can be adjusted to make it located at the center, and a new scan can be performed to obtain a new image”;
0090: “the plurality of images to be processed may include regions of interest at multiple angles. Accordingly, the plurality of images to be processed may also be understood as corresponding to a three-dimensional image.”; Wherein the region of interest constitutes an anatomical landmark.).
Regarding claim 8, Wenzel in view of Han and Liu discloses: The image processing apparatus according to claim 6, wherein the predetermined axis includes at least two or more axes passing through the predetermined standard plane, and wherein the at least one processor causes the image processing apparatus to select either or any one of the two or more axes for each rotational movement (Liu: 0090: “the plurality of images to be processed may include regions of interest at multiple angles. Accordingly, the plurality of images to be processed may also be understood as corresponding to a three-dimensional image.”).
Regarding claim 9, Wenzel in view of Han and Liu discloses: The image processing apparatus according to claim 5, wherein the at least one movement is movement at a predetermined interval (Liu: 0090: “The processing device 120 may select a plurality of continuous images that are adjacent to the initial image, for example, in space and/or time, from the image sequence as the plurality of images to be processed.”; Wherein the acquisition of spatially adjacent images constitutes a predetermined interval.).
Regarding claim 11, Wenzel in view of Han discloses: The image processing apparatus according to claim 1, wherein the predetermined standard plane includes at least one standard plane (Wenzel: 0059-0060: “In step 201, a current set of acquisition parameters may be determined. The current set of parameters may for example be determined by the control system 111. For example, the set of acquisition parameters may comprise at least one of: number of voxels in X- and Y-direction, the voxel size in mm, the center of voxel (0,0) in DICOM patient coordinates using for example DICOM Tag "ImagePositionPatient", and 3D orientation of the slice in DICOM patient coordinates using for example DICOM Tag: "ImageOrientationPatient”; Wherein the initial acquisition parameters for the survey image data represent the standard plane parameters), wherein the at least one processor further causes the image processing apparatus to:
assign the anatomical landmark position to the at least one standard plane, wherein the data acquired by the image processing apparatus includes the anatomical landmark position assigned by the image processing apparatus (Wenzel: Abstract: “The present disclosure relates to a method for medical imaging method for locating anatomical landmarks of a predetermining defined anatomy. The method comprises: a) providing a machine learning model for predicting anatomical landmarks in image data obtained using a set of acquisition parameters and for predicting a subsequent set of acquisition parameters of the set of acquisition parameters for subsequent acquisition of image data; b) determining 5 a current set of acquisition parameters; c) receiving survey image data representing a slice of the anatomy, the survey image data having the current set of current acquisition parameters; d) identifying anatomical landmarks in the received image data using the machine learning model”), and
wherein the at least one processor causes the image processing apparatus to: acquire the learning model by using, as the learning data, a pair of a cross section image representing a cross section in the vicinity of the predetermined standard plane and a position based on the anatomical landmark position assigned by the image processing apparatus (Han: 063-064: “In Step 420, landmark feature extraction module 121 may identify at least one landmark point in the training image.…These landmark points can either be manually picked by user 112 and provided to landmark feature extraction module 121, or automatically detected by landmark feature extraction module 121. In some embodiments, automatic detection may be performed through, e.g., image registration between the training image and a predetermined landmark map of the same anatomical region. For example, a predetermined landmark map may define a particular mapping or spacing of various landmark points for a particular anatomical region. If user 112 is processing medical images showing a particular anatomical region of a patient, then user 112 could thus select a predetermined landmark map that corresponds to that anatomical region. Upon selection, the landmark points defined by the predetermined map may then be provided to module 121.”).
Wenzel in view of Han does not disclose expressly: wherein the predetermined standard plane includes at least two or more standard planes.
Liu discloses: multiple sets of training samples each containing an initial image and a set of images to be processed generated from each initial image (Liu: 0096: “In some embodiments, a large number of training samples may be used to train an initial model to obtain a processed model. Each training sample may include an initial training image, multiple training images to be processed, and a target training image (ie, a gold standard). A large number of training samples can be input to the initial model in batches.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the known technique as disclosed by Liu of including multiple initial images into Wenzel in view of Han by training the model disclosed by Wenzel in view of Han using different 3D image volumes. The suggestion/motivation for doing so would have been “training samples corresponding to different organs may be used to train a processing model, and the processing model may be used to process different organs.” (Liu: 0097). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Wenzel in view of Han with Liu to obtain the invention as specified in claim 11.
Regarding claim 18, Wenzel in view of Han discloses: The image processing apparatus according to claim 1, wherein the at least one processor causes the image processing apparatus to: acquire the vicinity cross section by moving the predetermined standard plane to the outside the predetermined standard plane, and acquires the vicinity cross section image on the basis of the acquired vicinity cross section and the three-dimensional image (Wenzel: 0025: “…reinforcement training of a deep learning system may be done by applying multi-planar reformatting (MPR) to the 3D brain scan in order to extract a slice matching desired scan parameters (e.g. size, orientation, origin etc.), simulating their acquisition on a scanner. Output variables of the deep learning network are the estimated positions of the anatomical landmarks as well as their confidence level, along with parameters for the next slice which may again be simulated by applying MPR on the dataset.”; Wherein the image slice parameters output by the deep learning model, used to extract subsequent image slices from the 3D brain scan using MPR, constitute vicinity cross section acquired by moving the originally input survey image, outside the predetermined standard plane.).
Wenzel in view of Han does not disclose expressly: acquire the vicinity cross section by moving the predetermined standard plane to the outside the predetermined standard plane by at least one of rotational movement or parallel movement.
Liu discloses: the acquisition of a plurality of images to be processed by causing parallel movement and rotational movement to an initial image (Liu: 0089-0091: “the plurality of images to be processed may include information of an initial image. For example, the multiple images to be processed may include an initial image or a portion of the initial image. For another example, the initial image may be directly used as one of the multiple images to be processed…In some embodiments, the plurality of images to be processed may include regions of interest at multiple angles. Accordingly, the plurality of images to be processed may also be understood as corresponding to a three-dimensional image… The processing device 120 may select a plurality of continuous images that are adjacent to the initial image, for example, in space and/or time, from the image sequence as the plurality of images to be processed.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the acquisition/selection of survey images using the modification of scan parameters disclosed by Wenzel in view of Han with the methods for the acquisition of images to be processed by using rotation and translation of an initial image disclosed by Liu. The suggestion/motivation for doing so would have been “For example, the regions of interest included in the at least two additional images may be the same as the region of interest included in the initial image” (Liu: 0079; Wherein by selecting images based on direct movement of the initial image within a 3d volume, the region of interest may be preserved). Therefore, it would have been obvious to combine Wenzel in view of Han with Liu to obtain the invention as specified in claim 18.
Regarding claim 20, Wenzel in view of Han discloses: The image processing apparatus according to claim 19, wherein the at least one processor causes the image processing apparatus to: project the anatomical landmark position from the predetermined standard plane to the vicinity cross section (Wenzel: 0016: “According to the invention the (survey-) image from which the anatomical landmarks are derived is being built-up during the landmark identification. In particular, a survey slice image is received from which landmarks may be detected. In the next iteration acquisition parameters are updated to generate a next survey image from which further anatomical landmarks are identified. The update may be dependent on the anatomical landmarks already identified from previous iterations…The identification of (additional) anatomical landmarks from the current survey image slice is carried out by the machine learning model that is inspired to set the next survey slice image acquisition from anatomical landmarks already identified and uses trained rules that may predict further anatomical landmarks form the already identified ones.”; Wherein the landmark positions of the vicinity, or subsequent, cross-sections are acquired based on the identified landmarks on the survey image and its relationship to the survey images).
Wenzel in view of Han does not disclose expressly: project the anatomical landmark position from the predetermined standard plane to the vicinity cross section by at least one of projection by a perpendicular line drawn to the vicinity cross section, rotational movement corresponding to rotational movement for acquiring the vicinity cross section, or registration between the standard plane image and the vicinity cross section image.
Liu discloses: the acquisition of a plurality of images to be processed by causing rotational movement to an initial image (Liu: 0089-0091: “the plurality of images to be processed may include information of an initial image. For example, the multiple images to be processed may include an initial image or a portion of the initial image. For another example, the initial image may be directly used as one of the multiple images to be processed…In some embodiments, the plurality of images to be processed may include regions of interest at multiple angles. Accordingly, the plurality of images to be processed may also be understood as corresponding to a three-dimensional image… The processing device 120 may select a plurality of continuous images that are adjacent to the initial image, for example, in space and/or time, from the image sequence as the plurality of images to be processed.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the acquisition/selection of survey images using the modification of scan parameters disclosed by Wenzel in view of Han with the methods for the acquisition of images to be processed by using rotation of an initial image disclosed by Liu. The suggestion/motivation for doing so would have been “For example, the regions of interest included in the at least two additional images may be the same as the region of interest included in the initial image” (Liu: 0079; Wherein by selecting images based on direct movement of the initial image within a 3d volume, the region of interest may be preserved). Therefore, it would have been obvious to combine Wenzel in view of Han with Liu to obtain the invention as specified in claim 20.
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wenzel in view of Han, and further in view of Otomaru et al. (US-20180064422-A1) hereinafter referenced as Otomaru.
Regarding claim 13, Wenzel in view of Han discloses: The image processing apparatus according to claim 12.
Wenzel in view of Han does not disclose expressly: wherein the at least one processor further causes the image processing apparatus to: correct the anatomical landmark position on the vicinity cross section image by using an anatomical landmark position specified by template matching using a template determined on the basis of the predetermined standard plane by using, as a basis, the anatomical landmark position acquired by the image processing apparatus.
Otomaru discloses: wherein the at least one processor further causes the image processing apparatus to: correct an anatomical landmark position on a vicinity cross section image by using an anatomical landmark position specified by template matching using a template determined on the basis of a predetermined standard plane by using, as a basis, the anatomical landmark position acquired by the image processing apparatus (Otomaru: 0017: “An image processing apparatus in the present embodiment calculates and displays a degree of matching of two points (a first point and a second point) in two images (a first image and a second image) that a user designated…The degree of matching is calculated based on a positional relationship of a position of the first point and the second point and a position of a feature part of an object obtained from the first image and the second image respectively… The user can interactively confirm the degree of matching of the designated pair of corresponding point candidates. For this reason, in a case in which points are determined to be a mismatch, it is possible to re-input the pair of corresponding points at more appropriate positions.”;
0019: “the first and the second image are configured as a set of two-dimensional tomographic images, and the position and the orientation of each of the two-dimensional tomographic images are assumed to be stored in the data server 150 after being transformed to a reference coordinate system (a coordinate system within a space based on the object). The first and the second images represented by the reference coordinate system are inputted to the image processing apparatus 100 via an image obtainment unit 102.”;
0022: “A registration execution unit 114 executes a registration between the first image and the second image by using a corresponding point group stored in the storage unit (not shown). Then, a registration result (deformation field and deformation image) is stored in the storage unit (not shown).”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the algorithms for candidate point matching between images taught by Otomaru for the correcting the predicted landmarks in the survey images disclosed by Wenzel in view of Han. The suggestion/motivation for doing so would have been “The present image processing apparatus is capable of calculating the degree of matching by using only a feature part of an object, unlike the technique of Japanese Patent Laid-Open No. 2016-21998 in which a plurality of already associated points are required in order to calculate the degree of matching…For this reason, in a case in which points are determined to be a mismatch, it is possible to re-input the pair of corresponding points at more appropriate positions. In the present embodiment, the first image and the second image are assumed to be images of a breast, where the same patient is captured at different orientations.” (Otomaru: 0017). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Wenzel in view of Han with Otomaru to obtain the invention as specified in claim 13.
Claim(s) 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wenzel in view of Han and Otomaru, and further in view of Ota et al. (JP2019118694A) hereinafter referenced as Ota.
Regarding claim 14, Wenzel in view of Han and Otomaru discloses: The image processing apparatus according to claim 13.
Wenzel in view of Han and Otomaru does not disclose expressly: wherein the at least one processor causes the image processing apparatus to: exclude, from the learning data, the vicinity cross section image for which the anatomical landmark position on the vicinity cross section image corresponding to the anatomical landmark position on the cross section image of the predetermined standard plane cannot be specified in the template matching.
Ota discloses: the discarding of an image from a training dataset based on a similarity determination between a template image and an image to be judged (Ota: 0051: “If the similarity is equal to or greater than a threshold, the similarity determination unit 13a transfers the medical image to be determined, which is set in the second frame buffer 13d, to the memory unit 7; if the similarity is less than the threshold, the similarity determination unit 13a discards the medical image without transferring it to the memory unit 7.”; 0054: “The similarity determination unit 13a can calculate the similarity by using…or the like (typically, template matching).”), wherein the similarity determination may be based on determining similarities between specific portions, or points, within the images (Ota: 0062: “When the similarity determination unit 13a performs similarity determination, it is possible to perform similarity determination with higher accuracy by, for example, targeting only a specific region or specific portion that the photographer pays attention to as the target of similarity determination.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the algorithms for performing similarity determination disclosed in Ota into Wenzel in view of Han and Otomaru by discarding survey images based on template matching targeting landmark points. The suggestion/motivation for doing so would have been “this makes it possible to apply highly accurate machine learning to the classifier, thereby enabling the construction of a classifier that can classify medical images with high accuracy.” (Ota: 0081). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Wenzel in view of Han and Otomaru with Ota to obtain the invention as specified in claim 14.
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wenzel in view of Han, and further in view of Ota.
Regarding claim 15, Wenzel in view of Han discloses: The image processing apparatus according to claim 1, wherein the at least one processor further causes the image processing apparatus to: display, on a display unit, the vicinity cross section image acquired by the image processing apparatus (Wenzel: 0054: “The control system 111 may be configured to receive data such as survey image data from the MRI scanning imaging system 101. For example, the processor 103 may be adapted to receive information (automatically or upon request) from the scanning imaging system 101 in a compatible digital form so that such information may be displayed on the display device 125. Such information may include operating parameters, alarm notifications, and other information related to the use, operation and function of the scanning imaging system 101.”).
Wenzel in view of Han does not disclose expressly: wherein the at least one processor further causes the image processing apparatus to: acquire a result of determination of whether or not the vicinity cross section image displayed on the display unit is to be adopted as the learning data, and wherein the at least one processor causes the image processing apparatus to: exclude, from the learning result, the vicinity cross section image shown in the determination result as the vicinity cross section image not to be adopted as the learning data.
Ota discloses: the exclusion of an image from a learning dataset based upon a determination result to not adopt the image as learning data (Ota: 0051: “The similarity determination unit 13a compares an image of interest set in the first frame buffer 13c with a medical image to be determined set in the second frame buffer 13d, and calculates the similarity between the image of interest and the medical image to be determined. If the similarity is equal to or greater than a threshold, the similarity determination unit 13a transfers the medical image to be determined, which is set in the second frame buffer 13d, to the memory unit 7; if the similarity is less than the threshold, the similarity determination unit 13a discards the medical image without transferring it to the memory unit 7.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to art to implement the algorithms for performing similarity determination based upon an image selected by the user disclosed in Ota into Wenzel in view of Han by discarding subsequent survey images based upon survey image template matching. The suggestion/motivation for doing so would have been “this makes it possible to apply highly accurate machine learning to the classifier, thereby enabling the construction of a classifier that can classify medical images with high accuracy.” (Ota: 0081). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Wenzel in view of Han with Ota to obtain the invention as specified in claim 15.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANTHONY J RODRIGUEZ whose telephone number is (703)756-5821. The examiner can normally be reached Monday-Friday 10am-7pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANTHONY J RODRIGUEZ/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights