DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see Remarks page 11, filed 11/04/2025, with respect to the rejection of claim 9 under 35 U.S.C. 112(b) have been fully considered and are persuasive. The rejection of claim 9 has been withdrawn.
Applicant's arguments, see Remarks pages 11-13, filed 11/04/2025, with respect to the rejection of amended claims 1, 16, and 17 under 35 U.S.C. 102(a)(1) have been fully considered but they are not persuasive.
On pages 12-13 of Remarks, Applicant argues:
PNG
media_image1.png
660
674
media_image1.png
Greyscale
Examiner respectfully disagrees. Paragraph 0025 of Wenzel discloses “…reinforcement training of a deep learning system may be done by applying multi-planar reformatting (MPR) to the 3D brain scan in order to extract a slice matching desired scan parameters (e.g. size, orientation, origin etc.), simulating their acquisition on a scanner. Output variables of the deep learning network are the estimated positions of the anatomical landmarks as well as their confidence level, along with parameters for the next slice which may again be simulated by applying MPR on the dataset.” Wherein the parameters for acquiring a next survey image slice from a 3D scan based on inputting a survey image slice into a deep learning network constitute the parameters of a vicinity cross-section, acquired based on moving a predetermined standard plane, which constitutes the initial survey image disclosed by paragraph 0016 of Wenzel. Thus, Wenzel discloses “acquire, on the basis of a vicinity cross section parameter representing a vicinity cross section obtained by moving the predetermined standard plane to the outside the predetermined standard plane on the basis of the standard plane parameter and the three-dimensional image, a vicinity cross section image.”
In addition, paragraph 0016 of Wenzel discloses “According to the invention the (survey-) image from which the anatomical landmarks are derived is being built-up during the landmark identification. In particular, a survey slice image is received from which landmarks may be detected. In the next iteration acquisition parameters are updated to generate a next survey image from which further anatomical landmarks are identified. The update may be dependent on the anatomical landmarks already identified from previous iterations…The identification of (additional) anatomical landmarks from the current survey image slice is carried out by the machine learning model that is inspired to set the next survey slice image acquisition from anatomical landmarks already identified and uses trained rules that may predict further anatomical landmarks form the already identified ones,” and paragraph 0025 of Wenzel discloses “Output variables of the deep learning network are the estimated positions of the anatomical landmarks as well as their confidence level, along with parameters for the next slice which may again be simulated by applying MPR on the dataset.” Wherein, the initial survey slice image, constituting the predetermined standard plane, is processed by the machine learning model in order to detect landmarks and update the acquisition parameters in order to predict the positions of further anatomical landmarks based on already detected ones. Thus, the iterative accumulation of landmark positions on survey slice images constitutes the projection of landmark positions.
Furthermore, the extraction of survey slice image data from a dataset of 3D brain scans with known landmark positions by applying multi-planar reformatting, disclosed by paragraph 0025 of Wenzel constitutes the augmentation of learning data.
Therefore, the rejection of claim 1 under 35 U.S.C. 102(a)(1) is maintained.
As per claim(s) 16 & 17, arguments made in rejecting claim(s) 1 are analogous.
Claim Interpretation
Note that according to the Federal Circuit’s 2004 Superguide v. DirecTV decision, “at least one of … and …” requires at least one instance of each and every item listed.
Claim 18 recite(s) “acquire the vicinity cross section by moving the predetermined standard plane to the outside the predetermined standard plane by at least one of rotational movement and parallel movement”
Claim 20 recite(s) “project the anatomical landmark position from the predetermined standard plane to the vicinity cross section by at least one of projection by a perpendicular line drawn to the vicinity cross section, rotational movement corresponding to rotational movement for acquiring the vicinity cross section, and registration between the standard plane image and the vicinity cross section image.”
If Applicant intends for an interpretation of only one of these items being required for claim interpretation, Applicant can amend the claim language to, instead recite, “at least one of … or …”. In SuperGuide, the Federal Circuit held that the plain meaning of “at least one of A, B, and C” means: at least one of A, at least one of B, and at least one of C. The Court held that if the applicant intended “at least one of A, B, and C” to mean A, B, or C, they should have used “OR.”
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-4, 10, 12, 16-17, 19, and 21 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Wenzel et al. (US 2020/0279371 A1) hereinafter referenced as Wenzel.
Regarding claim 1, Wenzel discloses: An image processing apparatus (Wenzel: Abstract) comprising: at least one memory storing a program; and at least one processor which, by executing the program, causes the image processing apparatus (Wenzel: 0031) to:
acquire data including a three-dimensional image obtained by capturing an image of a subject under examination (Wenzel: 0025: “the training can be done based on a retrospective dataset of 3D brain scans with known positions for anatomical landmarks.”), a standard plane parameter representing a predetermined standard plane in the three-dimensional image (Wenzel: 0024: “the execution of the learning algorithm comprises determining from the training set image data representing a slice corresponding to a given set of parameters, and executing the learning algorithm using the slice.”;
0059-0060: “In step 201, a current set of acquisition parameters may be determined. The current set of parameters may for example be determined by the control system 111. For example, the set of acquisition parameters may comprise at least one of: number of voxels in X- and Y-direction, the voxel size in mm, the center of voxel (0,0) in DICOM patient coordinates using for example DICOM Tag "ImagePositionPatient", and 3D orientation of the slice in DICOM patient coordinates using for example DICOM Tag: "ImageOrientationPatient”; Wherein the initial acquisition parameters for the survey image data represent the standard plane parameters), and an anatomical landmark position in the three-dimensional image (Wenzel: 0025: “For example, the training can be done based on a retrospective dataset of 3D brain scans with known positions for anatomical landmarks.”);
acquire, on the basis of a vicinity cross section parameter representing a vicinity cross section obtained by moving the predetermined standard plane to the outside the predetermined standard plane on the basis of the standard plane parameter and the three-dimensional image, a vicinity cross section image (Wenzel: 0025: “…reinforcement training of a deep learning system may be done by applying multi-planar reformatting (MPR) to the 3D brain scan in order to extract a slice matching desired scan parameters (e.g. size, orientation, origin etc.), simulating their acquisition on a scanner. Output variables of the deep learning network are the estimated positions of the anatomical landmarks as well as their confidence level, along with parameters for the next slice which may again be simulated by applying MPR on the dataset.”; Wherein the image slice parameters output by the deep learning model, used to extract subsequent image slices from the 3D brain scan using MPR, constitute vicinity cross section parameters.);
acquire the anatomical landmark position on the vicinity cross section image by projecting an anatomical landmark position in the predetermined standard plane onto the vicinity cross section on the basis of the anatomical landmark position and the relationship between the predetermined standard plane and the vicinity cross section (Wenzel: 0016: “According to the invention the (survey-) image from which the anatomical landmarks are derived is being built-up during the landmark identification. In particular, a survey slice image is received from which landmarks may be detected. In the next iteration acquisition parameters are updated to generate a next survey image from which further anatomical landmarks are identified. The update may be dependent on the anatomical landmarks already identified from previous iterations…The identification of (additional) anatomical landmarks from the current survey image slice is carried out by the machine learning model that is inspired to set the next survey slice image acquisition from anatomical landmarks already identified and uses trained rules that may predict further anatomical landmarks form the already identified ones.”;
0022: “The selection of the actions may be optimized by learning based on known landmarks marked on input images in order to maximize the future reward.”; 0025: “Output variables of the deep learning network are the estimated positions of the anatomical landmarks as well as their confidence level, along with parameters for the next slice which may again be simulated by applying MPR on the dataset.”; Wherein the landmark positions of the vicinity, or subsequent, cross-sections are acquired based on the identified landmarks on the survey image and its relationship to the survey images); and
acquire a learning model built by using, as performing augmentation of learning data, information including a pair of the vicinity cross section image and the anatomical landmark position acquired by the image processing apparatus to estimate, from the cross section image, an anatomical landmark position on the cross section image (Wenzel: 0024: “For example, arbitrary images for any set of parameters can be extracted from the training scans by planar reformatting with their corresponding ground truth information ( e.g. the position of anatomical landmarks, as manually marked by a technician, or as determined by automatic approaches such as SmartExam) …reinforcement training of a deep learning system may be done by applying multi-planar reformatting (MPR) to the 3D brain scan in order to extract a slice matching desired scan parameters (e.g. size, orientation, origin etc.), simulating their acquisition on a scanner. Output variables of the deep learning network are the estimated positions of the anatomical landmarks as well as their confidence level, along with parameters for the next slice which may again be simulated by applying MPR on the dataset.”; Wherein parameters for survey image slices, including initial acquisition parameters, are iteratively calculated to extract an image from a 3D scan to be input into a model to train it to estimate positions of landmarks on the slices, and wherein the extraction of subsequent image slices from a 3D scan by applying MPR constitutes learning data augmentation.).
Regarding claim 2, Wenzel discloses: The image processing apparatus according to claim 1, wherein the at least one processor causes the image processing apparatus to: acquire, on the basis of the standard plane parameter, a plurality of the vicinity cross section images defining a plurality of the cross sections in the vicinity of the predetermined standard plane; and acquire the anatomical landmark position in each of the plurality of vicinity cross section images. (Wenzel: 0016: “a survey slice image is received from which landmarks may be detected. In the next iteration acquisition parameters are updated to generate a next survey image from which further anatomical landmarks are identified. The update may be dependent on the anatomical landmarks already identified from previous iterations. This process may continue for a pre-determined number of iterations, which may be defined by a pre-set maximum number of iterations, or the number of iterations that achieves to find a pre-set maximum number of landmarks identified.”; Wherein survey images determined within the same 3D volume constitute being in the same vicinity, wherein the initial survey image constitutes the standard plane image.).
Regarding claim 3, Wenzel discloses: The image processing apparatus according to claim 1,
wherein the vicinity cross section image is a cross section image defining a cross section outside the predetermined standard plane. (Wenzel: 0025-0026: “reinforcement training of a deep learning system
may be done by applying multi-planar reformatting (MPR) to the 3D brain scan in order to extract a slice matching desired scan parameters (e.g. size, orientation, origin etc.), simulating their acquisition on a scanner. Output variables of the deep learning network are the estimated positions of the anatomical landmarks as well as their confidence level, along with parameters for the next slice which may again be simulated by applying MPR on the dataset.…According to one embodiment, the set of acquisition parameters comprises at least one of the following types: an indication of a slice of the anatomy; voxel size of the image data; number of voxels in the image data; the center of voxel in the image data; the 3D orientation of the slice.”; Wherein the slices have differing orientations and locations within the 3d volume.).
Regarding claim 4, Wenzel discloses: The image processing apparatus according to claim 1,
wherein the at least one processor causes the image processing apparatus to: project the anatomical landmark position acquired by the image processing apparatus on the vicinity cross section image to acquire the anatomical landmark position on the vicinity cross section image (Wenzel: 0021: “the method further comprises providing a training set of image data with a known set of landmarks and multiple sets of acquisition parameters, and executing learning algorithm on the training set for generating the machine learning model. For example the training set may comprise a sequence of annotated images indicating the set of landmarks.”;
0025: “Output variables of the deep learning network are the estimated positions of the anatomical landmarks as well as their confidence level, along with parameters for the next slice which may again be simulated by applying MPR on the dataset.”).
Regarding claim 10, Wenzel discloses: The image processing apparatus according to claim 1,
wherein the at least one processor further causes the image processing apparatus to: acquire an input three-dimensional image and an input standard plane parameter representing a predetermined standard plane on the input three-dimensional image (Wenzel: 0058-0059: “FIG. 2 is a flowchart of a method for locating anatomical landmarks of a predefined anatomy (or anatomical structure) using a scanning imaging system e.g. an MRI scanning imaging system. The anatomy may be part of a subject (e.g. 318 of FIG. 3) to be imaged. The anatomy may for example be a heart, brain, knee, spine, shoulder, breast, etc. The imaging of the anatomy may result in images that can be used to perform further actions such as treatment delivery. In step 201, a current set of acquisition parameters may be determined. The current set of parameters may for example be determined by the control system 111.”);
acquire an input cross section image based on each of the input three-dimensional image and the input standard plane parameter (Wenzel: 0059-0060: “”the set of acquisition parameters may comprise at least one of: number of voxels in X- and Y-direction, the voxel size in mm, the center of voxel (0,0) in DICOM patient coordinates using for example DICOM Tag "ImagePositionPatient", and 3D orientation of the slice in DICOM patient coordinates using for example DICOM Tag: "ImageOrientationPatient". DICOM stands for Digital Imaging and Communications in Medicine. In step 203, survey image data representing a slice of the anatomy may be received e.g. at the control system 111. For example, the receiving of the survey image data may automatically be performed.); and
estimate an anatomical landmark position on the input cross section image by using the learning model acquired by the image processing apparatus (Wenzel: 0061: “In step 205, anatomical landmarks may be identified in the acquired image data using the machine learning model. For example, a confidence level may be assigned to the identified anatomical landmarks.”).
Regarding claim 12, Wenzel discloses: The image processing apparatus according to claim 19 (Disclosed in 35 U.S.C. 102(a)(1) rejection below), wherein the at least one processor causes the image processing apparatus to: specify an anatomical landmark position on the vicinity cross section image corresponding to an anatomical landmark position on the standard plane image of the predetermined standard plane on the basis of a correspondence relationship between a pixel on the standard plane image of the predetermined standard plane and a pixel on the vicinity cross section image to acquire the anatomical landmark position on the vicinity cross section image (Wenzel: 0055: “The AI component 150 may be configured for a robust and fast detecting of anatomical landmarks by adaptive acquisition. The AI component 150 as further described herein may be configured to dynamically adapt/drive the acquisition process of the survey scan based on the partial anatomical information already available, and therefore reducing the number of required images and, consequently, overall scan time for localizing anatomical landmarks required for fully automated scan planning.”;
0057: “The AI component 150 may be configured to perform machine learning on training sets in order to generate one or more machine learning models for predicting anatomical landmarks in image data obtained using a set of acquisition parameters and for predicting a subsequent set of acquisition parameters of the set of acquisition parameters for subsequent acquiring of image data.”; Wherein the usage of the initial survey image and landmarks positions estimated by the model for the determination of subsequent, or vicinity cross section, images used constitutes on the basis of pixel relationships).
As per claim(s) 16, arguments made in rejecting claim(s) 1 are analogous.
As per claim(s) 17, arguments made in rejecting claim(s) 1 are analogous. In addition, 0070 of Wenzel discloses: A non-transitory computer readable medium that stores a program, wherein the program causes a computer to execute the steps.
Regarding claim 19, Wenzel discloses: The image processing apparatus according to claim 1, wherein the at least one processor causes the image processing apparatus to: acquire, on the basis of the three-dimensional image and the standard plane parameter, a standard plane image (Wenzel: 0058-0060: “FIG. 2 is a flowchart of a method for locating anatomical landmarks of a predefined anatomy (or anatomical structure) using a scanning imaging system e.g. an MRI scanning imaging system. The anatomy may be part of a subject (e.g. 318 of FIG. 3) to be imaged. The anatomy may for example be a heart, brain, knee, spine, shoulder, breast, etc. The imaging of the anatomy may result in images that can be used to perform further actions such as treatment delivery. For example, the set of acquisition parameters may comprise at least one of: number of voxels in X- and Y-direction, the voxel size in mm, the center of voxel (0,0) in DICOM patient coordinates using for example DICOM Tag "ImagePositionPatient", and 3D orientation of the slice in DICOM patient coordinates using for example DICOM Tag: "ImageOrientationPatient”… In step 203, survey image data representing a slice of the anatomy may be received e.g. at the control system 111.; Wherein the survey image data for the anatomical structure is acquired based on the 3d acquisition parameters).
Regarding claim 21, Wenzel discloses: The image processing apparatus according to claim 19, wherein the at least one processor causes the image processing apparatus to: acquire the learning model built by using, as learning data, information including a pair of the standard plane image and the anatomical landmark position in the predetermined standard plane (Wenzel: 0024: “the execution of the learning algorithm comprises determining from the training set image data representing a slice corresponding to a given set of parameters, and executing the learning algorithm using the slice. The image data of the slice may be indicative of the location of the set of landmarks of the anatomy. For example, the machine learning model may comprise a set of rules for choosing the set of parameters and the identification of the landmarks. ”;
0059-0060: “In step 201, a current set of acquisition parameters may be determined. The current set of parameters may for example be determined by the control system 111. For example, the set of acquisition parameters may comprise at least one of: number of voxels in X- and Y-direction, the voxel size in mm, the center of voxel (0,0) in DICOM patient coordinates using for example DICOM Tag "ImagePositionPatient", and 3D orientation of the slice in DICOM patient coordinates using for example DICOM Tag: "ImageOrientationPatient”; Wherein the training dataset comprises a survey image slice, comprising landmark locations, for a predetermined standard plane).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 5-9, 11, 18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Wenzel, and further in view of Liu et al. (WO2021239141A1) hereinafter referenced as Liu.
Regarding claim 5, Wenzel discloses: The image processing apparatus according to claim 1,
wherein the at least one processor causes the image processing apparatus to: acquire a plurality of the vicinity cross section images by causing at least one movement of the predetermined standard plane (Wenzel: 0025-0026: “reinforcement training of a deep learning system may be done by applying multi-planar reformatting (MPR) to the 3D brain scan in order to extract a slice matching desired scan parameters (e.g. size, orientation, origin etc.), simulating their acquisition on a scanner. Output variables of the deep learning network are the estimated positions of the anatomical landmarks as well as their confidence level, along with parameters for the next slice which may again be simulated by applying MPR on the dataset. According to one embodiment, the set of acquisition parameters comprises at least one of the following types: an indication of a slice of the anatomy; voxel size of the image data; number of voxels in the image data; the center of voxel in the image data; the 3D orientation of the slice.”; Wherein the calculation of a survey image based on the results of an input image constitutes the movement of the survey image to a different position); and
approximate the anatomical landmark position acquired by the image processing apparatus on the basis of the at least one movement to a position on the vicinity cross section image to thereby acquire the anatomical landmark position on each of the plurality of vicinity cross section images (Wenzel: 0055: “The AI component 150 as further described herein may be configured to dynamically adapt/drive the acquisition process of the survey scan based on the partial anatomical information already available, and therefore reducing the number of required images and, consequently, overall scan time for localizing anatomical”; Wherein the localization of anatomical landmarks based on subsequent survey images determined by the results of the previous images constitutes the landmark estimation based on movement).
Wenzel does not disclose expressly: acquire a plurality of the vicinity cross section images by causing at least one movement which is either parallel movement or rotational movement of the predetermined standard plane.
Liu discloses: the acquisition of a plurality of images to be processed by causing parallel movement and rotational movement to an initial image (Liu: 0089-0091: “the plurality of images to be processed may include information of an initial image. For example, the multiple images to be processed may include an initial image or a portion of the initial image. For another example, the initial image may be directly used as one of the multiple images to be processed…In some embodiments, the plurality of images to be processed may include regions of interest at multiple angles. Accordingly, the plurality of images to be processed may also be understood as corresponding to a three-dimensional image… The processing device 120 may select a plurality of continuous images that are adjacent to the initial image, for example, in space and/or time, from the image sequence as the plurality of images to be processed.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the acquisition/selection of survey images using the modification of scan parameters disclosed by Wenzel with the methods for the acquisition of images to be processed by using rotation and translation of an initial image disclosed by Liu. The suggestion/motivation for doing so would have been “For example, the regions of interest included in the at least two additional images may be the same as the region of interest included in the initial image” (Liu: 0079; Wherein by selecting images based on direct movement of the initial image within a 3d volume, the region of interest may be preserved). Further, one skilled in the art could have substituted the elements as described above by known methods with no change in their respective functions, and the substitution would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Wenzel with Liu to obtain the invention as specified in claim 5.
Regarding claim 6, Wenzel in view of Liu discloses: The image processing apparatus according to claim 5, wherein the parallel movement is parallel movement to the outside of the predetermined standard plane (Liu: 0091: “The processing device 120 may select a plurality of continuous images that are adjacent to the initial image, for example, in space and/or time, from the image sequence as the plurality of images to be processed.”; 0135: “This embodiment selects multiple two-dimensional images adjacent to a two-dimensional image from a medical image sequence, and constructs a corresponding first image with three dimensions based on the multiple two-dimensional images to add additional feature information (such as texture information, gradient information, and grayscale information of the structure).”; Wherein the acquisition of multiple spatially adjacent images constitutes the parallel movement of the initial image), while the rotational movement is rotational movement around a predetermined axis extending over the predetermined standard plane (Liu: 0087: “The processing device 120 may select an image in the image sequence as the initial image…a region of interest (such as a lesion) of the object may be determined based on the image generated by the scan. If the region of interest is not at the reconstruction center, the reconstruction parameters can be adjusted to make it located at the center, and a new scan can be performed to obtain a new image”; 0090: “the plurality of images to be processed may include regions of interest at multiple angles. Accordingly, the plurality of images to be processed may also be understood as corresponding to a three-dimensional image.”; Wherein the initial image containing the region of interest at the center and selecting images based on the region of interest at multiple angles constitutes the rotational movement around an axis.).
Regarding claim 7, Wenzel in view of Liu discloses: The image processing apparatus according to claim 6, wherein the predetermined axis is an axis passing through the at least one anatomical landmark in the predetermined standard plane (Liu: 0087: “The processing device 120 may select an image in the image sequence as the initial image…a region of interest (such as a lesion) of the object may be determined based on the image generated by the scan. If the region of interest is not at the reconstruction center, the reconstruction parameters can be adjusted to make it located at the center, and a new scan can be performed to obtain a new image”; 0090: “the plurality of images to be processed may include regions of interest at multiple angles. Accordingly, the plurality of images to be processed may also be understood as corresponding to a three-dimensional image.”; Wherein the region of interest constitutes an anatomical landmark.).
Regarding claim 8, Wenzel in view of Liu discloses: The image processing apparatus according to claim 6, wherein the predetermined axis includes at least two or more axes passing through the predetermined standard plane, and wherein the at least one processor causes the image processing apparatus to select either or any one of the two or more axes for each rotational movement (Liu: 0090: “the plurality of images to be processed may include regions of interest at multiple angles. Accordingly, the plurality of images to be processed may also be understood as corresponding to a three-dimensional image.”).
Regarding claim 9, Wenzel in view of Liu discloses: The image processing apparatus according to claim 5, wherein the at least one movement is movement at a predetermined interval (Liu: 0090: “The processing device 120 may select a plurality of continuous images that are adjacent to the initial image, for example, in space and/or time, from the image sequence as the plurality of images to be processed.”; Wherein the acquisition of spatially adjacent images constitutes a predetermined interval.).
Regarding claim 11, Wenzel discloses: The image processing apparatus according to claim 1,
wherein the predetermined standard plane includes at least one standard plane (Wenzel: 0024: “the execution of the learning algorithm comprises determining from the training set image data representing a slice corresponding to a given set of parameters, and executing the learning algorithm using the slice.”;
0059-0060: “In step 201, a current set of acquisition parameters may be determined. The current set of parameters may for example be determined by the control system 111. For example, the set of acquisition parameters may comprise at least one of: number of voxels in X- and Y-direction, the voxel size in mm, the center of voxel (0,0) in DICOM patient coordinates using for example DICOM Tag "ImagePositionPatient", and 3D orientation of the slice in DICOM patient coordinates using for example DICOM Tag: "ImageOrientationPatient”; Wherein the initial acquisition parameters for the survey image data represent the standard plane parameters), wherein the at least one processor further causes the image processing apparatus to:
assign the anatomical landmark position to the at least one standard plane, wherein the data acquired by the image processing apparatus includes the anatomical landmark position assigned by the image processing apparatus (Wenzel: 0025: “According to one embodiment, the training set comprises image data representing a 3D volume of the anatomy. The image data may be indicative of the location of the set of landmarks of the anatomy… arbitrary images for any set of parameters can be extracted from the training scans by planar reformatting with their corresponding ground truth information (e.g. the position of anatomical landmarks, as manually marked by a technician, or as determined by automatic approaches such as SmartExam).”), and
wherein the at least one processor causes the image processing apparatus to: acquire the learning model by using, as the learning data, a pair of a cross section image representing a cross section in the vicinity of the predetermined standard plane and a position based on the anatomical landmark position assigned by the image processing apparatus (Wenzel: 0024: “For example, arbitrary images for any set of parameters can be extracted from the training scans by planar reformatting with their corresponding ground truth information ( e.g. the position of anatomical landmarks, as manually marked by a technician, or as determined by automatic approaches such as SmartExam) …reinforcement training of a deep learning system may be done by applying multi-planar reformatting (MPR) to the 3D brain scan in order to extract a slice matching desired scan parameters (e.g. size, orientation, origin etc.), simulating their acquisition on a scanner. Output variables of the deep learning network are the estimated positions of the anatomical landmarks as well as their confidence level, along with parameters for the next slice which may again be simulated by applying MPR on the dataset.”; Wherein parameters for image slices, including initial acquisition parameters, are iteratively calculated to extract an image to be input into a model to train it to estimate positions of landmarks on the slices, wherein the input images are labeled.).
Wenzel does not disclose expressly: wherein the predetermined standard plane includes at least two or more standard planes.
Liu discloses: multiple sets of training samples each containing an initial image and a set of images to be processed generated from each initial image (Liu: 0096: “In some embodiments, a large number of training samples may be used to train an initial model to obtain a processed model. Each training sample may include an initial training image, multiple training images to be processed, and a target training image (ie, a gold standard). A large number of training samples can be input to the initial model in batches.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the known technique as disclosed by Liu of including multiple initial images into Wenzel by training the model disclosed by Wenzel using different 3D image volumes. The suggestion/motivation for doing so would have been “training samples corresponding to different organs may be used to train a processing model, and the processing model may be used to process different organs.” (Liu: 0097). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Wenzel with Liu to obtain the invention as specified in claim 11.
Regarding claim 18, Wenzel discloses: The image processing apparatus according to claim 1,
wherein the at least one processor causes the image processing apparatus to: acquire the vicinity cross section by moving the predetermined standard plane to the outside the predetermined standard plane, and acquires the vicinity cross section image on the basis of the acquired vicinity cross section and the three-dimensional image (Wenzel: 0025: “…reinforcement training of a deep learning system may be done by applying multi-planar reformatting (MPR) to the 3D brain scan in order to extract a slice matching desired scan parameters (e.g. size, orientation, origin etc.), simulating their acquisition on a scanner. Output variables of the deep learning network are the estimated positions of the anatomical landmarks as well as their confidence level, along with parameters for the next slice which may again be simulated by applying MPR on the dataset.”; Wherein the image slice parameters output by the deep learning model, used to extract subsequent image slices from the 3D brain scan using MPR, constitute vicinity cross section acquired by moving the originally input survey image, outside the predetermined standard plane.).
Wenzel does not disclose expressly: acquire the vicinity cross section by moving the predetermined standard plane to the outside the predetermined standard plane by at least one of rotational movement and parallel movement (Claim limitation is interpreted as “at least one…or…” as disclosed according to SuperGuide Interpretation above).
Liu discloses: the acquisition of a plurality of images to be processed by causing parallel movement and rotational movement to an initial image (Liu: 0089-0091: “the plurality of images to be processed may include information of an initial image. For example, the multiple images to be processed may include an initial image or a portion of the initial image. For another example, the initial image may be directly used as one of the multiple images to be processed…In some embodiments, the plurality of images to be processed may include regions of interest at multiple angles. Accordingly, the plurality of images to be processed may also be understood as corresponding to a three-dimensional image… The processing device 120 may select a plurality of continuous images that are adjacent to the initial image, for example, in space and/or time, from the image sequence as the plurality of images to be processed.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the acquisition/selection of survey images using the modification of scan parameters disclosed by Wenzel with the methods for the acquisition of images to be processed by using rotation and translation of an initial image disclosed by Liu. The suggestion/motivation for doing so would have been “For example, the regions of interest included in the at least two additional images may be the same as the region of interest included in the initial image” (Liu: 0079; Wherein by selecting images based on direct movement of the initial image within a 3d volume, the region of interest may be preserved). Further, one skilled in the art could have substituted the elements as described above by known methods with no change in their respective functions, and the substitution would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Wenzel with Liu to obtain the invention as specified in claim 18.
Regarding claim 20, Wenzel discloses: The image processing apparatus according to claim 19, wherein the at least one processor causes the image processing apparatus to: project the anatomical landmark position from the predetermined standard plane to the vicinity cross section (Wenzel: 0016: “According to the invention the (survey-) image from which the anatomical landmarks are derived is being built-up during the landmark identification. In particular, a survey slice image is received from which landmarks may be detected. In the next iteration acquisition parameters are updated to generate a next survey image from which further anatomical landmarks are identified. The update may be dependent on the anatomical landmarks already identified from previous iterations…The identification of (additional) anatomical landmarks from the current survey image slice is carried out by the machine learning model that is inspired to set the next survey slice image acquisition from anatomical landmarks already identified and uses trained rules that may predict further anatomical landmarks form the already identified ones.”;
0022: “The selection of the actions may be optimized by learning based on known landmarks marked on input images in order to maximize the future reward.”; 0025: “Output variables of the deep learning network are the estimated positions of the anatomical landmarks as well as their confidence level, along with parameters for the next slice which may again be simulated by applying MPR on the dataset.”; Wherein the landmark positions of the vicinity, or subsequent, cross-sections are acquired based on the identified landmarks on the survey image and its relationship to the survey images).
Wenzel does not disclose expressly: project the anatomical landmark position from the predetermined standard plane to the vicinity cross section by at least one of projection by a perpendicular line drawn to the vicinity cross section, rotational movement corresponding to rotational movement for acquiring the vicinity cross section, and registration between the standard plane image and the vicinity cross section image (Claim limitation is interpreted as “at least one…or…” as disclosed according to SuperGuide Interpretation above).
Liu discloses: the acquisition of a plurality of images to be processed by causing rotational movement to an initial image (Liu: 0089-0091: “the plurality of images to be processed may include information of an initial image. For example, the multiple images to be processed may include an initial image or a portion of the initial image. For another example, the initial image may be directly used as one of the multiple images to be processed…In some embodiments, the plurality of images to be processed may include regions of interest at multiple angles. Accordingly, the plurality of images to be processed may also be understood as corresponding to a three-dimensional image… The processing device 120 may select a plurality of continuous images that are adjacent to the initial image, for example, in space and/or time, from the image sequence as the plurality of images to be processed.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the acquisition/selection of survey images using the modification of scan parameters disclosed by Wenzel with the methods for the acquisition of images to be processed by using rotation of an initial image disclosed by Liu. The suggestion/motivation for doing so would have been “For example, the regions of interest included in the at least two additional images may be the same as the region of interest included in the initial image” (Liu: 0079; Wherein by selecting images based on direct movement of the initial image within a 3d volume, the region of interest may be preserved). Further, one skilled in the art could have substituted the elements as described above by known methods with no change in their respective functions, and the substitution would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Wenzel with Liu to obtain the invention as specified in claim 20.
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wenzel, and further in view of Han et al. (WO 2016/036516 A1) hereinafter referenced as Han.
Regarding claim 13, Wenzel discloses: The image processing apparatus according to claim 12,
wherein the at least one processor further causes the image processing apparatus to: correct the anatomical landmark position on the vicinity cross section determined on the basis of the predetermined standard plane by using, as a basis, the anatomical landmark position acquired by the image processing apparatus (Wenzel: 0025: “According to one embodiment, the training set comprises image data representing a 3D volume of the anatomy. The image data may be indicative of the location of the set of landmarks of the anatomy… arbitrary images for any set of parameters can be extracted from the training scans by planar reformatting with their corresponding ground truth information (e.g. the position of anatomical landmarks, as manually marked by a technician, or as determined by automatic approaches such as SmartExam).”; Wherein survey images used for training, are labeled).
Wenzel does not disclose expressly: wherein the at least one processor further causes the image processing apparatus to: correct the anatomical landmark position on the vicinity cross section image by using an anatomical landmark position specified by template matching using a template determined on the basis of the predetermined standard plane by using, as a basis, the anatomical landmark position acquired by the image processing apparatus.
Han discloses: the defining/correction of an anatomical landmark position on an image of an anatomical region by using an anatomical landmark position specified by template matching using a template by using, as a basis, the anatomical landmark position acquired by the image processing apparatus (Han: 0064: “automatic detection may be performed through, e.g., image registration between the training image and a predetermined landmark map of the same anatomical region. For example, a predetermined landmark map may define a particular mapping or spacing of various landmark points for a particular anatomical region. lf user 112 is processing medical images showing a particular anatomical region of a patient, then user 112 could thus select a predetermined landmark map that corresponds to that anatomical region. Upon selection, the landmark points defined by the predetermined map may then be provided to module 121”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the automatic detection algorithm for mapping landmark points disclosed by predetermined landmark maps onto anatomical images taught by Han for the defining/correcting of the acquired training survey images disclosed by Wenzel with a set of predefined template images prior to inputting them into the AI model for training. The suggestion/motivation for doing so would have been “a predetermined landmark map may define a particular mapping or spacing of various landmark points for a particular anatomical region.” (Han: 0064). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Wenzel with Han to obtain the invention as specified in claim 13.
Claim(s) 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wenzel in view of Han, and further in view of Ota et al. (JP2019118694A) hereinafter referenced as Ota.
Regarding claim 14, Wenzel in view of Han discloses: The image processing apparatus according to claim 13.
Wenzel in view of Han does not disclose expressly: wherein the at least one processor causes the image processing apparatus to: exclude, from the learning data, the vicinity cross section image for which the anatomical landmark position on the vicinity cross section image corresponding to the anatomical landmark position on the cross section image of the predetermined standard plane cannot be specified in the template matching.
Ota discloses: the discarding of an image from a training dataset based on a similarity determination between a template image and an image to be judged (Ota: 0051: “If the similarity is equal to or greater than a threshold, the similarity determination unit 13a transfers the medical image to be determined, which is set in the second frame buffer 13d, to the memory unit 7; if the similarity is less than the threshold, the similarity determination unit 13a discards the medical image without transferring it to the memory unit 7.”; 0054: “The similarity determination unit 13a can calculate the similarity by using…or the like (typically, template matching).”), wherein the similarity determination may be based on determining similarities between specific portions, or points, within the images (Ota: 0062: “When the similarity determination unit 13a performs similarity determination, it is possible to perform similarity determination with higher accuracy by, for example, targeting only a specific region or specific portion that the photographer pays attention to as the target of similarity determination.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the algorithms for performing similarity determination disclosed in Ota into Wenzel in view of Han by discarding survey images based on template matching targeting landmark points. The suggestion/motivation for doing so would have been “this makes it possible to apply highly accurate machine learning to the classifier, thereby enabling the construction of a classifier that can classify medical images with high accuracy.” (Ota: 0081). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Wenzel in view of Han with Ota to obtain the invention as specified in claim 14.
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wenzel, and further in view of Ota.
Regarding claim 15, Wenzel discloses: The image processing apparatus according to claim 1,
wherein the at least one processor further causes the image processing apparatus to: display, on a display unit, the vicinity cross section image acquired by the image processing apparatus (Wenzel: 0054: “The control system 111 may be configured to receive data such as survey image data from the MRI scanning imaging system 101. For example, the processor 103 may be adapted to receive information (automatically or upon request) from the scanning imaging system 101 in a compatible digital form so that such information may be displayed on the display device 125. Such information may include operating parameters, alarm notifications, and other information related to the use, operation and function of the scanning imaging system 101.”).
Wenzel does not disclose expressly: wherein the at least one processor further causes the image processing apparatus to: acquire a result of determination of whether or not the vicinity cross section image displayed on the display unit is to be adopted as the learning data, and wherein the at least one processor causes the image processing apparatus to: exclude, from the learning result, the vicinity cross section image shown in the determination result as the vicinity cross section image not to be adopted as the learning data.
Ota discloses: the exclusion of an image from a learning dataset based upon a determination result to not adopt the image as learning data (Ota: 0051: “The similarity determination unit 13a compares an image of interest set in the first frame buffer 13c with a medical image to be determined set in the second frame buffer 13d, and calculates the similarity between the image of interest and the medical image to be determined. If the similarity is equal to or greater than a threshold, the similarity determination unit 13a transfers the medical image to be determined, which is set in the second frame buffer 13d, to the memory unit 7; if the similarity is less than the threshold, the similarity determination unit 13a discards the medical image without transferring it to the memory unit 7.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to art to implement the algorithms for performing similarity determination based upon an image selected by the user disclosed in Ota into Wenzel by discarding subsequent survey images based upon survey image template matching. The suggestion/motivation for doing so would have been “this makes it possible to apply highly accurate machine learning to the classifier, thereby enabling the construction of a classifier that can classify medical images with high accuracy.” (Ota: 0081). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Wenzel with Ota to obtain the invention as specified in claim 15.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANTHONY J RODRIGUEZ whose telephone number is (703)756-5821. The examiner can normally be reached Monday-Friday 10am-7pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANTHONY J RODRIGUEZ/Examiner, Art Unit 2672 /GANDHI THIRUGNANAM/Primary Examiner, Art Unit 2672
Prosecution Insights