Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-3, 7, 9-11, 13, 15-16, and 18-22 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Krames et al. (U.S. PGPUB 20230113384).
With respect to claim 1, Krames et al. disclose a medical image processing device (paragraph 72, FIG. 1 shows a schematic representation of an exemplary embodiment of a medical imaging system 2) comprising: a processor (paragraph 72, processing unit (e.g., processor) 4),
wherein the processor is configured to acquire a 3D organ model corresponding to an organ to be observed, which is an observation target (paragraph 72, A three dimensional model 7, derived from preoperative image data, is stored on the storage unit 6),
acquire a medical image including the organ to be observed (paragraph 74, The intraoperative image acquisition device 8 captures intraoperative image data during an intraoperative operation and transmits it to the processing unit 4),
estimate an orientation matrix representing an orientation of the organ to be observed included in the medical image in a three-dimensional coordinate system used for the 3D organ model (paragraph 90, The knowledge of the partial deformation motion may be utilized to calculate a non-rigid transformation matrix. The non-rigid transformation matrix reproduces the partial deformation motion of the organ 10 when applied to the model 7), and
control display of the 3D organ model based on the orientation matrix (paragraph 90, By applying the non-rigid transformation matrix to the model 7, the model 7 is corrected for the partial deformation of the organ 10).
With respect to claim 2, Krames et al. disclose the medical image processing device according to claim 1, wherein the estimation of the orientation matrix is performed using the medical image and a learning model that is trained with the medical image and its corresponding correct orientation matrix as an input (paragraph 91, an artificial neural network, e.g., a deep learning network, is trained with typical patterns of difference vectors 42 of the unmoving points 20, 21, 22 resulting from typical mobilization motions. The artificial neural network may also estimate the non-rigid transformation matrix).
With respect to claim 3, Krames et al. disclose the medical image processing device according to claim 1, wherein the processor is configured to perform control of rotating the 3D organ model in the three-dimensional coordinate system (paragraph 80, In order to align and overlay the preoperative model 7 depicting the undeformed organ 10 onto the intraoperative image data depicting the deformed organ 10, the model 7 has to be corrected for the partial deformation. In order to do so, the partial deformation motion has to be recognized and an appropriate correction applied to the model 7).
With respect to claim 7, Krames et al. disclose the medical image processing device according to claim 1, wherein the processor is configured to estimate the orientation matrix using the 3D organ model in addition to the medical image (paragraph 49, The use of the standardized mobilization steps can simplify the estimation of the non-rigid transformation matrix, as the number of partial deformations is limited. The model can be calculated from preoperative CT and/or MR image data. The model can be a preoperative point cloud).
With respect to claim 9, Krames et al. disclose the medical image processing device according to claim 1, wherein the estimation of the orientation matrix is performed using a plurality of the medical images obtained at different timings (paragraph 83, In order to estimate the partial deformation motion 18, the change in the geometric feature vectors assigned to the unmoving points 20, 21, 22 between the first point in time shown in FIG. 4 and the second point in time shown in FIG. 5 is calculated, paragraph 90, By calculating the difference vectors 42 of the unmoving points 20, 21, 22, the change of position of the moving points 30, 31, 32 and thus the partial deformation motion is estimated. The knowledge of the partial deformation motion may be utilized to calculate a non-rigid transformation matrix).
With respect to claim 10, Krames et al. disclose the medical image processing device according to claim 1, wherein the estimation of the orientation matrix is performed using the medical image and an orientation matrix estimated before a timing at which the current estimation is performed (paragraph 91, In this case, the estimation of the change of position of the moving points and the partial deformation motion is realized by estimating the matrix).
With respect to claim 11, Krames et al. disclose the medical image processing device according to claim 1, wherein the display of the 3D organ model is controlled in conjunction with a field of view based on the medical image (paragraph 72, The intraoperative image acquisition device 8 is for example a laparoscope or an endoscope (e.g., having a CCD or CMOS image sensor), paragraph 74, The intraoperative image acquisition device 8 captures intraoperative image data during an intraoperative operation and transmits it to the processing unit 4. The processing unit 4 calculates point clouds from individual intraoperative images or an intraoperative image stream comprising multiple intraoperative images, paragraph 77, The model 7, which may also be a point cloud, is aligned and overlaid onto the intraoperative image data to enrich the intraoperative images with additional information). It is deemed inherent such an image capture device comprises a field of view.
With respect to claim 13, Krames et al. disclose the medical image processing device according to claim 1, wherein the processor is configured to determine whether or not the estimation of the orientation matrix works from the medical image, and stop the display of the 3D organ model in a case in which the estimation of the orientation matrix does not work (paragraph 53, By calculating the non-rigid transformation matrix, the neural network can simultaneously estimate the position change and the partial deformation motion). By being able to calculate the transformation matrix, this determines the orientation matrix works.
With respect to claim 15, Krames et al. disclose the medical image processing device according to claim 1, wherein the processor is configured to recognize a resection progress of the organ to be observed from the organ to be observed, and switch a display aspect of the 3D organ model according to the resection progress (paragraph 78, During laparoscopic surgery, due to the restricted space in the abdominal cavity, mobilization techniques are required to move parts of the organ 10 into a suitable position for resection. For example, the left side 16 of the liver is folded up to better access parts of the liver not visible otherwise).
With respect to claim 16, Krames et al. disclose the medical image processing device according to claim 1, wherein the processor is configured to, in a case in which a specific movement is recognized, activate display control of the 3D organ model using the orientation matrix (paragraph 80, When performing the mobilization steps during intraoperative surgery, the organ 10 will be partially deformed. In order to align and overlay the preoperative model 7 depicting the undeformed organ 10 onto the intraoperative image data depicting the deformed organ 10, the model 7 has to be corrected for the partial deformation. In order to do so, the partial deformation motion has to be recognized and an appropriate correction applied to the model 7). Deformation of the 3D organ corresponds to a specific movement.
With respect to claim 18, Krames et al. disclose the medical image processing device according to claim 1, wherein the medical image is any one of a rigid endoscope image or an ultrasound image (paragraph 72, The intraoperative image acquisition device 8 is for example a laparoscope or an endoscope).
With respect to claim 19, Krames et al. disclose the medical image processing device according to claim 1, wherein the 3D organ model is extracted from a radiation image or an MRI image (paragraph 49, The model can be calculated from preoperative CT and/or MR image data).
With respect to claim 20, Krames et al. disclose the medical image processing device according to claim 1, wherein the organ to be observed is any one of a liver, a kidney, a pancreas, a uterus, or a nerve (paragraph 78, FIG. 2 shows a schematic representation of an organ, in this case a liver, with an organ surface 12).
With respect to claim 21, Krames et al. disclose the medical image processing device according to claim 1, wherein the processor is configured to estimate a relative positional relationship between an imaging apparatus for capturing the medical image and the organ to be observed as a display matrix representing the relative positional relationship in the three-dimensional coordinate system (paragraph 46, estimate a non-rigid transformation matrix by utilizing the position change of the at least one neighboring moving point, wherein the non-rigid transformation matrix reproduces the partial deformation motion of the organ when applied to the model), and control display of the 3D organ model based on the display matrix (paragraph 47, apply the non-rigid transformation matrix to the model to correct the model for the partial deformation of the organ).
With respect to claim 22, Krames et al. disclose a method of operating a medical image processing device, the method comprising steps, performed by a processor, of the system of claim 1; see rationale for rejection of claim 1.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Krames et al. (U.S. PGPUB 20230113384) in view of Ni et al. (U.S. PGPUB 20200098166).
With respect to claim 4, Krames et al. disclose the medical image processing device according to claim 1, wherein the processor is configured to periodically change an orientation of the 3D organ model in the three-dimensional coordinate system in a case in which a degree of overlap between structures in the organ to be observed is equal to or greater than a threshold value or in a case in which the orientation of the organ to be observed does not change for a certain period of time.
Ni et al., who also deal with visualizing a structure, disclose a method wherein the processor is configured to periodically change an orientation of the 3D organ model in the three-dimensional coordinate system in a case in which a degree of overlap between structures in the organ to be observed is equal to or greater than a threshold value (paragraph 12, the disclosed visible apparatus also detects self-occlusion among different body parts of the first 3D human body model, paragraph 63, the first joint of the plurality of joints of the first 3D human body model in the rendered state may be determined as the occluded joint when the visibility score of the first joint may be less than the defined second threshold value, paragraph 64, At 416, the rotation angle and the rotation axis of the determined set of occluded joints 316 may be updated to a defined default value in the skeleton 312 of the first 3D human body model 302 in the rendered state) or in a case in which the orientation of the organ to be observed does not change for a certain period of time. Ni et al. rotate, or change the orientation of the body part when the degree of occlusion, or overlap, reaches a threshold.
Krames et al. and Ni et al. are in the same field of endeavor, namely computer graphics.
Before the effective filing date of the claimed invention, it would have been obvious to apply the method wherein the processor is configured to periodically change an orientation of the 3D organ model in the three-dimensional coordinate system in a case in which a degree of overlap between structures in the organ to be observed is equal to or greater than a threshold value or in a case in which the orientation of the organ to be observed does not change for a certain period of time, as taught by Ni et al., to the Krames et al. system, because this would reconstruct a full 3D model of the human subject with accuracy by detecting the occluded portions (or parts) of the first 3D human body model (paragraph 12 of Ni et al.), thus visualize occluded portions.
Claim(s) 5, 12, 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Krames et al. (U.S. PGPUB 20230113384) in view of Lavi et al. (U.S. PGPUB 20220172368).
With respect to claim 5, Krames et al. disclose the medical image processing device according to claim 1. However, Krames et al. do not expressly disclose the processor is configured to determine a degree of overlap between structures in the 3D organ model, and change an orientation of the 3D organ model such that the degree of overlap between structures falls within a specific range.
Lavi et al., who also deal with medical imaging, disclose a method wherein the processor is configured to determine a degree of overlap between structures in the 3D organ model (paragraph 169, The range of direct overlap can be extended, e.g., by expanding the widths of the blood vessels in the 3-D model to be at least as large as the range at which overlap/near overlap is to be detected. One method of detecting overlap in such an embodiment (which does not necessarily require the explicit tracking of 3-D model regions as described in block 520) is to sum for each pixel the number of times that a ray extending through it from the viewing point also intersects the 3-D model. When the value is two or more, overlap is detected), and change an orientation of the 3D organ model such that the degree of overlap between structures falls within a specific range (paragraph 149, overlapping is mitigated and/or prevented by adjusting an imaging angle and/or a preferred imaging angle for a selected image, paragraph 152, a detected overlap is mitigated by adjusting an imaging angle so that instead of imaging from a position directly along ray 15, the position is offset within plane 5 enough to separate region 19 from the target region 3 in the image).
Krames et al. and Lavi et al. are in the same field of endeavor, namely computer graphics.
Before the effective filing date of the claimed invention, it would have been obvious to apply the method wherein the processor is configured to determine a degree of overlap between structures in the 3D organ model, and change an orientation of the 3D organ model such that the degree of overlap between structures falls within a specific range, as taught by Lavi et al., to the Krames et al. system, because foreshortening and/or overlap with other imaged features can interfere with measurement and/or verification (paragraph 87 of Lavi et al.), thus it would be beneficial to minimize overlap.
With respect to claim 12, Krames et al. as modified by Lavi et al. disclose the medical imaging processing device according to claim 1, wherein the processor is configured to recognize a surgical procedure from the medical image, and according to the surgical procedure, to turn on or off of display control of an angle of the 3D organ model or to adjust a display update speed (Lavi et al.: a viewing angle's optimality relies jointly on further criteria such as similarity of the viewing angle to an existing image's viewing angle, context of the image within a procedure (e.g., obtained at a particular heartbeat phase, obtained as part of a cineangiogram comprising a plurality of similar images, and/or obtained during a period of good contrast filling of blood vessels). A cineangiogram, as the term is used herein, refers to a set of angiographic images which are sufficiently similar that they can, for example, be displayed in sequence as a short “movie clip”. A cineangiogram provides information about one or more dynamic features of the imaged target, and/or features of the imaged target that are revealed by changes in imaging settings such as viewing angle, irradiation wavelength, filtering, or another parameter. A typical image frame acquisition rate for a cineangiogram is about 15-30 Hz). It would have been obvious to apply the teachings of Lavi et al. because an image viewing angle cost function (optionally including costs for different centers of the image view) is determined (paragraph 120 of Lavi et al.), thus optimizing the visualization of the displayed image.
With respect to claim 14, Krames et al. as modified by Lavi et al. disclose the medical image processing device according to claim 1, wherein the processor is configured to switch a display aspect of the 3D organ model in response to a display switching operation (Lavi et al.: paragraph 157, If there is viewing interference, then at block 508, the view is offset, for example, by adjusting the viewing angle). Adjusting the viewing angle comprises switching a display aspect. It would have been obvious to apply the teachings of Lavi et al. because this would optimize the visualization of the displayed image.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Krames et al. (U.S. PGPUB 20230113384) in view of Fidrich et al. (U.S. PGPUB 20070116334).
With respect to claim 6, Krames et al. disclose the medical image processing device according to claim 1. However, Krames et al. do not expressly disclose the processor is configured to calculate a correction amount for correcting a shape difference between a predetermined reference organ and the organ to be observed and correct the orientation matrix according to the correction amount, and control the display of the 3D organ model based on the corrected orientation matrix.
Fidrich et al., who also deal with medical imaging, disclose a method wherein the processor is configured to calculate a correction amount for correcting a shape difference between a predetermined reference organ and the organ to be observed (paragraph 32, the computer software may correct the shape of an existing three-dimensional anatomical object from planar curves).
Krames et al. and Fidrich et al. are in the same field of endeavor, namely computer graphics.
Before the effective filing date of the claimed invention, it would have been obvious to apply the method wherein the processor is configured to calculate a correction amount for correcting a shape difference between a predetermined reference organ and the organ to be observed, as taught by Fidrich et al., to the Krames et al. to correct the orientation matrix according to the correction amount, and control the display of the 3D organ model based on the corrected orientation matrix, because this would allow a user to be more efficient and effective in diagnosing and treating medical conditions (paragraph 11 of Fidrich et al.).
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Krames et al. (U.S. PGPUB 20230113384) in view of Wang et al. (U.S. PGPUB 20210150673).
With respect to claim 8, Krames et al. disclose the medical image processing device according to claim 1. However, Krames et al. do not expressly disclose the processor is configured to apply a process of smoothing a change in angle of the 3D organ model in a time direction to the orientation matrix.
Wang et al., who also deal with animating a body, disclose a method wherein the processor is configured to apply a process of smoothing a change in angle of the 3D organ model in a time direction to the orientation matrix (paragraph 46, In S5, the method includes performing image stabilization on the video frame sequence based on the smooth rotation matrix sequence, paragraph 47, Thereby, the motion trajectory of the human body can be smoothed).
Krames et al. and Wang et al. are in the same field of endeavor, namely computer graphics.
Before the effective filing date of the claimed invention, it would have been obvious to apply the method wherein the processor is configured to apply a process of smoothing a change in angle of the 3D organ model in a time direction to the orientation matrix, as taught by Wang et al., to the Krames et al. system, because this would ensure that the output video is more stable, the motion jitter can be effectively removed, and the human motion in the video is made closer to the actual motion, thereby improving the video quality (paragraph 47 of Wang et al.).
Claim(s) 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Krames et al. (U.S. PGPUB 20230113384) in view of Barasofsky et al. (U.S. PGPUB 20240386572).
With respect to claim 17, Krames et al. disclose the medical image processing device according to claim 2. However, Krames et al. do not expressly disclose the processor is configured to receive a user correction regarding a display direction of the 3D organ model, and correction information regarding the user correction is used for training the learning model or for determining a correction amount for correcting a shape difference between a predetermined reference organ and the organ to be observed.
Barasofsky et al., who also deal with medical imaging, disclose a method wherein the processor is configured to receive a user correction regarding a display direction of the 3D organ model (paragraph 39, the user input is used to quickly and efficiently retrain parts of the deep network (e.g., neural network) and minimally adjust the network weights to accommodate the user feedback in a minimal amount of time), and correction information regarding the user correction is used for training the learning model or for determining a correction amount for correcting a shape difference between a predetermined reference organ and the organ to be observed (paragraph 39, automatically segmenting lesions from three-dimensional (3D) models of anatomical structures using deep learning (e.g., machine learning) and correcting the segmentation using interactive user feedback in the form of annotations of inaccurate segmentation points located inside and outside the original automatic segmentation). Correcting the shape difference is from performing segmentation.
Krames et al. and Barasofsky et al. are in the same field of endeavor, namely computer graphics.
Before the effective filing date of the claimed invention, it would have been obvious to apply the method wherein the processor is configured to receive a user correction regarding a display direction of the 3D organ model, and correction information regarding the user correction is used for training the learning model or for determining a correction amount for correcting a shape difference between a predetermined reference organ and the organ to be observed, as taught by Barasofsky et al., to the Krames et al. system, because training the deep network provides for more accurate segmentation for each use-case (e.g., biopsy, resection, etc.), thereby reducing the amount of time a clinician must correct or otherwise modify the automatic segmentation (paragraph 39 of Barasofsky et al.).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW GUS YANG whose telephone number is (571)272-5514. The examiner can normally be reached M-F 9 AM - 5:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571)272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW G YANG/Primary Examiner, Art Unit 2614
9/30/26
Prosecution Insights