DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed 07/14/2026 has been entered. Claim 25 has been canceled. Claims 1-16 and 18-20 remain pending in the application.
Response to Arguments
Applicant’s arguments, see Remarks, filed 07/14/2026, with respect to the rejections of claim 1 under 35 U.S.C. 103 as being unpatentable over by Duindam in view of Miyamoto have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, new grounds of rejections are made in view of Korsten (US20120321154) and Klein (US20160000303).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 7-8, 10, 13, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Duindam (US20190320878) in view of Korsten (US20120321154) and Klein (US20160000303).
Regarding claim 1, Duindam teaches a system (100) (Fig. 1, [0029]) comprising:
a processor (112) ([0040], [0118]);
a display (110) (Fig. 1, [0034]); and
a memory (112) having computer readable instructions stored thereon that, when executed by the processor ([0040], [0118]), cause the system (100) to:
receive intra-operative three-dimensional image data from an imaging system (330), wherein a portion of the intra-operative three-dimensional image data corresponds to an instrument disposed in a patient anatomy (502) (Fig. 3-5, Abstract, [0006], [0057], “A three-dimensional imaging system 330 is arranged near the patient P to obtain three-dimensional images of the patient while the elongate device 310 is extended within the patient”, [0062], “intraoperative three-dimensional image data of a patient anatomy is obtained from an imaging system”, [0063], “the three-dimensional image data may include data representing at least a portion of a medical instrument 304 positioned within the anatomy of the patient P as shown in FIGS. 3A-B and 4A-D”).
However, Duindam fails to teach causing the system to: generate two-dimensional projection image data from the intra-operative three-dimensional image data; and identify, within the two-dimensional projection image data, a three-dimensional location of a portion of the instrument.
In an analogous image-guided procedure field of endeavor, Korsten teaches such a feature. Korsten teaches obtaining an image data volume in which at least a portion of a needle (540) has been inserted, with a probe (520) configured for obtaining three-dimensional measurements ([0012], [0134]), therefore teaching wherein the 3D image data volume is intraoperative (needle has been inserted). Korsten teaches wherein the disclosed method may be applied to multidimensional image data including CT, MRI, and ultrasound ([0064]). Korsten teaches projecting the image data volume onto a projection plane, thereby teaching generating two-dimensional image data from the intra-operative three-dimensional image data ([0022-0023], [0026-0028], [0140-0143]). Korsten teaches reducing the three-dimensional data set to two dimensions via projection ([0147], [0177], [0187]). Korsten teaches a step 420, the 3D image data volume is obtained of tissue comprising the needle, and in step 450 the data volume is projected onto a projection image, thereby generating a projection image ([0169-0170], [0187]). Korsten further teaches a projected pixel with intensity above a predetermined threshold is assumed to correspond to the needle plane tip, and once the tip is found, the needle line can be determined ([0140]). Korsten teaches projecting the image data volume onto a first projection plane and onto a second projection plane, allowing for the 3D needle line can be found as the intersection of the two needle planes ([0039-0040]). Korsten teaches wherein the needle line is localized in 3d image data by projecting the 3D image data onto a projection plane (creating the projection image) ([0140], “For the localization of the needle line (also referred to as needle axis) in a 3D image data one can use parallel projection”, [0142], “To determine the position of the needle in 3D US data, the parallel projection can be used”), thereby teaching wherein a 3D location of a portion of an instrument is identified within 2D projection image data. Korsten also teaches determining the three dimensional location of the needle from the needle line and needle plane ([0036-0037], “It is also possible to determine the three dimensional location of the needle more exactly”, [0171], “The method may also determine the location of the needle and/or needle line”).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to generate 2D projection image data from the intraoperative 3D image data and to identify the 3D location of a portion of the instrument within the 2D projection image as taught by Korsten ([0022-0023], [0026-0028], [0036-0037], [0039-0040], [0140-0143], [0171], [0187]). By projecting the instrument from a 3D volume to a 2d projection, the problem of finding (locating) the instrument in 3D is simplified considerably as recognized by Korsten ([0023]). Moreover, Korsten teaches wherein the method may be applied to CT and other 3D modalities ([0064]). By locating the instrument in three-dimensional space, the instrument may predictably be more accurately tracked and thus guided by a procedure.
However, the modified combination noted above fails to teach causing the system to: display the two-dimensional projection image data on the display.
In an analogous image-guided procedure field of endeavor, Klein teaches such a feature. Klein teaches a method and system for navigating through a patient’s bronchial tree using a bronchoscope, guided based on a three-dimensional model generate from CT image data (Abstract, [0003]). Klein teaches calculating a maximum intensity projection (MIP) from 3D data (Fig. 3, [0006], [0008], [0043]). Klein teaches wherein the 3D data comprises CT image data ([0041-0042]). Klein teaches wherein the MIP algorithm may be tuned to highlight lesion-density tissue and filter out most other densities in the CT volume ([0043]), therefore confirming the MIP is a projection image generated from a 3D image (CT volume). Klein further teaches displaying (402) the maximum intensity projection at step s318 (Figs. 3-4, [0011], [0043]). Moreover, Klein teaches wherein the system includes a display (206) for displaying (Fig. 2, [0037], [0055]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam in view of Korsten to display the projection image on a display as taught by Klein (Figs. 2-4, [0006], [0011], [0037], [0043]). Generation and display of the projection image (MIP) may allow for a user to aim at a center of a target, thus guiding the user as recognized by Klein ([0043]). Moreover, displaying the projection may predictably allow a user to confirm an instrument’s position.
Regarding claim 2, Duindam in view of Korsten and Klein teaches the invention as claimed above in claim 1.
Duindam further teaches wherein the computer readable instructions, when executed by the processor (112) ([0118]), cause the system (100) to:
segment, based on the identified three-dimensional location of the portion of the instrument, the portion of the intra-operative three-dimensional image data corresponding to the instrument ([0007], [0028], “As used herein, the term “shape” refers to a set of poses, positions, or orientations measured along an object” [0066], “…the shape information may be used to identify a search region or search area that includes the voxels corresponding to the medical instrument. By using the shape information, the segmentation process employed to segment the medical instrument from the rest of the three-dimensional image data”, [0071], wherein shape and thus position is measured in a 3D reference frame and thus comprises a 3D location, [0095], “The shape data may be used to help find the representation of the medical instrument in the image data. Accordingly, the computing system may perform segmentation of the image data to identify those portions of the intraoperative images that correspond to the medical instrument”).
Regarding claim 7, Duindam in view of Korsten and Klein teaches the invention as claimed above in claim 1.
Duindam further teaches wherein the computer readable instructions, when executed by the processor (112) ([0118]), cause the system (100) to:
generate a model of the patient anatomy based on pre-operative image data ([0084], “a preoperative model of the patient anatomy is generated from the preoperative image data”); and
update the model based on the intra-operative three-dimensional image data ([0062], “Thus, the intraoperative image data may correspond to two-dimensional, three-dimensional, or four-dimensional (including e.g., time based or velocity based information) images”, [0111], “the computing system may update the preoperative model based on the intraoperative image data”).
Regarding claim 8, Duindam in view of Korsten and Klein teaches the invention as claimed above in claim 7.
Duindam further teaches wherein updating the model comprises revising a location of an anatomical target ([0111], “update the preoperative model based on the intraoperative image data… the preoperative model includes a patient anatomy, a target such as a tumor… anatomical structures and/or the instrument may have shifted from a location represented in the preoperative model…The computing system may update a size, a location, and/or another property of the target based on the intraoperative image data… In such examples, the size, shape, and/or relative location of the other anatomical structures may also up updated in the preoperative model”).
Regarding claim 10, Duindam in view of Korsten and Klein teaches the invention as claimed above in claim 1.
Duindam further teaches wherein the computer readable instructions, when executed by the processor (112) ([0118]), cause the system (100) to:
generate a model of the patient anatomy based on pre-operative image data ([0084], “a preoperative model of the patient anatomy is generated from the preoperative image data”); and
register the model to the intra-operative three-dimensional image data based at least in part on a location of an anatomical target in each of the model and the intra-operative three-dimensional image data ([0104], “the preoperative model is registered to the intraoperative image reference frame”, [0105], “the preoperative reference frame may be registered to the intraoperative reference frame using feature-based registration to compare the respective models and/or images, and the computing system locates corresponding fiducial features in the intraoperative model/images and the preoperative model/images. Fiducial features include both artificial and anatomical features with a distinguishing characteristic”, wherein fiducial features including anatomical features comprise an anatomical target).
Regarding claim 13, Duindam in view of Korsten and Klein teaches the invention as claimed above in claim 1.
Duindam further teaches wherein the imaging system (330) comprises a cone-beam computed tomography system ([0057], “In some embodiments, the imaging system 330 includes a mobile rotational imaging element such as that of a mobile C-arm cone-beam CT imaging system for capturing intraoperative 3D images and/or fluoroscopic 2D images”).
Regarding claim 19, Duindam in view of Korsten and Klein teaches the invention as claimed above in claim 1.
However, Duindam fails to teach wherein the computer readable instructions, when executed by the processor, cause the system to: select a region of interest within the intra-operative three dimensional data based on the identified three-dimensional location of the portion of the instrument; generate two-dimensional projection image data from the selected region of interest within the intra-operative three-dimensional image data; and display the two-dimensional projection image data from the selected region of interest on the display.
In an analogous image-guided procedure field of endeavor, Klein teaches such a feature. Klein teaches a method and system for navigating through a patient’s bronchial tree using a bronchoscope, guided based on a three-dimensional model generate from CT image data (Abstract, [0003]). Klein teaches calculating a maximum intensity projection (MIP) from 3D data (Fig. 3, [0006], [0008], [0043]). Klein teaches wherein the 3D data comprises CT image data ([0041-0042]). Klein teaches wherein the MIP algorithm may be tuned to highlight lesion-density tissue and filter out most other densities in the CT volume ([0043]), therefore confirming the MIP is a projection image generated from a 3D image (CT volume). Moreover, Klein teaches wherein the MIP is calculated “in a limited range from the distal tip 93” ([0016], [0043]). The limited range defines the sub-volume or region of interest of the 3D data that the projection image is generated from. Klein teaches wherein the limited range may be 35 mm from the distal tip (93) ([0043]). Klein then teaches displaying the MIP after calculating the MIP in a limited the range from the distal tip (93) (Fig. 3, [0043]). Klein therefore teaches wherein the limited range, or region of interest, is based on a location of a portion of an instrument (the tip) and thus wherein the two-dimensional projection image data (MIP) is generated and displayed from the region of interest (limited range).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to select a region of interest within the 3D image data based on its location and to generate and display the 2D projection image from the region of interest as taught by Klein (Fig. 3, [0016], [0041-0043]). Duindam and Korsten teaches wherein the 3D image data is acquired intraoperatively, and Korsten teaches wherein the location of the instrument is identified in 3D. Therefore, the combination of Duindam in view of Korsten further modified by Klein would result wherein the MIP generated by Klein would be of the intraoperative 3D image and wherein limited range (region of interest) about the tip is based on an identified location of the tip. By using the range about the already identified location, it would predictably further reduce processing load, which Klein already suggests regarding limiting the range of the MIP algorithm/calculation ([0043]).
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Duindam (US20190320878) in view of Korsten (US20120321154) and Klein (US20160000303) as applied to claim 2 above, and further in view of Simon (US20110268248).
Regarding claim 3, Duindam in view of Korsten and Klein teaches the invention as claimed above in claim 2.
Duindam further teaches wherein the computer readable instructions, when executed by the processor (112) ([0118]), cause the system (100) to:
register the intra-operative three-dimensional image data to shape data from the instrument by comparing the shape data to the portion of the intra-operative three-dimensional data corresponding to the instrument (Abstract, [0007], [0069], wherein the segmented shape comprises the portion of the intra-operative three-dimensional image data corresponding to the instrument, [0071-0072], “At an operation 508, the segmented shape of the medical instrument may be registered with the shape data obtained from the medical instrument”).
However, Duindam fails to teach displaying a two-dimensional projection of the shape data on the two-dimensional projection image data.
In an analogous imaged guide surgery field of endeavor ([0003]), Simon teaches such a feature. Simon teaches acquiring fluoroscopic images with a fluoroscopic C-arm x-ray imaging device (100) (Fig. 1, [0041], [0044]). Simon further teaches a surgical instrument (140) which is locationally tracked three dimensionally ([0054]). Simon teaches calculating a projection of the instrument (140) into each fluoroscopic image as the instrument is moved ([0061]). Simon teaches wherein the instrument (140) is overlaid on the fluoroscopic images and that the graphical representation of the instrument (140) represents where the actual surgical instrument would appear ([0061]). Because the projection/graphical representation of the instrument includes the instrument’s shape, Simon therefore teaches displaying a two-dimensional projection of shape data on two-dimensional projection image data (fluoroscopic images).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to project or overlay the shape of the instrument onto the 2D projection images as taught by Simon ([0061]). Having the instrument be overlaid onto the 2D projection images in real-time as the instrument is moved may predictably help an operator guide said instrument during surgery.
Claims 4-5 are rejected under 35 U.S.C. 103 as being unpatentable over Duindam (US20190320878) in view of Korsten (US20120321154), Klein (US20160000303), and Simon (US20110268248) as applied to claim 3 above, and further in view of Ekin (US20210012526).
Regarding claim 4, Duindam in view of Korsten, Klein, and Simon teaches the invention as claimed above in claim 3.
However, Duindam fails to teach wherein the computer readable instructions, when executed by the processor, cause the system to: identify one or more regions of the shape data that is misaligned with the portion of the intra-operative three-dimensional image data corresponding to the instrument; and display the one or more regions with at least one visual property different than one or more regions of the shape data that is aligned with the portion of the intra-operative three-dimensional image data corresponding to the instrument.
In an analogous image guided surgery field of endeavor ([0001]), Ekin teaches such a feature. Ekin teaches registering the shape of an interventional instrument (33) from optical shape sensing with a shape of the interventional instrument (33) in an image (Abstract, [0045]). Ekin further teaches an accuracy determination unit (12) configured to determine the accuracy of alignment based on the shape and images of the instrument ([0050]). Ekin teaches the accuracy determination unit (12) is configured to determine the alignment accuracy for different regions of the instrument (33) as defined by the position and shape and also the respective images, thereby determining aligned and misaligned regions ([0050]). Ekin teaches visualizing the accuracy of alignment and outputting it to a display (Fig. 6, [0050]). Ekin teaches the misaligned portions may be highlighted using another (different) visual property such as color ([0050]). Ekin teaches and shows in figure 6 wherein the misaligned segment or portion of the instrument is indicated by broken lines (Fig. 6, [0050]). An annotated figure 6 is reproduced below for clarity.
PNG
media_image1.png
310
444
media_image1.png
Greyscale
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to visualize instrument alignment/registration accuracy as taught by Ekin (Fig. 6, [0050]). By visualizing misalignment, a user may subsequently take actions to correct or update the alignment and registration as recognized by Ekin ([0066]), thereby improving registration accuracy.
Regarding claim 5, Duindam in view of Korsten, Klein, Simon, and Ekin teaches the invention as claimed above in claim 4.
However, Duindam fails to teach wherein the at least one visual property comprises at least one of a color, a brightness, a linetype, a pattern, or an opacity.
In an analogous image guided surgery field of endeavor ([0001]), Ekin teaches such a feature. Ekin teaches registering the shape of an interventional instrument (33) from optical shape sensing with a shape of the interventional instrument (33) in an image (Abstract, [0045]). Ekin further teaches an accuracy determination unit (12) configured to determine the accuracy of alignment based on the shape and images of the instrument ([0050]). Ekin teaches the accuracy determination unit (12) is configured to determine the alignment accuracy for different regions of the instrument (33) as defined by the position and shape and also the respective images, thereby determining aligned and misaligned regions ([0050]). Ekin teaches visualizing the accuracy of alignment and outputting it to a display (Fig. 6, [0050]). Ekin teaches the misaligned portions may be highlighted using another (different) visual property such as color, intensities, and symbols ([0018], [0050]). Ekin teaches and shows in figure 6 wherein the misaligned segment or portion of the instrument is indicated by broken lines (Fig. 6, [0050]). An annotated figure 6 is reproduced below for clarity.
PNG
media_image1.png
310
444
media_image1.png
Greyscale
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to have the adjusted visual property comprise a color, linetype, brightness (intensity) or pattern as taught by Ekin (Fig. 6, [0018], [0050]). These visual properties may be used to indicate or visualize differences (alignment accuracy) and thus indicate misaligned and aligned portions to a user as recognized by Ekin (Fig. 6, [0018], [0050]).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Duindam (US20190320878) in view of Korsten (US20120321154), Klein (US20160000303), and Simon (US20110268248) as applied to claim 3 above, and further in view of Adebar (US20200297442).
Regarding claim 6, Duindam in view of Korsten, Klein, and Simon teaches the invention as claimed above in claim 6.
However, Duindam fails to teach wherein the computer readable instructions, when executed by the processor, cause the system to: receive a user input; and based on the user input, adjust at least one of a position or rotation of the shape data with respect to the intra-operative three-dimensional image data.
In an analogous image guided surgery field of endeavor, Adebar teaches such a feature. Adebar teaches tracking a medical instrument (104) and registering and displaying the medical instrument together with surgical images ([0039]). Adebar teaches wherein a user may update an existing registration between the medical instrument and the actual patient anatomy (e.g. images of the patient anatomy) because movement of the patient may cause an existing registration to become unreliable and unsuitable ([0099]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to have a user update the existing registration between the instrument and the images of the patient anatomy as taught by Adebar ([0099]). Movement of a patient may cause an existing registration to become unreliable and unsuitable for use as recognized by Adebar ([0099]), thereby needing an update. Modifying Duindam to have a user update the registration of a medical instrument with the patient anatomy would predictably result in receiving a user input and based on the user input, adjusting a position or a rotation (e.g. orientation) of shape data that is used to represent the instrument with respect to the intra-operative 3D image data that the instrument is registered to.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Duindam (US20190320878) in view of Korsten (US20120321154) and Klein (US20160000303) as applied to claim 8 above, and further in view of Tata (US20220354380).
Regarding claim 9, Duindam in view of Korsten and Klein teaches the invention as claimed above in claim 8; Duindam teaches wherein updating the model comprises revising a location of an anatomical target ([0111]).
Duindam further teaches wherein the computer readable instructions, when executed by the processor (112) ([0118]), cause the system (100) to:
generate a navigation path through the patient anatomy based on the pre-operative image data ([0085], “the operator can generate a navigation path through the anatomic passageways in the preoperative model to guide the medical instrument to a target… a navigation path is determined through the anatomic passageways of the preoperative model to guide the medical instrument to the target”).
However, Duindam fails to explicitly teach wherein updating the model comprises revising a navigation path to correspond to the revised location of the anatomical target.
In an analogous image guided procedure field of endeavor, Tata teaches such a feature. Tata teaches displaying an anatomy model (12) of a patient on a display (68) ([0031]). Tata teaches updating the anatomy model (12) based on camera and position sensor data ([0039]). Tata teaches wherein the updating may include rerouting [i.e. revising] navigation paths to a desired destination ([0039]). Tata further teaches displaying the navigation path in the updated anatomy model ([0010], [0053]) Tata teaches revising a navigation path (B →180) to correspond to a revised location (288 → 188b) of an anatomical target (Fig. 7, [0056-0057]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to revise the navigation path to correspond to a revised location of an anatomical target as taught by Tata (Fig. 7, [0039], [0056-0057]). Duindam teaches revising the target location in the pre-operative patient model ([0111]). Therefore it’d be obvious to subsequently revise the navigation route to the revised target location as explicitly taught by Tata (Fig. 7, [0039], [0056-0057]). By revising the navigation route to correspond to the revised location, a user may accurately/reliably navigate towards the correct target location.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Duindam (US20190320878) in view of Korsten (US20120321154) and Klein (US20160000303) as applied to claim 1 above, and further in view of Li (US20080283771).
Regarding claim 11, Duindam in view of Korsten and Klein teaches the invention as claimed above in claim 1.
However, Duindam fails to teach wherein the computer readable instructions, when executed by the processor, cause the system to: extract a three-dimensional boundary of an anatomical target from a model of the patient anatomy generated based on pre-operative image data; and display a projection of the three-dimensional boundary of the anatomical target on the two-dimensional projection image data.
In an analogous image guided surgery field of endeavor, Li teaches such a feature. Li teaches acquiring 4D image data of an anatomy of interest of a patient (110) with an intracardiac echocardiography (ICE) catheter (105) ([0050]) Li teaches extracting a surface model [i.e. a 3D boundary] of the imaged anatomy from the generated 4D ICE model ([0050]). Li further teaches acquiring fluoroscopic imaging data [i.e. 2D projection image data] ([0051]). Li teaches wherein the surface model may be projected in combination with the 2D fluoroscopic imaging data ([0051]). Li therefore teaches extracting a 3D boundary (surface model) of an anatomical target from a model of the patient anatomy; and displaying a projection of the boundary on 2D projection image data (fluoroscopic image).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to extract a surface from a anatomic model and to project that surface onto a 2D projection image as taught by Li ([0050-0051]). Projecting the 3D boundary of an anatomic target onto 2D projection image may improve a user’s visual understanding of the procedure, e.g. improve understanding of depth or 3D spatial relationship between the target and an instrument.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Duindam (US20190320878) in view of Korsten (US20120321154), Klein (US20160000303), and Li (US20080283771) as applied to claim 11 above, and further in view of Jiang (US20220304640).
Regarding claim 12, Duindam in view of Korsten, Klein, and Li teaches the invention as claimed above in claim 11.
However, Duindam fails to teach wherein the computer readable instructions, when executed by the processor, cause the system to: receive an input from a user to manipulate at least one of a location or a dimension of the projection of the three-dimensional boundary.
In an analogous image guided procedure field of endeavor, Jiang teaches such a feature. Jiang teaches extracting a contour of an anatomic target ([0041], [0043]). Jiang teaches registering and superimposing [i.e. projecting] the contour onto an image ([0044], [0046]). Jiang teaches a user is allowed to adjust a position of the contour that is to be superimposed onto the image ([0050]). Jiang teaches receiving a shift instruction from a user and adjusting the position of the contour according to the shift instruction ([0050], [0060]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to allow a user to adjust a position of the projected boundary as taught by Jiang ([0050], [0060]). By allowing a user to adjust the position of the boundary, an alignment or registration of the boundary to the image may be made more accurate as recognized by Jiang ([0050]).
Claims 14-16 are rejected under 35 U.S.C. 103 as being unpatentable over Duindam (US20190320878) in view of Korsten (US20120321154) and Klein (US20160000303) as applied to claim 1 above, and further in view of Mizuta (US20170319163).
Regarding claim 14, Duindam in view of Korsten and Klein teaches the invention as claimed above in claim 1.
However, Duindam fails to teach wherein the two-dimensional projection image data comprises at least one maximum intensity projection of the intra-operative three-dimensional image data based on voxel intensity values.
In an analogous three-dimensional imaging field of endeavor, Mizuta teaches such a feature. Mizuta teaches acquiring three-dimensional volume data ([0053]). Mizuta teaches a MIP-axis setting unit (20) sets a predetermined axis of a subject and denotes an axis orthogonal to each of directions for projecting an MIP image with respect to the three-dimensional volume data ([0053]). Mizuta teaches a MIP image is a maximum intensity projection image in which the maximum pixel (voxel) projection path is set as the luminance/intensity value ([0003], the MIP is generated from a 3D volume and so the “pixel” is a voxel). Mizuta teaches generating and displaying a first MIP image (T1) with respect to a first projection direction and a second MIP image (T2) with respect to a second projection direction orthogonal to the first direction (Figs. 7B & 9, [0004], [0074], [0076]). As shown in figure 9, Mizuta teaches wherein displaying 2D projection image data comprises displaying a plurality of views with different view orientations, the orientations being orthogonal to each other.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to have the projection image comprise a MIP image based on voxel intensity as taught by Mizuta (Figs. 7B & 9, [0003-0004], [0074], [0076]). The MIP images may be useful for approximating a position in which certain objects are such as radioactive pharmaceuticals as recognized by Mizuta ([0004-0005]).
Regarding claim 15, Duindam in view of Korsten, Klein, and Mizuta teaches the invention as claimed above in claim 14.
However, Duindam fails to teach wherein displaying the two-dimensional projection image data on the display comprises displaying a plurality of views with different view orientations.
In an analogous three-dimensional imaging field of endeavor, Mizuta teaches such a feature. Mizuta teaches acquiring three-dimensional volume data ([0053]). Mizuta teaches a MIP-axis setting unit (20) sets a predetermined axis of a subject and denotes an axis orthogonal to each of directions for projecting an MIP image with respect to the three-dimensional volume data ([0053]). Mizuta teaches a MIP image is a maximum intensity projection image in which the maximum pixel (voxel) projection path is set as the luminance/intensity value ([0003], the MIP is generated from a 3D volume and so the “pixel” is a voxel). Mizuta teaches generating and displaying a first MIP image (T1) with respect to a first projection direction and a second MIP image (T2) with respect to a second projection direction orthogonal to the first direction (Figs. 7B & 9, [0004], [0074], [0076]). As shown in figure 9, Mizuta teaches wherein displaying 2D projection image data comprises displaying a plurality of views with different view orientations, the orientations being orthogonal to each other.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to display a plurality of maximum intensity projection (MIP) images with different view orientations as taught by Mizuta (Figs. 7B & 9, [0003-0004], [0074], [0076]). By referring to MIP images acquired in different projection directions, such as orthogonal to each other, an operator may know an approximating position of certain objects such as radioactive pharmaceuticals as recognized by Mizuta (Fig. 9, [0004-0005]).
Regarding claim 16, Duindam in view of Korsten, Klein, and Mizuta teaches the invention as claimed above in claim 15.
However, Duindam fails to teach wherein the plurality of views comprise at least a first view and a second view, wherein an orientation of the first view is orthogonal to an orientation of the second view.
In an analogous three-dimensional imaging field of endeavor, Mizuta teaches such a feature. Mizuta teaches acquiring three-dimensional volume data ([0053]). Mizuta teaches a MIP-axis setting unit (20) sets a predetermined axis of a subject and denotes an axis orthogonal to each of directions for projecting an MIP image with respect to the three-dimensional volume data ([0053]). Mizuta teaches a MIP image is a maximum intensity projection image in which the maximum pixel (voxel) projection path is set as the luminance/intensity value ([0003], the MIP is generated from a 3D volume and so the “pixel” is a voxel). Mizuta teaches generating and displaying a first MIP image (T1) with respect to a first projection direction and a second MIP image (T2) with respect to a second projection direction orthogonal to the first direction (Figs. 7B & 9, [0004], [0074], [0076]). As shown in figure 9, Mizuta teaches wherein displaying 2D projection image data comprises displaying a plurality of views with different view orientations, the orientations being orthogonal to each other.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to display a plurality of maximum intensity projection (MIP) images with orthogonal view directions as taught by Mizuta (Figs. 7B & 9, [0003-0004], [0074], [0076]). By referring to MIP images acquired in orthogonal projection directions, an operator may know an approximating position of certain objects such as radioactive pharmaceuticals as recognized by Mizuta (Fig. 9, [0004-0005]).
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Duindam (US20190320878) in view of Korsten (US20120321154), Klein (US20160000303), and Mizuta (US20170319163) as applied to claim 16 above, and further in view of Sra (US20180317864).
Regarding claim 18, Duindam in view of Korsten, Klein, and Mizuta teaches the invention as claimed above in claim 16.
However, Duindam fails to teach wherein identifying the three-dimensional location of the portion of the instrument comprises: receiving a first user input indicating a first two-dimensional location of the portion of the instrument in the first view; and receiving a second user input indicating a second two-dimensional location of the portion of the instrument in the second view.
In an analogous image guided surgery field of endeavor, Sra teaches such a feature. Sra teaches a fluoroscopy system (10) and acquiring an image from a first angle (View 1) and an image from a second angle (View 2) (Figs. 4-1 & 4-2, [0037], [0056], [0067]). Sra teaches wherein a user may place an orientation marker 75-1 on the View 1 image and an orientation marker 75-2 on the View 2 image, the orientation markers being placed on an instrument comprising a catheter (73) and cryoballoon (70) (Figs. 4-1 & 4-2, [0068]). Sra teaches the 2D coordinates of the orientation markers 75-1 and 75-2 are then acquired by the computer system ([0068]). Sra teaches determining the 3D location and orientation of the cryoballoon (70) based on the orientation markers 75-1 and 75-2 from the View 1 and View 2 images ([0071], [0094]). Sra therefore teaches wherein identifying the 3D location of a portion of an instrument (i.e. cryoballoon 70) comprises receiving a first user input indicating a first two-dimensional location (orientation marker 75-1) of the portion of the instrument in the first view (View 1); and receiving a second user input indicating a second two-dimensional location (orientation marker 75-2) of the portion of the instrument in the second view (View 2) (Figs. 4-1 & 4-2, [0067-0068], [0094]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to additionally identify the 3D location of the instrument by having a user select two 2D points from two images of different views as taught by Sra (Figs. 4-1 & 4-2, [0067-0068], [0094]). By additionally determining the 3D location of the instrument in this manner, a separate position sensor isn’t required, and a model of the instrument may be placed at the determined 3D coordinates to aid in visualization as recognized by Sra (Abstract, [0100]).
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Duindam (US20190320878) in view of Korsten (US20120321154) and Klein (US20160000303) as applied to claim 1 above, and further in view of Sra (US20180317864) and Bar Shalev (US7024028).
Regarding claim 20, Duindam in view of Korsten and Klein teaches the invention as claimed above in claim 1.
However, Duindam fails to teach wherein displaying the two-dimensional image data on the display comprises displaying a first view with a view plane having a view plane orientation.
In an analogous image-guided procedure field of endeavor, Klein teaches such a feature. Klein teaches a method and system for navigating through a patient’s bronchial tree using a bronchoscope, guided based on a three-dimensional model generate from CT image data (Abstract, [0003]). Klein teaches calculating a maximum intensity projection (MIP) from 3D data (Fig. 3, [0006], [0008], [0043]). Klein teaches wherein the 3D data comprises CT image data ([0041-0042]). Klein teaches wherein the MIP algorithm may be tuned to highlight lesion-density tissue and filter out most other densities in the CT volume ([0043]), therefore confirming the MIP is a projection image generated from a 3D image (CT volume). Klein further teaches displaying (402) the maximum intensity projection at step s318 (Figs. 3-4, [0011], [0043]). Displaying a 2D projection image implies displaying the image in a first view with a view plane having a view plane orientation, wherein the view plane is the plane of the 2D image and the view plane orientation is arbitrary.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to generate and display a 2D projection image in a first view with a view plane having a view plane orientation as taught by Klein (Figs. 3-4, [0011], [0043]). Generation and display of the projection image (MIP) may allow for a user to aim at a center of a target, thus guiding the user as recognized by Klein ([0043]). Moreover, displaying the projection may predictably allow a user to confirm an instrument’s position.
However, the modified combination noted above fails to teach wherein identifying the three-dimensional location of the portion of the instrument comprises: receiving a user input indicating a location of the portion of the instrument in the first view, wherein the indicated location is identifiable by a first coordinate value and a second coordinate value of respective orthogonal first and second axes within the view plane.
In an analogous image guided surgery field of endeavor, Sra teaches such a feature. Sra teaches a fluoroscopy system (10) and acquiring an image from a first angle (View 1) and an image from a second angle (View 2) (Figs. 4-1 & 4-2, [0037], [0056], [0067]). Sra teaches wherein a user may place an orientation marker 75-1 on the View 1 image and an orientation marker 75-2 on the View 2 image, the orientation markers being placed on an instrument comprising a catheter (73) and cryoballoon (70) (Figs. 4-1 & 4-2, [0068]). Sra teaches the 2D coordinates of the orientation markers 75-1 and 75-2 are then acquired by the computer system ([0068]). Sra teaches determining the 3D location and orientation of the cryoballoon (70) based on the orientation markers 75-1 and 75-2 from the View 1 and View 2 images ([0071], [0094]). Sra therefore teaches receiving user input indicating a location of a portion of an instrument in a first (and second) view, wherein the indicated location is identifiable by a first coordinate value and a second coordinate value of respective orthogonal first and second axes with in a view plane (a 2D location; a location in 2 dimensional space).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to receive user input indicating a 2D location of the instrument in first and second views as taught by Sra (Figs. 4-1 & 4-2, [0067-0068], [0094]). By additionally determining the 3D location of the instrument in this manner, a separate position sensor isn’t required, and a model of the instrument may be placed at the determined 3D coordinates to aid in visualization as recognized by Sra (Abstract, [0100]).
However, the modified combination noted above fails to teach identifying a third coordinate value associated with the indicated location by retrieving a stored coordinate value of a voxel producing a maximum intensity at the indicated location of the portion of the instrument in the first view, wherein the third coordinate value represents an axis orthogonal to the view plane orientation.
In an analogous identification of a location of a region of interest field of endeavor, Bar Shalev teaches such a feature. Bar Shalev teaches generating or assembling a maximum intensity projection (MIP) frame (51) based on first and second dimensions and based on each of the maximum intensity pixels in the frame (Column 4 lines 33-39). Bar Shalev further teaches the third dimension of each pixel having a maximum intensity along each ray is stored in a virtual frame (52) (Column 4 lines 39-45). Bar Shalev teaches when clicking on a 2D location, for example a X, Y location, a “fetch” order may retrieve the third coordinate in the Z dimension (Column 2 lines 25-32, Column 2 lines 44-52, Column 4 lines 41-45). Bar Shalev therefore teaches identifying a third coordinate value (Z-depth direction) associated with an indicated location (X, Y location) by retrieving a stored coordinate value of a voxel producing maximum intensity at the indicated location (X, Y) location in a first view (X, Y view), wherein the third coordinate represents an axis orthogonal to the view plane direction (Z-axis). Sra above teaches wherein the indicated or clicked on location is a portion of an instrument.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to store and retrieve the third dimension or Z-axis dimension coordinate having the maximum intensity as taught by Bar Shalev (Column 2 lines 25-32, Column 2 lines 44-52, Column 4 lines 33-45). By determining the 3D location in this manner, only one click needs to be performed by a user for initiating the first X-Y coordinate, thereby requiring less work since Sra above teaches requiring two clicks in two separate view planes to identify the 3D location of an instrument. Since Korsten and Sra teaches identifying the 3D location of an instrument, Duindam further modified by the teachings of Bar Shalev would predictably result wherein the third coordinate value identified is associated with the indicated location of the instrument.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TOMMY T LY whose telephone number is (571) 272-6404. The examiner can normally be reached M-F 12:00pm-8:00pm eastern time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anhtuan Nguyen can be reached at 571-272-4963. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TOMMY T LY/ Examiner, Art Unit 3797
/SERKAN AKAR/ Primary Examiner, Art Unit 3797
Prosecution Insights