DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
Applicant’s election without traverse of Group I (Claims 1-16 and 18-20) in the reply filed on 03/24/2026 is acknowledged.
Claim 25 is withdrawn from further consideration pursuant to 37 CFR 1.142(b) as being drawn to a nonelected invention, there being no allowable generic or linking claim. Election was made without traverse in the reply filed on 03/24/2026.
Priority
This application is a 371 of PCT/US2022/082437 filed 12/27/2022 which claims benefit of provisional application 63/295,701 filed 12/31/2021.
Information Disclosure Statement
The information disclosure statement (IDS) submitted were filed on 07/10/2024 and 07/23/2024. The submissions are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 7-8, 10, and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Duindam (US20190320878) in view of Miyamoto (US20110245660).
Regarding claim 1, Duindam teaches a system (100) (Fig. 1, [0029]) comprising:
a processor (112) ([0040], [0118]);
a display (110) (Fig. 1, [0034]); and
a memory (112) having computer readable instructions stored thereon that, when executed by the processor ([0040], [0118]), cause the system (100) to:
receive intra-operative three-dimensional image data from an imaging system (330), wherein a portion of the intra-operative three-dimensional image data corresponds to an instrument disposed in a patient anatomy (502) (Fig. 3-5, Abstract, [0006], [0057], “A three-dimensional imaging system 330 is arranged near the patient P to obtain three-dimensional images of the patient while the elongate device 310 is extended within the patient”, [0062], “intraoperative three-dimensional image data of a patient anatomy is obtained from an imaging system”, [0063], “the three-dimensional image data may include data representing at least a portion of a medical instrument 304 positioned within the anatomy of the patient P as shown in FIGS. 3A-B and 4A-D”).
However, Duindam fails to teach causing the system to: generate two-dimensional projection image data from the intra-operative three-dimensional image data; display the two-dimensional projection image data on the display; and identify, within the two-dimensional projection image data, a three-dimensional location of a portion of the instrument.
In an analogous image guided surgery field of endeavor, Miyamoto teaches such a feature. Miyamoto teaches an three-dimensional image obtaining unit (11) configured to obtain three-dimensional images (Fig. 2, [0055], [0057]). Miyamoto teaches wherein the three-dimensional images may be obtained during surgery ([0110]). Miyamoto teaches generating a 2D projection image using the three-dimensional images as input ([0058], “generates a projection image IP which is an image generated by projecting image information on a plurality of visual lines set by visual line setting unit 12 onto a predetermined projection plane”). Miyamoto further teaches displaying the generated 2D projection image on a display device ([0059]). Moreover, Miyamoto teaches wherein a user may use a pointing device and click on a position of a treatment tool on a 2D image (e.g. an axial cross-sectional image or projection image) and a treatment tool position setting unit (13) identifies the position of the treatment tool position in the coordinate space of the three-dimensional image (Claim 7, [0062]). Miyamoto therefore teaches generating a 2D projection image from intraoperative 3D images, displaying the 2D projection image, and identifying, within a 2D image (e.g. a 2D projection image), a three-dimensional location of a portion of an instrument.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to generate and display a 2D projection image from the intraoperative image and identify a 3D location of the instrument within the 2D projection image as taught by Miyamoto (Fig. 2, [0058-0059], [0062], [0110]). The displayed projection image may allow for easier understanding of an approach path of a treatment tool as recognized by Miyamoto ([0010]). Since Miyamoto teaches displaying the 2D projection image ([0059]) and wherein a user may use a 2D image to select a position of an instrument to identify in 3D space ([0062]), Duindam modified by the teachings of Miyamoto would predictably result in the identifying, within the two-dimensional projection image data, a three-dimensional location of a portion of the instrument.
Regarding claim 2, Duindam in view of Miyamoto teaches the invention as claimed above in claim 1.
Duindam further teaches wherein the computer readable instructions, when executed by the processor (112) ([0118]), cause the system (100) to:
segment, based on the identified three-dimensional location of the portion of the instrument, the portion of the intra-operative three-dimensional image data corresponding to the instrument ([0007], [0028], “As used herein, the term “shape” refers to a set of poses, positions, or orientations measured along an object” [0066], “…the shape information may be used to identify a search region or search area that includes the voxels corresponding to the medical instrument. By using the shape information, the segmentation process employed to segment the medical instrument from the rest of the three-dimensional image data”, [0071], wherein shape and thus position is measured in a 3D reference frame and thus comprises a 3D location, [0095], “The shape data may be used to help find the representation of the medical instrument in the image data. Accordingly, the computing system may perform segmentation of the image data to identify those portions of the intraoperative images that correspond to the medical instrument”).
Regarding claim 7, Duindam in view of Miyamoto teaches the invention as claimed above in claim 1.
Duindam further teaches wherein the computer readable instructions, when executed by the processor (112) ([0118]), cause the system (100) to:
generate a model of the patient anatomy based on pre-operative image data ([0084], “a preoperative model of the patient anatomy is generated from the preoperative image data”); and
update the model based on the intra-operative three-dimensional image data ([0062], “Thus, the intraoperative image data may correspond to two-dimensional, three-dimensional, or four-dimensional (including e.g., time based or velocity based information) images”, [0111], “the computing system may update the preoperative model based on the intraoperative image data”).
Regarding claim 8, Duindam in view of Miyamoto teaches the invention as claimed above in claim 7.
Duindam further teaches wherein updating the model comprises revising a location of an anatomical target ([0111], “update the preoperative model based on the intraoperative image data… the preoperative model includes a patient anatomy, a target such as a tumor… anatomical structures and/or the instrument may have shifted from a location represented in the preoperative model…The computing system may update a size, a location, and/or another property of the target based on the intraoperative image data… In such examples, the size, shape, and/or relative location of the other anatomical structures may also up updated in the preoperative model”).
Regarding claim 10, Duindam in view of Miyamoto teaches the invention as claimed above in claim 1.
Duindam further teaches wherein the computer readable instructions, when executed by the processor (112) ([0118]), cause the system (100) to:
generate a model of the patient anatomy based on pre-operative image data ([0084], “a preoperative model of the patient anatomy is generated from the preoperative image data”); and
register the model to the intra-operative three-dimensional image data based at least in part on a location of an anatomical target in each of the model and the intra-operative three-dimensional image data ([0104], “the preoperative model is registered to the intraoperative image reference frame”, [0105], “the preoperative reference frame may be registered to the intraoperative reference frame using feature-based registration to compare the respective models and/or images, and the computing system locates corresponding fiducial features in the intraoperative model/images and the preoperative model/images. Fiducial features include both artificial and anatomical features with a distinguishing characteristic”, wherein fiducial features including anatomical features comprise an anatomical target).
Regarding claim 13, Duindam in view of Miyamoto teaches the invention as claimed above in claim 1.
Duindam further teaches wherein the imaging system (330) comprises a cone-beam computed tomography system ([0057], “In some embodiments, the imaging system 330 includes a mobile rotational imaging element such as that of a mobile C-arm cone-beam CT imaging system for capturing intraoperative 3D images and/or fluoroscopic 2D images”).
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Duindam (US20190320878) in view of Miyamoto (US20110245660) as applied to claim 2 above, and further in view of Simon (US20110268248).
Regarding claim 3, Duindam in view of Miyamoto teaches the invention as claimed above in claim 2.
Duindam further teaches wherein the computer readable instructions, when executed by the processor (112) ([0118]), cause the system (100) to:
register the intra-operative three-dimensional image data to shape data from the instrument by comparing the shape data to the portion of the intra-operative three-dimensional data corresponding to the instrument (Abstract, [0007], [0069], wherein the segmented shape comprises the portion of the intra-operative three-dimensional image data corresponding to the instrument, [0071-0072], “At an operation 508, the segmented shape of the medical instrument may be registered with the shape data obtained from the medical instrument”).
However, Duindam fails to teach displaying a two-dimensional projection of the shape data on the two-dimensional projection image data.
In an analogous imaged guide surgery field of endeavor ([0003]), Simon teaches such a feature. Simon teaches acquiring fluoroscopic images with a fluoroscopic C-arm x-ray imaging device (100) (Fig. 1, [0041], [0044]). Simon further teaches a surgical instrument (140) which is locationally tracked three dimensionally ([0054]). Simon teaches calculating a projection of the instrument (140) into each fluoroscopic image as the instrument is moved ([0061]). Simon teaches wherein the instrument (140) is overlaid on the fluoroscopic images and that the graphical representation of the instrument (140) represents where the actual surgical instrument would appear ([0061]). Because the projection/graphical representation of the instrument includes the instrument’s shape, Simon therefore teaches displaying a two-dimensional projection of shape data on two-dimensional projection image data (fluoroscopic images).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to project or overlay the shape of the instrument onto the 2D projection images as taught by Simon ([0061]). Having the instrument be overlaid onto the 2D projection images in real-time as the instrument is moved may predictably help an operator guide said instrument during surgery.
Claims 4-5 are rejected under 35 U.S.C. 103 as being unpatentable over Duindam (US20190320878) in view of Miyamoto (US20110245660) and Simon (US20110268248) as applied to claim 3 above, and further in view of Ekin (US20210012526).
Regarding claim 4, Duindam in view of Miyamoto and Simon teaches the invention as claimed above in claim 3.
However, Duindam fails to teach wherein the computer readable instructions, when executed by the processor, cause the system to: identify one or more regions of the shape data that is misaligned with the portion of the intra-operative three-dimensional image data corresponding to the instrument; and display the one or more regions with at least one visual property different than one or more regions of the shape data that is aligned with the portion of the intra-operative three-dimensional image data corresponding to the instrument.
In an analogous image guided surgery field of endeavor ([0001]), Ekin teaches such a feature. Ekin teaches registering the shape of an interventional instrument (33) from optical shape sensing with a shape of the interventional instrument (33) in an image (Abstract, [0045]). Ekin further teaches an accuracy determination unit (12) configured to determine the accuracy of alignment based on the shape and images of the instrument ([0050]). Ekin teaches the accuracy determination unit (12) is configured to determine the alignment accuracy for different regions of the instrument (33) as defined by the position and shape and also the respective images, thereby determining aligned and misaligned regions ([0050]). Ekin teaches visualizing the accuracy of alignment and outputting it to a display (Fig. 6, [0050]). Ekin teaches the misaligned portions may be highlighted using another (different) visual property such as color ([0050]). Ekin teaches and shows in figure 6 wherein the misaligned segment or portion of the instrument is indicated by broken lines (Fig. 6, [0050]). An annotated figure 6 is reproduced below for clarity.
PNG
media_image1.png
310
444
media_image1.png
Greyscale
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to visualize instrument alignment/registration accuracy as taught by Ekin (Fig. 6, [0050]). By visualizing misalignment, a user may subsequently take actions to correct or update the alignment and registration as recognized by Ekin ([0066]), thereby improving registration accuracy.
Regarding claim 5, Duindam in view of Miyamoto, Simon, and Ekin teaches the invention as claimed above in claim 4.
However, Duindam fails to teach wherein the at least one visual property comprises at least one of a color, a brightness, a linetype, a pattern, or an opacity.
In an analogous image guided surgery field of endeavor ([0001]), Ekin teaches such a feature. Ekin teaches registering the shape of an interventional instrument (33) from optical shape sensing with a shape of the interventional instrument (33) in an image (Abstract, [0045]). Ekin further teaches an accuracy determination unit (12) configured to determine the accuracy of alignment based on the shape and images of the instrument ([0050]). Ekin teaches the accuracy determination unit (12) is configured to determine the alignment accuracy for different regions of the instrument (33) as defined by the position and shape and also the respective images, thereby determining aligned and misaligned regions ([0050]). Ekin teaches visualizing the accuracy of alignment and outputting it to a display (Fig. 6, [0050]). Ekin teaches the misaligned portions may be highlighted using another (different) visual property such as color, intensities, and symbols ([0018], [0050]). Ekin teaches and shows in figure 6 wherein the misaligned segment or portion of the instrument is indicated by broken lines (Fig. 6, [0050]). An annotated figure 6 is reproduced below for clarity.
PNG
media_image1.png
310
444
media_image1.png
Greyscale
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to have the adjusted visual property comprise a color, linetype, brightness (intensity) or pattern as taught by Ekin (Fig. 6, [0018], [0050]). These visual properties may be used to indicate or visualize differences (alignment accuracy) and thus indicate misaligned and aligned portions to a user as recognized by Ekin (Fig. 6, [0018], [0050]).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Duindam (US20190320878) in view of Miyamoto (US20110245660) and Simon (US20110268248) as applied to claim 3 above, and further in view of Adebar (US20200297442).
Regarding claim 6, Duindam in view of Miyamoto and Simon teaches the invention as claimed above in claim 6.
However, Duindam fails to teach wherein the computer readable instructions, when executed by the processor, cause the system to: receive a user input; and based on the user input, adjust at least one of a position or rotation of the shape data with respect to the intra-operative three-dimensional image data.
In an analogous image guided surgery field of endeavor, Adebar teaches such a feature. Adebar teaches tracking a medical instrument (104) and registering and displaying the medical instrument together with surgical images ([0039]). Adebar teaches wherein a user may update an existing registration between the medical instrument and the actual patient anatomy (e.g. images of the patient anatomy) because movement of the patient may cause an existing registration to become unreliable and unsuitable ([0099]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to have a user update the existing registration between the instrument and the images of the patient anatomy as taught by Adebar ([0099]). Movement of a patient may cause an existing registration to become unreliable and unsuitable for use as recognized by Adebar ([0099]), thereby needing an update. Modifying Duindam to have a user update the registration of a medical instrument with the patient anatomy would predictably result in receiving a user input and based on the user input, adjusting a position or a rotation (e.g. orientation) of shape data that is used to represent the instrument with respect to the intra-operative 3D image data that the instrument is registered to.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Duindam (US20190320878) in view of Miyamoto (US20110245660) as applied to claim 8 above, and further in view of Tata (US20220354380).
Regarding claim 9, Duindam in view of Miyamoto teaches the invention as claimed above in claim 8; Duindam teaches wherein updating the model comprises revising a location of an anatomical target ([0111]).
Duindam further teaches wherein the computer readable instructions, when executed by the processor (112) ([0118]), cause the system (100) to:
generate a navigation path through the patient anatomy based on the pre-operative image data ([0085], “the operator can generate a navigation path through the anatomic passageways in the preoperative model to guide the medical instrument to a target… a navigation path is determined through the anatomic passageways of the preoperative model to guide the medical instrument to the target”).
However, Duindam fails to explicitly teach wherein updating the model comprises revising a navigation path to correspond to the revised location of the anatomical target.
In an analogous image guided procedure field of endeavor, Tata teaches such a feature. Tata teaches displaying an anatomy model (12) of a patient on a display (68) ([0031]). Tata teaches updating the anatomy model (12) based on camera and position sensor data ([0039]). Tata teaches wherein the updating may include rerouting [i.e. revising] navigation paths to a desired destination ([0039]). Tata further teaches displaying the navigation path in the updated anatomy model ([0010], [0053]) Tata teaches revising a navigation path (B →180) to correspond to a revised location (288 → 188b) of an anatomical target (Fig. 7, [0056-0057]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to revise the navigation path to correspond to a revised location of an anatomical target as taught by Tata (Fig. 7, [0039], [0056-0057]). Duindam teaches revising the target location in the pre-operative patient model ([0111]). Therefore it’d be obvious to subsequently revise the navigation route to the revised target location as explicitly taught by Tata (Fig. 7, [0039], [0056-0057]). By revising the navigation route to correspond to the revised location, a user may accurately/reliably navigate towards the correct target location.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Duindam (US20190320878) in view of Miyamoto (US20110245660) as applied to claim 1 above, and further in view of Li (US20080283771).
Regarding claim 11, Duindam in view of Miyamoto teaches the invention as claimed above in claim 1.
However, Duindam fails to teach wherein the computer readable instructions, when executed by the processor, cause the system to: extract a three-dimensional boundary of an anatomical target from a model of the patient anatomy generated based on pre-operative image data; and display a projection of the three-dimensional boundary of the anatomical target on the two-dimensional projection image data.
In an analogous image guided surgery field of endeavor, Li teaches such a feature. Li teaches acquiring 4D image data of an anatomy of interest of a patient (110) with an intracardiac echocardiography (ICE) catheter (105) ([0050]) Li teaches extracting a surface model [i.e. a 3D boundary] of the imaged anatomy from the generated 4D ICE model ([0050]). Li further teaches acquiring fluoroscopic imaging data [i.e. 2D projection image data] ([0051]). Li teaches wherein the surface model may be projected in combination with the 2D fluoroscopic imaging data ([0051]). Li therefore teaches extracting a 3D boundary (surface model) of an anatomical target from a model of the patient anatomy; and displaying a projection of the boundary on 2D projection image data (fluoroscopic image).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to extract a surface from a anatomic model and to project that surface onto a 2D projection image as taught by Li ([0050-0051]). Projecting the 3D boundary of an anatomic target onto 2D projection image may improve a user’s visual understanding of the procedure, e.g. improve understanding of depth or 3D spatial relationship between the target and an instrument.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Duindam (US20190320878) in view of Miyamoto (US20110245660) and Li (US20080283771) as applied to claim 11 above, and further in view of Jiang (US20220304640).
Regarding claim 12, Duindam in view of Miyamoto and Li teaches the invention as claimed above in claim 11.
However, Duindam fails to teach wherein the computer readable instructions, when executed by the processor, cause the system to: receive an input from a user to manipulate at least one of a location or a dimension of the projection of the three-dimensional boundary.
In an analogous image guided procedure field of endeavor, Jiang teaches such a feature. Jiang teaches extracting a contour of an anatomic target ([0041], [0043]). Jiang teaches registering and superimposing [i.e. projecting] the contour onto an image ([0044], [0046]). Jiang teaches a user is allowed to adjust a position of the contour that is to be superimposed onto the image ([0050]). Jiang teaches receiving a shift instruction from a user and adjusting the position of the contour according to the shift instruction ([0050], [0060]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to allow a user to adjust a position of the projected boundary as taught by Jiang ([0050], [0060]). By allowing a user to adjust the position of the boundary, an alignment or registration of the boundary to the image may be made more accurate as recognized by Jiang ([0050]).
Claims 14-16 are rejected under 35 U.S.C. 103 as being unpatentable over Duindam (US20190320878) in view of Miyamoto (US20110245660) as applied to claim 1 above, and further in view of Mizuta (US20170319163).
Regarding claim 14, Duindam in view of Miyamoto teaches the invention as claimed above in claim 1.
However, Duindam fails to teach wherein the two-dimensional projection image data comprises at least one maximum intensity projection of the intra-operative three-dimensional image data based on voxel intensity values.
In an analogous three-dimensional imaging field of endeavor, Mizuta teaches such a feature. Mizuta teaches acquiring three-dimensional volume data ([0053]). Mizuta teaches a MIP-axis setting unit (20) sets a predetermined axis of a subject and denotes an axis orthogonal to each of directions for projecting an MIP image with respect to the three-dimensional volume data ([0053]). Mizuta teaches a MIP image is a maximum intensity projection image in which the maximum pixel (voxel) projection path is set as the luminance/intensity value ([0003], the MIP is generated from a 3D volume and so the “pixel” is a voxel). Mizuta teaches generating and displaying a first MIP image (T1) with respect to a first projection direction and a second MIP image (T2) with respect to a second projection direction orthogonal to the first direction (Figs. 7B & 9, [0004], [0074], [0076]). As shown in figure 9, Mizuta teaches wherein displaying 2D projection image data comprises displaying a plurality of views with different view orientations, the orientations being orthogonal to each other.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to have the projection image comprise a MIP image based on voxel intensity as taught by Mizuta (Figs. 7B & 9, [0003-0004], [0074], [0076]). The MIP images may be useful for approximating a position in which certain objects are such as radioactive pharmaceuticals as recognized by Mizuta ([0004-0005]).
Regarding claim 15, Duindam in view of Miyamoto and Mizuta teaches the invention as claimed above in claim 14.
However, Duindam fails to teach wherein displaying the two-dimensional projection image data on the display comprises displaying a plurality of views with different view orientations.
In an analogous three-dimensional imaging field of endeavor, Mizuta teaches such a feature. Mizuta teaches acquiring three-dimensional volume data ([0053]). Mizuta teaches a MIP-axis setting unit (20) sets a predetermined axis of a subject and denotes an axis orthogonal to each of directions for projecting an MIP image with respect to the three-dimensional volume data ([0053]). Mizuta teaches a MIP image is a maximum intensity projection image in which the maximum pixel (voxel) projection path is set as the luminance/intensity value ([0003], the MIP is generated from a 3D volume and so the “pixel” is a voxel). Mizuta teaches generating and displaying a first MIP image (T1) with respect to a first projection direction and a second MIP image (T2) with respect to a second projection direction orthogonal to the first direction (Figs. 7B & 9, [0004], [0074], [0076]). As shown in figure 9, Mizuta teaches wherein displaying 2D projection image data comprises displaying a plurality of views with different view orientations, the orientations being orthogonal to each other.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to display a plurality of maximum intensity projection (MIP) images with different view orientations as taught by Mizuta (Figs. 7B & 9, [0003-0004], [0074], [0076]). By referring to MIP images acquired in different projection directions, such as orthogonal to each other, an operator may know an approximating position of certain objects such as radioactive pharmaceuticals as recognized by Mizuta (Fig. 9, [0004-0005]).
Regarding claim 16, Duindam in view of Miyamoto and Mizuta teaches the invention as claimed above in claim 15.
However, Duindam fails to teach wherein the plurality of views comprise at least a first view and a second view, wherein an orientation of the first view is orthogonal to an orientation of the second view.
In an analogous three-dimensional imaging field of endeavor, Mizuta teaches such a feature. Mizuta teaches acquiring three-dimensional volume data ([0053]). Mizuta teaches a MIP-axis setting unit (20) sets a predetermined axis of a subject and denotes an axis orthogonal to each of directions for projecting an MIP image with respect to the three-dimensional volume data ([0053]). Mizuta teaches a MIP image is a maximum intensity projection image in which the maximum pixel (voxel) projection path is set as the luminance/intensity value ([0003], the MIP is generated from a 3D volume and so the “pixel” is a voxel). Mizuta teaches generating and displaying a first MIP image (T1) with respect to a first projection direction and a second MIP image (T2) with respect to a second projection direction orthogonal to the first direction (Figs. 7B & 9, [0004], [0074], [0076]). As shown in figure 9, Mizuta teaches wherein displaying 2D projection image data comprises displaying a plurality of views with different view orientations, the orientations being orthogonal to each other.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to display a plurality of maximum intensity projection (MIP) images with orthogonal view directions as taught by Mizuta (Figs. 7B & 9, [0003-0004], [0074], [0076]). By referring to MIP images acquired in orthogonal projection directions, an operator may know an approximating position of certain objects such as radioactive pharmaceuticals as recognized by Mizuta (Fig. 9, [0004-0005]).
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Duindam (US20190320878) in view of Miyamoto (US20110245660) and Mizuta (US20170319163) as applied to claim 16 above, and further in view of Sra (US20180317864).
Regarding claim 18, Duindam in view of Miyamoto and Mizuta teaches the invention as claimed above in claim 16.
However, Duindam fails to teach wherein identifying the three-dimensional location of the portion of the instrument comprises: receiving a first user input indicating a first two-dimensional location of the portion of the instrument in the first view; and receiving a second user input indicating a second two-dimensional location of the portion of the instrument in the second view.
In an analogous image guided surgery field of endeavor, Sra teaches such a feature. Sra teaches a fluoroscopy system (10) and acquiring an image from a first angle (View 1) and an image from a second angle (View 2) (Figs. 4-1 & 4-2, [0037], [0056], [0067]). Sra teaches wherein a user may place an orientation marker 75-1 on the View 1 image and an orientation marker 75-2 on the View 2 image, the orientation markers being placed on an instrument comprising a catheter (73) and cryoballoon (70) (Figs. 4-1 & 4-2, [0068]). Sra teaches the 2D coordinates of the orientation markers 75-1 and 75-2 are then acquired by the computer system ([0068]). Sra teaches determining the 3D location and orientation of the cryoballoon (70) based on the orientation markers 75-1 and 75-2 from the View 1 and View 2 images ([0071], [0094]). Sra therefore teaches wherein identifying the 3D location of a portion of an instrument (i.e. cryoballoon 70) comprises receiving a first user input indicating a first two-dimensional location (orientation marker 75-1) of the portion of the instrument in the first view (View 1); and receiving a second user input indicating a second two-dimensional location (orientation marker 75-2) of the portion of the instrument in the second view (View 2) (Figs. 4-1 & 4-2, [0067-0068], [0094]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to additionally identify the 3D location of the instrument by having a user select two 2D points from two images of different views as taught by Sra (Figs. 4-1 & 4-2, [0067-0068], [0094]). By additionally determining the 3D location of the instrument in this manner, a separate position sensor isn’t required, and a model of the instrument may be placed at the determined 3D coordinates to aid in visualization as recognized by Sra (Abstract, [0100]).
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Duindam (US20190320878) in view of Miyamoto (US20110245660) as applied to claim 1 above, and further in view of Kadir (US20100316272).
Regarding claim 19, Duindam in view of Miyamoto teaches the invention as claimed above in claim 1.
However, Duindam fails to teach wherein the computer readable instructions, when executed by the processor, cause the system to: select a region of interest within the intra-operative three dimensional data; generate two-dimensional projection image data from the selected region of interest within the intra-operative three-dimensional image data; and display the two-dimensional projection image data from the selected region of interest on the display.
In an analogous generating and displaying of projection images field of endeavor, Kadir teaches such a feature. Kadir teaches segmenting a CT image to provide for a region of interest in the form of a 3D surface ([0067]) and wherein selection of the region of interest/segmentation may be automatic or provided, i.e. drawn, by a user ([0083]). Kadir teaches generating and displaying a modified intensity projection image based on the region of interest (Claim 1, [0020], [0066], [0071]). Kadir therefore teaches selecting a region of interest within a 3D image (CT image) and generating and displaying a projection image from the selected region of interest.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to select a region of interest in the 3D image and generate and display the selected region of interest as taught by Kadir (Claim 1, [0020], [0066-0067], [0071], [0083]). By selecting a region of interest for generating and displaying a projection image of, a user or operator may predictably more easily visualize a region of interest for guidance of a procedure or surgery.
However, the modified combination noted above fails to teach wherein the region of interest is based on the identified three-dimensional location of the portion of the instrument.
In an analogous image guided surgery field of endeavor, Miyamoto teaches such a feature. Miyamoto teaches an three-dimensional image obtaining unit (11) configured to obtain three-dimensional images (Fig. 2, [0055], [0057]). Miyamoto teaches wherein a user may use a pointing device and click on a position of a treatment tool on a 2D image (e.g. an axial cross-sectional image or projection image) and a treatment tool position setting unit (13) identifies the position of the treatment tool position in the coordinate space of the three-dimensional image (Claim 7, [0062]). Miyamoto therefore teaches wherein an instrument (treatment tool) may be an object/region of interest in the 3D image and includes its 3D location. Kadir above teaches wherein user may select the region of interest.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to have the region or object of interest be the instrument as taught by Miyamoto (Claim 7, [0062]). Having the instrument be the region of interest and thus displayed as a 2D projection may predictably aid in image guidance as the projection of the instrument may be overlaid on an anatomical image of the patient. Moreover as Miyamoto provides the 3D location of the tool and Kadir teaches wherein a user may select the region of interest, Duindam modified by the teachings of Miyamoto and Kadir would predictably result in using the said identified 3D location for selecting the instrument to generate and display a 2D projection of.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Duindam (US20190320878) in view of Miyamoto (US20110245660) as applied to claim 1 above, and further in view of Sra (US20180317864) and Bar Shalev (US7024028).
Regarding claim 20, Duindam in view of Miyamoto teaches the invention as claimed above in claim 1.
However, Duindam fails to teach wherein displaying the two-dimensional image data on the display comprises displaying a first view with a view plane having a view plane orientation.
In an analogous image guided surgery field of endeavor, Miyamoto teaches such a feature. Miyamoto teaches generating a 2D projection image using the three-dimensional images as input ([0058]). Miyamoto further teaches displaying the generated 2D projection image on a display device ([0059]). Displaying a 2D projection image implies displaying the image in a first view with a view plane having a view plane orientation, wherein the view plane is the plane of the 2D image and the view plane orientation is arbitrary.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to generate and display a 2D projection image in a first view with a view plane having a view plane orientation as taught by Miyamoto (Fig. 2, [0058-0059]). The displayed projection image may allow for easier understanding of an approach path of a treatment tool as recognized by Miyamoto ([0010]).
However, the modified combination noted above fails to teach wherein identifying the three-dimensional location of the portion of the instrument comprises: receiving a user input indicating a location of the portion of the instrument in the first view, wherein the indicated location is identifiable by a first coordinate value and a second coordinate value of respective orthogonal first and second axes within the view plane.
In an analogous image guided surgery field of endeavor, Sra teaches such a feature. Sra teaches a fluoroscopy system (10) and acquiring an image from a first angle (View 1) and an image from a second angle (View 2) (Figs. 4-1 & 4-2, [0037], [0056], [0067]). Sra teaches wherein a user may place an orientation marker 75-1 on the View 1 image and an orientation marker 75-2 on the View 2 image, the orientation markers being placed on an instrument comprising a catheter (73) and cryoballoon (70) (Figs. 4-1 & 4-2, [0068]). Sra teaches the 2D coordinates of the orientation markers 75-1 and 75-2 are then acquired by the computer system ([0068]). Sra teaches determining the 3D location and orientation of the cryoballoon (70) based on the orientation markers 75-1 and 75-2 from the View 1 and View 2 images ([0071], [0094]). Sra therefore teaches receiving user input indicating a location of a portion of an instrument in a first (and second) view, wherein the indicated location is identifiable by a first coordinate value and a second coordinate value of respective orthogonal first and second axes with in a view plane (a 2D location; a location in 2 dimensional space).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to receive user input indicating a 2D location of the instrument in first and second views as taught by Sra (Figs. 4-1 & 4-2, [0067-0068], [0094]). By additionally determining the 3D location of the instrument in this manner, a separate position sensor isn’t required, and a model of the instrument may be placed at the determined 3D coordinates to aid in visualization as recognized by Sra (Abstract, [0100]).
However, the modified combination noted above fails to teach identifying a third coordinate value associated with the indicated location by retrieving a stored coordinate value of a voxel producing a maximum intensity at the indicated location of the portion of the instrument in the first view, wherein the third coordinate value represents an axis orthogonal to the view plane orientation.
In an analogous identification of a location of a region of interest field of endeavor, Bar Shalev teaches such a feature. Bar Shalev teaches generating or assembling a maximum intensity projection (MIP) frame (51) based on first and second dimensions and based on each of the maximum intensity pixels in the frame (Column 4 lines 33-39). Bar Shalev further teaches the third dimension of each pixel having a maximum intensity along each ray is stored in a virtual frame (52) (Column 4 lines 39-45). Bar Shalev teaches when clicking on a 2D location, for example a X, Y location, a “fetch” order may retrieve the third coordinate in the Z dimension (Column 2 lines 25-32, Column 2 lines 44-52, Column 4 lines 41-45). Bar Shalev therefore teaches identifying a third coordinate value (Z-depth direction) associated with an indicated location (X, Y location) by retrieving a stored coordinate value of a voxel producing maximum intensity at the indicated location (X, Y) location in a first view (X, Y view), wherein the third coordinate represents an axis orthogonal to the view plane direction (Z-axis). Sra above teaches wherein the indicated or clicked on location is a portion of an instrument.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the invention of Duindam to store and retrieve the third dimension or Z-axis dimension coordinate having the maximum intensity as taught by Bar Shalev (Column 2 lines 25-32, Column 2 lines 44-52, Column 4 lines 33-45). By determining the 3D location in this manner, only one click needs to be performed by a user for initiating the first X-Y coordinate, thereby requiring less work since Sra above teaches requiring two clicks in two separate view planes to identify the 3D location of an instrument. Since Miyamoto and Sra teaches identifying the 3D location of an instrument, Duindam further modified by the teachings of Bar Shalev would predictably result wherein the third coordinate value identified is associated with the indicated location of the instrument.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TOMMY T LY whose telephone number is (571) 272-6404. The examiner can normally be reached M-F 12:00pm-8:00pm eastern time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anhtuan Nguyen can be reached at 571-272-4963. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TOMMY T LY/ Examiner, Art Unit 3797
/SERKAN AKAR/ Primary Examiner, Art Unit 3797
Prosecution Insights