DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1-13 and 15-21 are pending in this application. Claims 1, 7, 10-11, and 15 are amended, and Claims 1-13 and 15-21 have been examined on the merits.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-4, 11-13, 15-17, and 19-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Brown (US20200375495A1).
Regarding Claim 1,
Brown teaches a device for navigation in hollow anatomical structures (corresponding disclosure in at least [0007], where a device is used for navigation in hollow anatomical structures (luminal network) “a method of registering a luminal network to a 3D model of the luminal network with real-time feedback”), comprising: a processor configured to:
obtain 3D image data of a hollow structure in a region of interest of a subject, wherein the 3D image data comprises a coordinate space (corresponding disclosure in at least [0048], where there are sensors for a coordinate space “ The coordinates of reference sensors 74 are sent to workstation 80, which includes and application 81 which uses data collected by sensors 74 to calculate a patient coordinate frame of reference” and further in [0056], where the sensor data, which provides the coordinate space is used in the 3D modeling “The registration process generally involves the clinician navigating EM sensor 94 through the airways of the patient's lungs to acquire location data. The location data is then compared to the 3D model to register the 3D model with the patient's airways”);
obtain an estimated current pose of a tool with a tool tip inserted in the hollow structure; wherein the tool does not have imaging capabilities (corresponding disclosure in at least [0045], where there is a tool tip (extended working channel (EWC) of the bronchoscope), in which the position can be tracked, the tip itself not being the imaging “a locatable guide (LG) 92, including an electromagnetic (EM) sensor 94, is inserted into EWC 96 and locked into position such that EM sensor 94 extends a desired distance beyond a distal tip 93 of EWC 96. The location of EM sensor 94, and thus the distal end of EWC 96, within an electromagnetic field generated by electromagnetic field generator 76 can be derived by tracking module 72, and workstation”);
transfer the estimated current pose of the tool tip to the coordinate space of the 3D image data based on the registration of the tool tip with the coordinate space of the 3D image data (corresponding disclosure in at least [0045], where the tool tip position is generated and derived (electromagnetic field generator derived by tracking module) “ The location of EM sensor 94, and thus the distal end of EWC 96, within an electromagnetic field generated by electromagnetic field generator 76 can be derived by tracking module 72, and workstation” and further in [0048], where that information is sent to the coordinate space “electromagnetic field generator 76 is positioned beneath the patient. Electromagnetic field generator 76 and the plurality of reference sensors 74 are interconnected with tracking module 72, which derives the location of each reference sensor 74. One or more of reference sensors 74 are attached to the chest of the patient. The coordinates of reference sensors 74 are sent to workstation”);
generate, from the 3D image data, a rendered image showing a scene inside the hollow structure relating to the transferred estimated current pose of the tool tip (corresponding disclosure in at least Figure 6 and [0059], where imaging continues based on the amount of data acquired, which is related to the current pose of the tool tip “The clinician may navigate EM sensor 94 through the patient's airways into a first region of the patient's lungs. The clinician may choose to acquire location data in any lung region and in any order during the data acquisition portion of the registration process. EM sensor 94 may be navigated down multiple branches of the airways in the first region of the patient's lungs to acquire location data spread throughout the lung region. By acquiring location data in various branches of the airways spread throughout the lung region, application 81 may generate a more accurate shape to correlate with the 3D model”); and
provide the rendered image to a user (corresponding disclosure in at least Figure 6, where an image is provided to the user of the scene inside the hollow structure).
PNG
media_image1.png
397
508
media_image1.png
Greyscale
Figure 6 of Brown
Regarding Claim 2, 16, and 19
Brown further teaches wherein the processor is further configured to:
obtain current image data of the region of interest acquired by an interventional imaging device arranged in the hollow structure in a current pose (corresponding disclosure in at least [0058], where there is current image data (real-time video feed) “ Video feed 606 may present a real-time video feed captured by the video imaging system of bronchoscope” of the region of interest “The regions of the patient's lungs may correspond to the patient's lung lobes, and may include regions 701, 702, 703, 704, and/or 705. Alternatively, the regions may correspond to other sub-regions of a lung or of a particular lobe of a lung”),
wherein the current image data comprises image data relating to a tool with a tool tip inserted in the hollow structure (corresponding disclosure in at least [0057], where the image data is related to the tool tip inserted (distal tip of the tool inserted into the hollow structure) “ the clinician inserts LG 92 into EWC 96, and the combination into bronchoscope 50 such that EM sensor 94 projects out from the distal end of bronchoscope …EM sensor 94 may be embedded within the distal tip of EWC 96 and operate independently of LG 92”);
and register the interventional imaging device in the current pose with the coordinate space of the 3D image data (corresponding disclosure in at least [0056], where the interventional imaging device (location data) is registered with the 3D image data (3D model), the imaging data being in a coordinate space being from the reference sensors in [0048] “The registration process generally involves the clinician navigating EM sensor 94 through the airways of the patient's lungs to acquire location data. The location data is then compared to the 3D model to register the 3D model with the patient's airways”);
estimate the current pose of the tool tip visible in the current image data (corresponding disclosure in at least [0045], where there is a tool tip (extended working channel (EWC) of the bronchoscope), in which the position can be tracked, the tip itself not being the imaging “a locatable guide (LG) 92, including an electromagnetic (EM) sensor 94, is inserted into EWC 96 and locked into position such that EM sensor 94 extends a desired distance beyond a distal tip 93 of EWC 96. The location of EM sensor 94, and thus the distal end of EWC 96, within an electromagnetic field generated by electromagnetic field generator 76 can be derived by tracking module 72, and workstation”); and
transfer the estimated current pose of the tool tip from the current image data to the coordinate space of the 3D image data based on the registration of the interventional imaging device with the coordinate space of the 3D image data (corresponding disclosure in at least [0059] and Figure 9, where the tool tip position is based on the registration data, with Figure 9 displaying the estimated current pose in the image “ The clinician may choose to acquire location data in any lung region and in any order during the data acquisition portion of the registration process. EM sensor 94 may be navigated down multiple branches of the airways in the first region of the patient's lungs to acquire location data spread throughout the lung region” ).
PNG
media_image2.png
512
536
media_image2.png
Greyscale
Figure 9 of Brown
Regarding Claim 3, 17, and 20
Brown further teaches wherein to register the interventional imaging device with the coordinate space of the 3D image data,
the processor is further configured to estimate a current pose of the interventional imaging device using the current image data of the region of interest acquired by the interventional imaging device (corresponding disclosure in at least [0058], where the position of the imaging device is estimated via the image data (video feed) “Video feed 606 may present a real-time video feed captured by the video imaging system of bronchoscope 50” the position being estimated with a sensor device “the clinician inserts LG 92 into EWC 96, and the combination into bronchoscope 50 such that EM sensor 94 projects out from the distal end of bronchoscope”).
Regarding Claim 4,
Brown further teaches wherein the 3D image data is at least one: CT image data, CBCT image data, or MRI image data of the subject (corresponding disclosure in at least [0051], where the data is CT “During procedure planning, workstation 80 utilizes computed tomographic (CT) image data for generating and viewing the 3D model of the patient's airways”);
and wherein the current image data comprises at least one of camera image data from an endoscope or bronchoscope, image data from an ultrasound transducer arrangement, or optical coherence tomography image data (corresponding disclosure in at least [0013], where images are from a bronchoscope “the method further includes presenting on the user interface live bronchoscopic images.”).
Regarding Claim 11,
Brown further teaches a system for navigation in hollow anatomical structures, comprising:
an interventional imaging device configured for insertion in hollow structures (corresponding disclosure in at least [0044], where a device (bronchoscope) is inserted into a hollow structure (patient airway) “a bronchoscope 50 configured for insertion through the patient's mouth and/or nose into the patient's airways”);
a tool with a tool tip configured for insertion in the hollow structure (corresponding disclosure in at least [0045], where there is a tool with a tool tip (EWC) which is inserted “including an electromagnetic (EM) sensor 94, is inserted into EWC 96 and locked into position such that EM sensor 94 extends a desired distance beyond a distal tip 93 of EWC 96.”);
and the device for navigation in hollow anatomical structures according claim 1; and a display; wherein the interventional imaging device provides the current image data of the region of interest of a subject; and wherein the display shows the generated rendered image (corresponding disclosure in at least [0046], where there is a display “a video display, for displaying the video images received from the video imaging system of bronchoscope 50.”).
Regarding Claim 12,
Brown teaches A method for navigation in hollow anatomical structures, comprising:
Providing 3D image data of a hollow structure in a region of interest of a subject, wherein the 3D image data comprises a coordinate space (corresponding disclosure in at least [0048], where there are sensors for a coordinate space “ The coordinates of reference sensors 74 are sent to workstation 80, which includes and application 81 which uses data collected by sensors 74 to calculate a patient coordinate frame of reference” and further in [0056], where the sensor data, which provides the coordinate space is used in the 3D modeling “The registration process generally involves the clinician navigating EM sensor 94 through the airways of the patient's lungs to acquire location data. The location data is then compared to the 3D model to register the 3D model with the patient's airways”);
Providing a current pose of a tool with a tool tip inserted in the hollow structure; wherein the tool does not have imaging capabilities (corresponding disclosure in at least [0045], where there is a tool tip (extended working channel (EWC) of the bronchoscope), in which the position can be tracked, the tip itself not being the imaging “a locatable guide (LG) 92, including an electromagnetic (EM) sensor 94, is inserted into EWC 96 and locked into position such that EM sensor 94 extends a desired distance beyond a distal tip 93 of EWC 96. The location of EM sensor 94, and thus the distal end of EWC 96, within an electromagnetic field generated by electromagnetic field generator 76 can be derived by tracking module 72, and workstation”);
Transferring the estimated current pose of the tool tip to the coordinate space of the 3D image data based on the registration of the tool tip with the coordinate space of the 3D image data (corresponding disclosure in at least [0059] and Figure 9, where the tool tip position is based on the registration data, with Figure 9 displaying the estimated current pose in the image “ The clinician may choose to acquire location data in any lung region and in any order during the data acquisition portion of the registration process. EM sensor 94 may be navigated down multiple branches of the airways in the first region of the patient's lungs to acquire location data spread throughout the lung region” );
Generating from the 3D image data, a rendered image showing a scene inside the hollow structure relating to the transferred estimated current pose of the tool tip; and providing the rendered image to a user (corresponding disclosure in at least Figure 6 and [0059], where imaging continues based on the amount of data acquired, which is related to the current pose of the tool tip “The clinician may navigate EM sensor 94 through the patient's airways into a first region of the patient's lungs. The clinician may choose to acquire location data in any lung region and in any order during the data acquisition portion of the registration process. EM sensor 94 may be navigated down multiple branches of the airways in the first region of the patient's lungs to acquire location data spread throughout the lung region. By acquiring location data in various branches of the airways spread throughout the lung region, application 81 may generate a more accurate shape to correlate with the 3D model”).
Regarding Claim 13,
Brown further teaches Arranging an interventional imaging device in the hollow structure (corresponding disclosure in at least [0044], where the device is arranged (inserted) into the airways (hollow structure) “a bronchoscope 50 configured for insertion through the patient's mouth and/or nose into the patient's airways”);
Providing current image data of the region of interest acquired by the interventional imaging device in the current pose (corresponding disclosure in at least [0058], where there is a real-time feed of the bronchoscope (current pose) which will acquire image data of the region of interest “Video feed 606 may present a real-time video feed captured by the video imaging system of bronchoscope”);
wherein the current image data comprises image data relating to a tool with a tool tip inserted in the hollow structure (corresponding disclosure in at least [0045], where there is a tool tip (extended working channel (EWC) of the bronchoscope), in which the position can be tracked, the tip itself not being the imaging “a locatable guide (LG) 92, including an electromagnetic (EM) sensor 94, is inserted into EWC 96 and locked into position such that EM sensor 94 extends a desired distance beyond a distal tip 93 of EWC 96. The location of EM sensor 94, and thus the distal end of EWC 96, within an electromagnetic field generated by electromagnetic field generator 76 can be derived by tracking module 72, and workstation”);
registering the interventional imaging device in the current pose within the coordinate space of the 3D image data; estimating the current pose of the tool tip (corresponding disclosure in at least [0056], where the interventional imaging device (location data) is registered with the 3D image data (3D model), the imaging data being in a coordinate space being from the reference sensors in [0048] “The registration process generally involves the clinician navigating EM sensor 94 through the airways of the patient's lungs to acquire location data. The location data is then compared to the 3D model to register the 3D model with the patient's airways”);
and transferring the estimated current pose of the tool tip from the current image data to the coordinate space of the 3D image data based on the registration of the interventional imaging device within the coordinate space of the 3D image data (corresponding disclosure in at least [0059] and Figure 9, where the tool tip position is based on the registration data, with Figure 9 displaying the estimated current pose in the image “ The clinician may choose to acquire location data in any lung region and in any order during the data acquisition portion of the registration process. EM sensor 94 may be navigated down multiple branches of the airways in the first region of the patient's lungs to acquire location data spread throughout the lung region”).
PNG
media_image2.png
512
536
media_image2.png
Greyscale
Figure 9 of Brown
Regarding Claim 15,
Brown teaches a non-transitory computer readable medium having stored the a computer program comprising instructions which (Corresponding disclosure in at least [0054], where there is a medium for executing the processes “computer-readable storage media includes RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, CD-ROM, DVD, Blu-Ray or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by workstation 80”), when executed by a processor, cause the processor to:
obtain 3D image data of a hollow structure in a region of interest of a subject, wherein the 3D image data comprises a coordinate space (corresponding disclosure in at least [0048], where there are sensors for a coordinate space “ The coordinates of reference sensors 74 are sent to workstation 80, which includes and application 81 which uses data collected by sensors 74 to calculate a patient coordinate frame of reference” and further in [0056], where the sensor data, which provides the coordinate space is used in the 3D modeling “The registration process generally involves the clinician navigating EM sensor 94 through the airways of the patient's lungs to acquire location data. The location data is then compared to the 3D model to register the 3D model with the patient's airways”);
obtain an estimated current pose of a tool with a tool tip inserted in the hollow structure; wherein the tool does not have imaging capabilities (corresponding disclosure in at least [0045], where there is a tool tip (extended working channel (EWC) of the bronchoscope), in which the position can be tracked, the tip itself not being the imaging “a locatable guide (LG) 92, including an electromagnetic (EM) sensor 94, is inserted into EWC 96 and locked into position such that EM sensor 94 extends a desired distance beyond a distal tip 93 of EWC 96. The location of EM sensor 94, and thus the distal end of EWC 96, within an electromagnetic field generated by electromagnetic field generator 76 can be derived by tracking module 72, and workstation”);
transfer the estimated current pose of the tool tip to the coordinate space of the 3D image data based on the registration of the tool tip with the coordinate space of the 3D image data (corresponding disclosure in at least [0045], where the tool tip position is generated and derived (electromagnetic field generator derived by tracking module) “ The location of EM sensor 94, and thus the distal end of EWC 96, within an electromagnetic field generated by electromagnetic field generator 76 can be derived by tracking module 72, and workstation” and further in [0048], where that information is sent to the coordinate space “electromagnetic field generator 76 is positioned beneath the patient. Electromagnetic field generator 76 and the plurality of reference sensors 74 are interconnected with tracking module 72, which derives the location of each reference sensor 74. One or more of reference sensors 74 are attached to the chest of the patient. The coordinates of reference sensors 74 are sent to workstation”);
generate, from the 3D image data, a rendered image showing a scene inside the hollow structure relating to the transferred estimated current pose of the tool tip (corresponding disclosure in at least Figure 6 and [0059], where imaging continues based on the amount of data acquired, which is related to the current pose of the tool tip “The clinician may navigate EM sensor 94 through the patient's airways into a first region of the patient's lungs. The clinician may choose to acquire location data in any lung region and in any order during the data acquisition portion of the registration process. EM sensor 94 may be navigated down multiple branches of the airways in the first region of the patient's lungs to acquire location data spread throughout the lung region. By acquiring location data in various branches of the airways spread throughout the lung region, application 81 may generate a more accurate shape to correlate with the 3D model”);
and provide the rendered image to a user (corresponding disclosure in at least Figure 6, where an image is provided to the user of the scene inside the hollow structure).
PNG
media_image1.png
397
508
media_image1.png
Greyscale
Figure 6 of Brown
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 5, and 7-9 are rejected under 35 U.S.C. 103 as being unpatentable over Brown (US20200375495A1) in view of Soper (US20190038365A1).
Regarding Claim 5, Brown teaches the limitations of Claim 1 and further teaches wherein, to register the interventional imaging device with the coordinate space of the 3D image data, the processor is further configured to: provide initial 3D image data of the interventional imaging device within the region of interest;
track the interventional imaging device (corresponding disclosure in at least [0047], where there is a tracking device for the imaging device “ Tracking system 70 is configured for use with catheter guide assemblies 90, 100 to track the position of EM sensor 94 as it moves in conjunction with EWC 96 through the airways of the patient, as detailed below”); and
adapt the initial pose based on the tracking (corresponding disclosure in at least [0065], where the sensors for the device are used to demonstrate the position, and the clinician determines based on the updated images to change course (adapt) “ As the clinician advances EM sensor 94 through the patient's airways, the displayed slice 1304 changes based on the position of EM sensor 94 relative to the registered 3D volume.” And further in [0066] “it is determined whether the registration is acceptable. For example, the clinician may determine whether the registration is acceptable… by determining that the EM sensor indicator 1302 does not stray from within the patient's airways as presented in the displayed slice 1304, the clinician accepts the registration by activating the “accept registration” button 1306. However, if the clinician determines that the registration is not acceptable, for example, if EM sensor indicator 1302 strays from within the patient's airways as presented in the displayed slice 1304, the clinician may decline the registration by activating the “decline registration” button 1308, and proceed to repeat the registration process … the system 10 may continue to track the location of EM sensor 94 within the patient's airways relative to the 3D volume and may continue to update and improve the registration during a subsequent navigation procedure.”).
Brown does not teach segment the interventional imaging device in the initial 3D image data generating an initial pose
Soper, in a similar field of endeavor, teaches a similar concept (calibration and imaging of surgical devices) of segment the interventional imaging device in the initial 3D image data generating an initial pose (corresponding disclosure in at least [0047], where the instrument is segmented “The catheter system 202 may optionally include a shape sensor system 222 (also called “shape sensor 222”) for determining the position, orientation, speed, velocity, pose, and/or shape of the catheter tip at distal end 218 and/or of one or more segments 224 along the body 216. The entire length of the body 216, between the distal end 218 and the proximal end 217, may be effectively divided into the segments 224”);
It would have been obvious to a person having ordinary skill in the art before the effective filing date to have incorporated segmenting the interventional imaging device in the initial 3D image data generating an initial pose as taught by Soper. One of the ordinary skill in the art would have been motivated to incorporate this because it provides more accuracy during registration and minimizes errors with estimating the pose of the device during navigation.
Regarding Claim 7, Brown teaches the limitations of Claim 1 and further teaches wherein, to register the interventional imaging device with the coordinate space of the 3D image data, the data processor is further configured to:
provide 2D image data from the interventional imaging device within the region of interest (corresponding disclosure in at least [0051], where there is 2D image data (the slices) within the region of interest (the pathway),
wherein the 2D image data comprises at least two 2D images with different viewing directions (corresponding disclosure in at least [0065], where the image data can be selected to be in a different viewing direction (coronal, axial, or sagittal) “the slice 1304 displayed in FIG. 11 is from the coronal direction, the clinician may alternatively select one of the axial or sagittal directions by activating a display bar “);
estimate a current pose of the interventional imaging device based on the 2D image data (extended working channel (EWC) of the bronchoscope), in which the position can be tracked, the tip itself not being the imaging “a locatable guide (LG) 92, including an electromagnetic (EM) sensor 94, is inserted into EWC 96 and locked into position such that EM sensor 94 extends a desired distance beyond a distal tip 93 of EWC 96. The location of EM sensor 94, and thus the distal end of EWC 96, within an electromagnetic field generated by electromagnetic field generator 76 can be derived by tracking module 72, and workstation”);
track the interventional imaging device (corresponding disclosure in at least [0044], where there is a tracking system for the interventional imaging device “ monitoring equipment 60 coupled to bronchoscope 50 for displaying video images received from bronchoscope 50; a tracking system 70 including a tracking module 72, a plurality of reference sensors 74, and an electromagnetic field generator 76; a workstation 80 including software and/or hardware used to facilitate pathway planning, identification of target tissue, navigation to target tissue, and digitally marking the biopsy location”); and
adapt the initial pose of the interventional imaging device based on the tracking (corresponding disclosure in at least [0047], where there is a 6-DOF tracking system for the device, which provides every pose of the device from the initial position “LG 92 and EWC 96 are selectively lockable relative to one another via a locking mechanism 99. A six degrees-of-freedom electromagnetic tracking system”).
Brown does not teach triangulate a 3D structure from the at least two images based on the estimated camera pose and the features visible in the at least two images;
initially segment structures within the 3D image data;
to register the triangulated 3D structure with a corresponding structure segmented in the 3D image data;
register and initialize the interventional imaging device in the coordinate space of the 3D image data.
Soper in a similar field of endeavor, teaches triangulate a 3D structure from the at least two images based on the estimated camera pose and the features visible in the at least two images (corresponding disclosure in at least [0067], where a 3D structure (position) is determined using triangulation and visible features from images (data capture by image sensors) “More specifically, optical tracking systems use data captured from image sensors to triangulate the three dimensional position of the teleoperational assembly, the medical instrument, and/or patient between cameras calibrated to provide overlapping projections.”)
initially segment structures within the 3D image data (corresponding disclosure in at least [0062], where the structures in the 3D image are segmented “he composite representation and the image data set describe the various locations and shapes of the passageways and their connectivity. More specifically, during the segmentation process the images are partitioned into segments or elements (e.g., pixels or voxels) that share certain characteristics or computed properties such as color, density, intensity, and texture. This segmentation process results in a two- or three-dimensional reconstruction that forms a model of the target anatomy based on the obtained image”)
to register the triangulated 3D structure with a corresponding structure segmented in the 3D image data; register and initialize the interventional imaging device in the coordinate space of the 3D image data (corresponding disclosure in at least [0063], where the device and the 3D image data are registered prior or during (initialization) “ the anatomic model data, a medical instrument used to perform the medical procedure (e.g., instrument system 200), and the patient anatomy are co-registered in a common reference frame prior to and/or during the course of an image-guided surgical procedure on the patient” and further in [0056], where it’s described that coordinate space is used “Systems and methods to reduce these errors are described below and may be used to generate more accurate registrations of the optical fiber and, consequently, the medical instrument to the surgical coordinate system and to the anatomic model during the procedure”).
It would have been obvious to a person having ordinary skill in the art before the effective filing date to have incorporated triangulating a 3D structure from the at least two images, registering the triangulated 3D structure with a corresponding structure segmented in the 3D image data, and further register and initialize the interventional imaging device in the coordinate space of the 3D image data as taught by Soper.
One of the ordinary skill in the art would have been motivated to incorporate this because these are all methods for reducing errors and providing more accurate registration within a surgical environment between image data and surgical instruments used.
Regarding Claim 8, the combined references of Brown and Soper teach the limitations of Claim 5 and further teach wherein the processor is configured to provide the tracking by at least one of:
i) relative pose estimation based on a sequence of images (corresponding disclosure in at least [0066] of Brown, where image tracking is completed sequentially (continuous video or images, which track the position) “ the system 10 may continue to track the location of EM sensor 94 within the patient's airways relative to the 3D volume and may continue to update and improve the registration during a subsequent navigation procedure”); and
ii) at least one electromagnetic tracking, robotic manipulation data, and shape sensing (corresponding disclosure in at least [0040] of Soper, where tracking is completed with electromagnetic tracking “The registration system of the present disclosure, for example, generally includes at least one sensor whose position is tracked within an electromagnetic field”).
Regarding Claim 9, Brown teaches the limitations of Claim 1, but does not teach obtaining current fluoroscopy image data and provide an updated estimation of the current pose of the tool tip based on the current fluoroscopy image data.
Soper, in a similar field of endeavor, teaches Obtain current fluoroscopy image data (corresponding disclosure in at least [0005], where fluoroscopic image data is obtained “The one or more processors perform a method including receiving, from a fluoroscopic imager, first fluoroscopic image data of the first fiducial marker positioned in the known configuration within the surgical coordinate space and receiving shape information from the shape sensor.”); and
provide an updated estimation of the current pose of the tool tip based on the current fluoroscopy image data (corresponding disclosure in at least [0039], where the tip of the instrument is shown in real-time (updated) based on the fluoroscopic image “display system 110 may display a virtual navigational image in which the actual location of the medical instrument 104 is registered (i.e., dynamically referenced)… with a virtual image of the internal surgical site from the viewpoint of the location of the tip of the instrument”).
It would have been obvious to a person having ordinary skill in the art before the effective filing date to have incorporated obtaining current fluoroscopy image data and providing an updated estimation of the current pose of the tool tip based on the current fluoroscopy image data as taught by Soper. One of the ordinary skill in the art would have been motivated to incorporate this because fluoroscopy image data is a commonly used real-time imaging modality particularly for obtaining patient anatomy.
Claims 6, 18, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Brown (US20200375495A1) in view of Soper (US20190038365A1) and in further view of Kadoury (US20140193053A1).
Regarding Claims 6,18, and 21, Brown teaches the limitations of Claims 1, 12, and 15, and further teaches wherein, to register the interventional imaging device with the coordinate space of the 3D image data, the data processor is further configured to: provide initial 2D image data of the interventional imaging device within the region of interest (corresponding disclosure in at least [0019], where the 3D image data is comprised of 2D slices (2D image data), which corresponds to the region of interest (location of the sensor) “wherein movement of the sensor results in presentation of a 2D slice of the luminal network at the location of the sensor has moved to”);
wherein the initial 2D image data comprises at least one initial 2D image (corresponding disclosure in at least [0019], where there is at least one 2D image “wherein movement of the sensor results in presentation of a 2D slice of the luminal network at the location of the sensor has moved to”);
to register the at least one initial 2D image with the 3D image data (corresponding disclosure in at least [0017], where the 2D slice is registered with the 3D image as it is taken from the image “the method further includes presenting on the user interface a 2D slice of the 3D model” and further in [0065], where the registration is explained further “ View 1300 presents the clinician with a stationary EM sensor indicator 1302 overlaid on a displayed slice 1304 of the 3D volume of the currently loaded navigation plan, for example, as shown in FIG. 13. The displayed slice 1304 may move about the EM sensor indicator 1302, and a different slice 1304 of the 3D volume may be displayed”);
track the interventional imaging device (corresponding disclosure in at least [0044], where there is a tracking system for the interventional imaging device “ monitoring equipment 60 coupled to bronchoscope 50 for displaying video images received from bronchoscope 50; a tracking system 70 including a tracking module 72, a plurality of reference sensors 74, and an electromagnetic field generator 76; a workstation 80 including software and/or hardware used to facilitate pathway planning, identification of target tissue, navigation to target tissue, and digitally marking the biopsy location”); and
adapt the initial pose of the interventional imaging device based on the tracking (corresponding disclosure in at least [0047], where there is a 6-DOF tracking system for the device, which provides every pose of the device from the initial position “LG 92 and EWC 96 are selectively lockable relative to one another via a locking mechanism 99. A six degrees-of-freedom electromagnetic tracking system”).
Brown does not teach the segmentation of imaging device.
Soper, in a similar field of endeavor, teaches a similar concept (calibration and imaging of surgical devices) of segmenting the interventional imaging device (corresponding disclosure in at least [0047], where the instrument is segmented and includes sensors on each segment, which shows how the device would be segmented in images “The catheter system 202 may optionally include a shape sensor system 222 (also called “shape sensor 222”) for determining the position, orientation, speed, velocity, pose, and/or shape of the catheter tip at distal end 218 and/or of one or more segments 224 along the body 216. The entire length of the body 216, between the distal end 218 and the proximal end 217, may be effectively divided into the segments 224” and further in Figure 9B, where the markers of the segmentation are shown).
PNG
media_image3.png
316
358
media_image3.png
Greyscale
Figure 9B of Soper
It would have been obvious to a person having ordinary skill in the art before the effective filing date to have incorporated segmentation of the interventional imaging device as taught by Soper. One of the ordinary skill in the art would have been motivated to incorporate this because it provides more accuracy during registration and minimizes errors with estimating the pose of the device during navigation.
The combined references of Brown and Soper do not teach initializing the segmented interventional imaging device within the coordinate space of the 3D image data.
Kadoury, in a similar field of endeavor, teaches a similar concept (image registration) of initialize the segmented interventional imaging device within the coordinate space of the 3D image data (corresponding disclosure in at least [0041], where there is initialization between 3D image data to an initial pose/image “The processing module 134 automatically initializes an intra-operative to pre-operative image registration search (e.g., US-CT) space, utilizing knowledge of an approximate pose of the ultrasound probe 122 (from tracking data) during an image plane acquisition”).
It would have been obvious to a person having ordinary skill in the art before the effective filing date to have incorporated initialization between 3D image data to an initial pose or image as taught by Kadoury. One of the ordinary skill in the art would have been motivated to incorporate this because the initialization provides accurate registration between the initial 3D data and the recorded images being taken.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Brown (US20200375495A1) in view of Flossmann (US20200078133A1).
Regarding Claim 10, Brown teaches the limitations of Claim 1, but does not teach wherein the processor is further configured to: generate a confidence estimate; wherein the confidence estimate relates to at least one of a quality of images used to estimate the pose, a quality of a pose estimation, and a quality of registration of the tool tip with the coordinate space of the 3D image data; and provide the confidence estimate to the user.
Flossmann, in a similar field of endeavor, teaches a similar concept (surgical navigation instruments and tracking) of generate a confidence estimate; wherein the confidence estimate relates to at least one of a quality of images used to estimate the pose, a quality of a pose estimation, and a quality of registration; and provide the confidence estimate to the user (corresponding disclosure in at least [0053], where there is a confidence estimate (value) regarding the image quality, which is used to show the quality of the pose (the tracked position) “ the first tracked position has associated a first confidence value, the second tracked position has associated a second confidence value and calculating the virtual image is based on the first confidence value and the second confidence value. A confidence value for example represents the quality of a tracked position. Calculating the virtual image for example utilizes the best combined quality of the first tracked position and the second tracked position. The higher a confidence value, the higher the quality, or accuracy, of the corresponding tracked position” and further in [0054], where the quality of registration between the track positions to the image are shown with the confidence value “calculating the virtual image is based on the first tracked position if the first confidence value is above a first threshold and calculating the virtual image is based on the second tracked position if the second confidence value is above a second threshold. If both confidence values are above their respective thresholds, calculating the virtual image may be based on both the first tracked position and the second tracked position in common or based on the one of the first and second tracked position having the higher confidence value”).
It would have been obvious to a person having ordinary skill in the art before the effective filing date to have incorporated a confidence estimate for relating to the quality of images as taught by Flossmann . One of the ordinary skill in the art would have been motivated to incorporate this because it demonstrates the quality, reliability, and precision of the provided image and position.
Response to Arguments
Applicant’s arguments filed 03/30/2026 regarding the 35 U.S.C 112b rejections have been considered and are withdrawn in light of the amendments.
Applicant's arguments filed 03/30/2026 regarding the 35 U.S.C. 102 and 103 rejections have been fully considered but they are not persuasive.
Regarding Claim 1 (and similarly claim 12 and 15), Applicant argues prior art Brown does not teach “generate, from the 3D image data, a rendered image showing a scene inside the hollow structure relating to the transferred estimated current pose of the tool tip” nor the step of “provide the rendered image to a user”.
However, Brown does display a rendered image, which is from 3D image data ([0048], where the 3D data is obtained, see office action above), the rendered image being inside the structure related to the estimated current pose of the tool tip as recited in Figure 13 and [0065] of Brown (“View 1300 presents the clinician with a stationary EM sensor indicator 1302 overlaid on a displayed slice 1304 of the 3D volume of the currently loaded navigation plan, for example, as shown in FIG. 13. The displayed slice 1304 may move about the EM sensor indicator 1302, and a different slice 1304 of the 3D volume may be displayed as the position of EM sensor 94 within the patient's airways change, with EN sensor indicator 1302 remaining stationary, as can be seen by comparison of FIGS. 11 and 12. Although the slice 1304 displayed in FIG. 11 is from the coronal direction, the clinician may alternatively select one of the axial or sagittal directions by activating a display bar 1310. As the clinician advances EM sensor 94 through the patient's airways, the displayed slice 1304 changes based on the position of EM sensor 94 relative to the registered 3D volume”). The 3D volume changes as well as the image slice (both considered image renderings from the 3D image data), based on the view of the tool tip, or the EM sensor. This tracking is consistently displayed to the user, thus providing “the rendered image to a user”. Through the process described in Brown of surveying and feedback, there is constant generation of a rendered image from the 3D image, which shows the scene inside the hollow structure (the airways) which relates to the estimated current pose of the tool tip (the EM sensor), as further discussed above and in the office action.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KAITLYN KIM whose telephone number is (571)272-1821. The examiner can normally be reached Monday-Friday 6-2 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anne Kozak can be reached at (571) 270-0552. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/K.E.K./ Examiner, Art Unit 3797
/SERKAN AKAR/ Primary Examiner, Art Unit 3797
Prosecution Insights