DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
Claims 1, 8, and 15 have been amended.
Claims 1-20 are currently pending.
In light of the claim amendments, the objection to the Title has been withdrawn.
In light of the claim amendments, the Section 101 rejection has been withdrawn.
Claim Objections
Claim 1 is objected to because of the following informalities:
Claim 1 should read “generate one or more virtual images from the 3D representation at virtual camera poses determined based on the position and orientation of the EM sensor of the catheter;”
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Each of claims 1, 8, and 15 recite or similarly recite (1) determining a refined anatomical location of the camera; (2) updating an EM-sensor-based estimate of the position and orientation of the catheter within the reference coordinate frame based on the refined anatomical location; and (3) refining a registration between the reference coordinate frame and the 3D representation using the updated position and orientation.
First, the use of the terms “refined,” “update,” and “refine” imply that a step occurred prior to the recited step and that the recited step is based upon the prior step (e.g., for a position or location to be refined the existing position/location must be known). However, the claims do not require that (1) an anatomical location of the camera already exists; (2) that an estimate of the position and orientation of the catheter already exists; or (3) that a registration between the reference coordinate frame and the 3D representation already exists. As such, it is unclear to one having ordinary skill in the art as to what is required by the claim.
With respect to (2), the indefiniteness is compounded by claim 1 reciting “update an EM-sensor-based estimate of the position and orientation of the catheter.” The phrase “the position and orientation” was previously recited and refers to the “EM sensor of the catheter,” not to the catheter in general. As mentioned, the term “update” implies that the element/feature already exist and “the position and orientation” supports this interpretation as well. However, “an EM-sensor-based estimate” is a newly recited element/feature.
Second, the combination of (1), (2), and (3) could be interpreted different ways, only one of which is supported by Applicant’s disclosure. For example, the combination of (1), (2), and (3) could be interpreted as determining a location of the camera within the 3D representation, then updating the position and orientation of the EM sensor in the reference coordinate frame based on the location of the camera in the 3D representation, then determining a registration between the reference coordinate frame and the 3D representation based on the updated position and orientation of the EM sensor within the coordinate frame. This interpretation is not supported by Applicant’s disclosure.
Instead, the registration is determined by the anatomical location of the camera within the 3D representation and the position and orientation of the EM sensor of the catheter within the reference coordinate frame when the anatomical location was determined. (see, e.g., [0041] and [0064] of Applicant’s disclosure).
To address the above issues and for the purposes of a compact prosecution, Examiner is interpreting the last four paragraphs of claim 1 as follows:
determine an anatomical location of the camera within the 3D representation by comparing branching structures identified in the real-time images with branching structures represented in the 3D representation;
determine a registration between the reference coordinate frame and the 3D representation using the anatomical location of the camera and the position and orientation of the EM sensor of the catheter within the reference coordinate frame when the anatomical location was determined; and
register a location of the catheter to the 3D representation using the registration between the reference coordinate frame and the 3D representation.
Claim 15 has similar limitations as those discussed above with respect to claim 1 and are rejected as being indefinite under Section 112(b) for the same rationale.
To address the above issues and for the purposes of a compact prosecution, Examiner is interpreting the last four paragraphs of claim 8 as follows:
determine an anatomical location of the camera within the 3D representation by comparing the branching structures identified in the real-time images with branching structures represented in the 3D representation;
determine a registration between the reference coordinate frame and the 3D representation using the anatomical location of the camera and the position and orientation of the EM sensor of the EWC within the reference coordinate frame when the anatomical location was determined; and
register a location of the catheter to the 3D representation using the registration between the reference coordinate frame and the 3D representation.
All dependent claims depend directly or indirectly from claims 1, 8, or 15. Accordingly, claims 1-20 are rejected under Section 112(b).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 5-7, 15, 16, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Appl. Publ. No. 2020/0297444 A1 to Camarillo et al. (hereinafter “CAMARILLO”) and Mori, Kensaku, et al. “Hybrid bronchoscope tracking using a magnetic tracking sensor and image registration.” International conference on medical image computing and computer-assisted intervention. Berlin, Heidelberg: Springer Berlin Heidelberg, 2005 (hereinafter “MORI”).
CAMRILLO discloses “systems and techniques for localizing and/or navigating a medical instrument within a luminal network.” (Abstract). CAMRILLO’s system includes an elongate body having an imaging device (e.g., video camera) positioned on a distal portion of the elongate body. (Id and [0094]). CAMARILLO determines a location of the distal portion of the elongate body within the luminal network based on images of one or more branchings of the luminal network. (Id).
With respect to claim 1, CAMARILLO discloses a system for performing a surgical procedure. Figure 20 shows a block diagram of a “localization system 90” that estimates a location of a medical instrument within a patient during a medical procedure. ([0108]). The system includes a catheter including a camera and an electromagnetic (EM) sensor. “[T]he distal end of the working channel 204 and/or the distal end of the elongate body 208 can be provided with EM sensors for tracking their position within the passageways 214. Additionally or alternatively, an imaging device 210 can be disposed at a distal end of the elongate body 208 of the medical instrument 202.” ([0124], see also Figure 26 showing an imaging device 804 at different states within a luminal network 800). “the instrument may be equipped with a camera to provide vision data (or image data) 92. The localization module 95 may process the vision data 92 to enable one or more vision-based (or image-based) location tracking modules or features… Intraoperatively, this library may be referenced by the robotic system in order to compare real-time images captured at the camera (e.g., a camera at a distal end of the endoscope) to those in the image library to assist localization.” ([0111]).
The system also includes a workstation (“control system 402”) operably coupled to the catheter, the workstation including a memory (“memory 412”) and a processor (“one or more processors 410”), the memory storing instructions. ([0156], see also [0190] for “instructions”). “The controller 406 can be coupled to the control system 402 and/or the medical instrument 408 to support guidance and control of the medical instrument 408. The controller 406 may correspond to the cart 11, the tower 30, the console 31, the controller 182 described above, or component(s) thereof.” ([0157]).
CAMARILLO also discloses that the instructions, when executed by the processor, cause the processor to:
receive pre-procedure images of a patient’s anatomy and generate a 3-dimensional (3D) representation of the patient’s anatomy based on the received pre-procedure images;. “[L]ocalization module 95… processes input data 91-94,” which includes preoperative model data 91 and vision or image data 92. ([0109]-[0111]). Preoperative model data 91 includes “[p]reoperative CT scans [that] are reconstructed into three-dimensional images, which are visualized, e.g. as “slices” of a cutaway view of the patient's internal anatomy… Techniques such as center-line geometry may be determined and approximated from the CT images to develop a three-dimensional volume of the patient's anatomy….” (emphasis added) ([0110]).
identify first anatomical landmarks within the generated 3D representation of the patient’s anatomy. “Some features of the localization module 95 may identify circular geometries in the preoperative model data 91 that correspond to anatomical lumens….” ([0112]) (see also [0130]: “The model generator 340 can generate a 3D volume of data from the series of 2D images, and can form the virtual 3D model of the internal surfaces of the anatomical luminal network from the 3D volume of data. For example, the model generator can apply segmentation to identify portions of the data corresponding to the tissue of the anatomical luminal network. As such, the resulting model can represent the luminal network, including interior surfaces of the tissue of the anatomical luminal network.”).
identify a position and orientation of the EM sensor of the catheter within a reference coordinate frame using the EM sensor. “As shown in FIG. 20, the localization system 90 may include a localization module 95 that processes input data 91-94 to generate location data 96 for the distal tip of a medical instrument. The location data 96 may be data or logic that represents a location and/or orientation of the distal end of the instrument relative to a frame of reference.” ([0109]). “The localization module 95 may use real-time EM tracking to generate a real-time location of the endoscope in a global coordinate system [i.e., reference coordinate frame] that may be registered to the patient's anatomy, represented by the preoperative model. In EM tracking, an EM sensor (or tracker) comprising one or more sensor coils embedded in one or more locations and orientations in a medical instrument (e.g., an endoscopic tool) ….” ([0114]). Input data includes “EM data 93.” ([0114]).
generate one or more virtual images from the 3D representation at virtual camera poses. CAMARILLO teaches “computationally position[ing] a virtual imaging device” at different locations within the model and generating “a virtual image at each location.” ([0133]). “A ‘virtual imaging device’ as described herein is not a physical imaging device, but rather a computational simulation of an image capture device. The simulation can generate virtual images based on virtual imaging device parameters including field of view, lens distortion, focal length, and brightness shading, which can in turn be based on parameters of an actual imaging device.” ([0133]). CAMARILLO does not explicitly teach that the poses are based on the position and orientation of the EM sensor.
receive real-time images of the patient’s anatomy from the camera of the catheter. “In some embodiments, the instrument may be equipped with a camera to provide vision data (or image data) 92. The localization module 95 may process the vision data 92 to enable one or more vision-based (or image-based) location tracking modules or features… Intraoperatively, this library may be referenced by the robotic system in order to compare real-time images captured at the camera (e.g., a camera at a distal end of the endoscope) to those in the image library to assist localization.” ([0111]).
identify second anatomical landmarks within the received real-time images, the second anatomical landmarks including branching structures of the luminal network visible in the real-time images. “The machine learning model 310 is a module configured to identify features, such as, for example, branchings, in real-time from images of the anatomical luminal network and compare them to the pre-computed feature(s) extracted from virtual images. The machine learning model 310 can scan the scope imaging data repository 380 for a match of a virtual feature to the feature extracted from an actual image….” (emphasis added) ([0136]) (see also, e.g., [0141] and [0162]).
[in light of the Section 112 rejection] determine an anatomical location of the camera within the 3D representation by comparing branching structures identified in the real-time images with branching structures represented in the 3D representation. “The machine learning model 310 is a module configured to identify features, such as, for example, branchings, in real-time from images of the anatomical luminal network and compare them to the pre-computed feature(s) extracted from virtual images. The machine learning model 310 can scan the scope imaging data repository 380 for a match of a virtual feature to the feature extracted from an actual image, and can use the location corresponding to the match as the position of the instrument (e.g., an endoscope) within the anatomical luminal network.” (emphasis added) ([0136]). “The machine learning model 310 can receive data input (e.g., from the scope imaging data 380) and identify features therein, such as branchings, as an output.” ([0137]). “Additionally or alternatively, the machine learning model 310 can output a location (e.g., a coordinate or relative location within the 3D model described above), a relative or absolute orientation relative to the 3D model described above, a distance or angular relationship to a branching or other identified feature in the 3D model.” ([0137]). (see also, e.g., [0141], [0162], [0177]).
[in light of the Section 112 rejection] determine a registration between the reference coordinate frame and the 3D representation using the anatomical location of the camera and the position and orientation of the EM sensor of the catheter within the reference coordinate frame when the anatomical location was determined. “The machine learning model 310 can scan the scope imaging data repository 380 for a match of a virtual feature to the feature extracted from an actual image, and can use the location corresponding to the match as the position of the instrument (e.g., an endoscope) within the anatomical luminal network.” (emphasis added) ([0136]). “The machine learning model 310 can output the position to the registration calculator 365 for use in generating an initial registration between the model and an EM field disposed around the patient and/or an updated registration.” (emphasis added) ([0136]).
register a location of the catheter to the 3D representation using the registration between the reference coordinate frame and the 3D representation. “The machine learning model 310 can output the position to the registration calculator 365 for use in generating an initial registration between the model and an EM field disposed around the patient and/or an updated registration.” (emphasis added) ([0136]). See generally [0143]-[0146]: “The registration calculator 365 is a module, for example, an algorithm, configured to identify a registration or mapping between the coordinate frame of the 3D model (e.g., a coordinate frame of the CT scanner used to generate the model) and the coordinate frame developed from the imaging and/or EM sensors.” (emphasis added) ([0143]). “Based on the received data, the location calculator 330 may perform, e.g., on-the-fly transformation between the imaging/EM sensor position data to a position in the 3D model. This can represent a preliminary estimate of the position of the distal end of the scope within the topography of the 3D model and can be provided as one input to the state estimator 342 for generating a final estimate of the scope position.” ([0146]).
However, CAMARILLO does not explicitly teach a processor that is configured to generate one or more virtual images from the 3D representation at virtual camera poses determined based on the position and orientation of the EM sensor. Nevertheless, as discussed above, CAMARILLO teaches generating one or more virtual images from the 3D representation at virtual camera poses. For example, CAMARILLO teaches “computationally position[ing] a virtual imaging device” at different locations within the model and generating “a virtual image at each location.” ([0133]). “A ‘virtual imaging device’ as described herein is not a physical imaging device, but rather a computational simulation of an image capture device.” ([0133]).
In the same field of endeavor, MORI teaches “a hybrid method for tracking a bronchoscope that uses a combination of magnetic sensor tracking and image registration.” (Abstract). “Because of respiratory motion, the magnetic sensor provides only the approximate position and orientation of the bronchoscope in the coordinate system of a CT image acquired before the examination. The sensor position and orientation is used as the starting point for an intensity-based registration between real bronchoscopic video images and virtual bronchoscopic images generated from the CT image. The output transformation of the image registration process is the position and orientation of the bronchoscope in the CT image.” (Abstract).
MORI teaches that one approach to tracking a bronchoscope is call “image-based tracking” in which “the position of the bronchoscope is determined by registration of real bronchoscopic (RB) video images and virtual bronchoscopic (VB) images generated from a CT image acquired before the examination.” (p.544, first full paragraph). MORI notes that “[i]mage-based tracking generally works very well, but one limitation is that when mistracking occurs in one frame, tracking of subsequent frames is difficult and the method often fails.” (p.544, second full paragraph). “To address this issue and make tracking more robust, we propose a hybrid method for tracking a bronchoscope that uses a combination of magnetic sensor tracking and image-based tracking (image registration).” (Id).
Notably, the virtual images in MORI are at virtual camera poses that are determined based on the position and orientation of the EM sensor. “Synthetic VB images are generated from the CT image using the viewpoint and view direction of the camera.” (p.545, top paragraph). “VB views are rendered by using only the sensor’s outputs.” (p.549, Figure 2 caption). The MORI system reduces computation time. “Although the sensor gives us only a rough estimation, precise estimation is done by image registration. Since the sensor’s outputs are used for initial estimation of image registration, it is possible to make the tracking system robust and to reduce computation time.” (p.549, first paragraph in 5. Discussion).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the CAMARILLO system by generating the one or more virtual images at virtual camera poses determined based on the position and orientation of the EM sensor, as taught in MORI. One of ordinary skill in the art would have been motivated to use the position and orientation of the EM sensor when determining how to generate the virtual images in order to reduce the computation time as taught in MORI. There would have been a reasonable expectation of success as CAMARILLO and MORI demonstrate that virtual images can be generated from 3D representations.
With respect to claim 2, CAMARILLO also discloses that the processor is configured to determine a distance between the camera and an identified anatomical landmark of the identified second anatomical landmarks within the received real-time images. “The machine learning model 430 can identify features, such as, for example, branchings, in real-time from images of the anatomical luminal network and may map or connect these features with pre-computed feature(s) extracted from the preoperative features described above…The machine learning model 430 can include one or more types of machine learning models, and can output information regarding a location and/or orientation of the imaging device. The location and/or orientation may be absolute, or may be relative to a preoperative model described. For example, when the information regarding location may be a positional relationship to a branching or other identified feature in the preoperative model.” (emphasis added) ([0162]).
With respect to claim 5, CAMARILLO also discloses that the system further comprises an extended working channel (EWC), the EWC configured to selectively receive the catheter and permit the catheter to access a luminal network of the patient. “The medical instrument 202 may comprise a working channel 204 having a first diameter that may be larger than certain of the passageways 214, and thus a distal end of the working channel 204 may not able to be positioned through the smaller-diameter airways around the target 212. Accordingly, an elongate body 208 of the medical instrument 202 extends from the working channel 204 of the medical instrument 202 and to the remaining distance to the target 212.” ([0123]).
With respect to claim 6, CAMARILLO also discloses that the processor is configured to continuously receive real-time images of the patient’s anatomy captured by the camera as the catheter is navigated through a luminal network of the patient. “The machine learning model 430 can identify features, such as, for example, branchings, in real-time from images of the anatomical luminal network and may map or connect these features with pre-computed feature(s) extracted from the preoperative features described above.” ([0162]).
With respect to claim 7 (depending from claim 6), CAMARILLO also discloses that the processor is configured to continuously identify second anatomical landmarks within the received real-time images corresponding to the identified first anatomical landmarks within the generated 3D representation of the patient’s anatomy as the catheter is navigated through the luminal network of the patient. “The image analyzer 428 can access volume-rendered or surface-rendered images of the luminal network from the model scans and can compare the rendered images with the real-time image and/or video frames from the imaging device 440.” ([0161]).
With respect to claim 15, CAMARILLO discloses a method of registering a location of a medical device to a 3D representation of a patient’s luminal network. Figure 20 shows a block diagram of a “localization system 90” that estimates a location of a medical instrument within a patient during a medical procedure. ([0108]). The system includes a catheter including a camera and an electromagnetic (EM) sensor. “[T]he distal end of the working channel 204 and/or the distal end of the elongate body 208 can be provided with EM sensors for tracking their position within the passageways 214. Additionally or alternatively, an imaging device 210 can be disposed at a distal end of the elongate body 208 of the medical instrument 202.” ([0124], see also Figure 26 showing an imaging device 804 at different states within a luminal network 800).
CAMARILLO also discloses the method includes:
generating a 3-dimensional (3D) representation of a patient’s luminal network based on pre-procedure images of the patient’s anatomy. “[L]ocalization module 95… processes input data 91-94,” which includes preoperative model data 91 and vision or image data 92. ([0109]-[0111]). Preoperative model data 91 includes “[p]reoperative CT scans [that] are reconstructed into three-dimensional images, which are visualized, e.g. as “slices” of a cutaway view of the patient's internal anatomy… Techniques such as center-line geometry may be determined and approximated from the CT images to develop a three-dimensional volume of the patient's anatomy….” (emphasis added) ([0110]). “Some features of the localization module 95 may identify circular geometries in the preoperative model data 91 that correspond to anatomical lumens….” ([0112])
identifying first anatomical landmarks within the generated 3D representation of the patient’s luminal network. “Some features of the localization module 95 may identify circular geometries in the preoperative model data 91 that correspond to anatomical lumens….” ([0112]) (see also [0130]: “The model generator 340 can generate a 3D volume of data from the series of 2D images, and can form the virtual 3D model of the internal surfaces of the anatomical luminal network from the 3D volume of data. For example, the model generator can apply segmentation to identify portions of the data corresponding to the tissue of the anatomical luminal network. As such, the resulting model can represent the luminal network, including interior surfaces of the tissue of the anatomical luminal network.”).
identifying a plurality of positions and orientations of an electromagnetic (EM) sensor disposed on a catheter within a reference coordinate frame as the catheter is navigated through the luminal network of the patient. “As shown in FIG. 20, the localization system 90 may include a localization module 95 that processes input data 91-94 to generate location data 96 for the distal tip of a medical instrument. The location data 96 may be data or logic that represents a location and/or orientation of the distal end of the instrument relative to a frame of reference.” ([0109]). “The localization module 95 may use real-time EM tracking to generate a real-time location of the endoscope in a global coordinate system [i.e., reference coordinate frame] that may be registered to the patient's anatomy, represented by the preoperative model. In EM tracking, an EM sensor (or tracker) comprising one or more sensor coils embedded in one or more locations and orientations in a medical instrument (e.g., an endoscopic tool) ... Once registered, an embedded EM tracker in one or more positions of the medical instrument (e.g., the distal tip of an endoscope) may provide real-time indications of the progression of the medical instrument through the patient's anatomy.” ([0114]). As such, a plurality of positions and orientations of the EM sensor are determined as the catheter is navigated through the luminal network. Note, “input data” includes “EM data 93.” ([0114]).
generating one or more virtual images from the 3D representation at virtual camera poses. CAMARILLO teaches “computationally position[ing] a virtual imaging device” at different locations within the model and generating “a virtual image at each location.” ([0133]). “A ‘virtual imaging device’ as described herein is not a physical imaging device, but rather a computational simulation of an image capture device. The simulation can generate virtual images based on virtual imaging device parameters including field of view, lens distortion, focal length, and brightness shading, which can in turn be based on parameters of an actual imaging device.” ([0133]). CAMARILLO does not explicitly teach that the poses are based on the position and orientation of the EM sensor.
receiving a plurality of real-time images captured by a camera disposed on the catheter as the catheter is navigated through the luminal network of the patient. “In some embodiments, the instrument may be equipped with a camera to provide vision data (or image data) 92. The localization module 95 may process the vision data 92 to enable one or more vision-based (or image-based) location tracking modules or features… Intraoperatively, this library may be referenced by the robotic system in order to compare real-time images captured at the camera (e.g., a camera at a distal end of the endoscope) to those in the image library to assist localization.” ([0111]).
identifying second anatomical landmarks within the received real-time images, the second anatomical landmarks including branching structures of the luminal network visible in the real-time images. “The machine learning model 310 is a module configured to identify features, such as, for example, branchings, in real-time from images of the anatomical luminal network and compare them to the pre-computed feature(s) extracted from virtual images. The machine learning model 310 can scan the scope imaging data repository 380 for a match of a virtual feature to the feature extracted from an actual image….” (emphasis added) ([0136]) (see also, e.g., [0141] and [0162]).
[in light of the Section 112 rejection] determining an anatomical location of the camera within the 3D representation by comparing the branching structures identified in the real-time images with the branching structures represented in the 3D representation. “The machine learning model 310 is a module configured to identify features, such as, for example, branchings, in real-time from images of the anatomical luminal network and compare them to the pre-computed feature(s) extracted from virtual images. The machine learning model 310 can scan the scope imaging data repository 380 for a match of a virtual feature to the feature extracted from an actual image, and can use the location corresponding to the match as the position of the instrument (e.g., an endoscope) within the anatomical luminal network.” (emphasis added) ([0136]). “The machine learning model 310 can receive data input (e.g., from the scope imaging data 380) and identify features therein, such as branchings, as an output.” ([0137]). “Additionally or alternatively, the machine learning model 310 can output a location (e.g., a coordinate or relative location within the 3D model described above), a relative or absolute orientation relative to the 3D model described above, a distance or angular relationship to a branching or other identified feature in the 3D model.” ([0137]). (see also, e.g., [0141], [0162], [0177]).
[in light of the Section 112 rejection] determining a registration between the reference coordinate frame and the 3D representation the anatomical location of the camera and the position and orientation of the EM sensor of the catheter within the reference coordinate frame when the anatomical location was determined. “The machine learning model 310 can scan the scope imaging data repository 380 for a match of a virtual feature to the feature extracted from an actual image, and can use the location corresponding to the match as the position of the instrument (e.g., an endoscope) within the anatomical luminal network.” (emphasis added) ([0136]). “The machine learning model 310 can output the position to the registration calculator 365 for use in generating an initial registration between the model and an EM field disposed around the patient and/or an updated registration.” (emphasis added) ([0136]).
registering a location of the catheter to the 3D representation of the patient's anatomy using the registration between the reference coordinate frame and the 3D representation. “The machine learning model 310 can output the position to the registration calculator 365 for use in generating an initial registration between the model and an EM field disposed around the patient and/or an updated registration.” (emphasis added) ([0136]). See generally [0143]-[0146]: “The registration calculator 365 is a module, for example, an algorithm, configured to identify a registration or mapping between the coordinate frame of the 3D model (e.g., a coordinate frame of the CT scanner used to generate the model) and the coordinate frame developed from the imaging and/or EM sensors.” (emphasis added) ([0143]). “Based on the received data, the location calculator 330 may perform, e.g., on-the-fly transformation between the imaging/EM sensor position data to a position in the 3D model. This can represent a preliminary estimate of the position of the distal end of the scope within the topography of the 3D model and can be provided as one input to the state estimator 342 for generating a final estimate of the scope position.” ([0146]).
However, CAMARILLO does not explicitly teach a method that includes generating one or more virtual images from the 3D representation at virtual camera poses determined based on the position and orientation of the EM sensor. Nevertheless, as discussed above, CAMARILLO teaches generating one or more virtual images from the 3D representation at virtual camera poses. For example, CAMARILLO teaches “computationally position[ing] a virtual imaging device” at different locations within the model and generating “a virtual image at each location.” ([0133]). “A ‘virtual imaging device’ as described herein is not a physical imaging device, but rather a computational simulation of an image capture device.” ([0133]).
In the same field of endeavor, MORI teaches “a hybrid method for tracking a bronchoscope that uses a combination of magnetic sensor tracking and image registration.” (Abstract). “Because of respiratory motion, the magnetic sensor provides only the approximate position and orientation of the bronchoscope in the coordinate system of a CT image acquired before the examination. The sensor position and orientation is used as the starting point for an intensity-based registration between real bronchoscopic video images and virtual bronchoscopic images generated from the CT image. The output transformation of the image registration process is the position and orientation of the bronchoscope in the CT image.” (Abstract).
MORI teaches that one approach to tracking a bronchoscope is call “image-based tracking” in which “the position of the bronchoscope is determined by registration of real bronchoscopic (RB) video images and virtual bronchoscopic (VB) images generated from a CT image acquired before the examination.” (p.544, first full paragraph). MORI notes that “[i]mage-based tracking generally works very well, but one limitation is that when mistracking occurs in one frame, tracking of subsequent frames is difficult and the method often fails.” (p.544, second full paragraph). “To address this issue and make tracking more robust, we propose a hybrid method for tracking a bronchoscope that uses a combination of magnetic sensor tracking and image-based tracking (image registration).” (Id).
Notably, the virtual images in MORI are at virtual camera poses that are determined based on the position and orientation of the EM sensor. “Synthetic VB images are generated from the CT image using the viewpoint and view direction of the camera.” (p.545, top paragraph). “VB views are rendered by using only the sensor’s outputs.” (p.549, Figure 2 caption). The MORI system reduces computation time. “Although the sensor gives us only a rough estimation, precise estimation is done by image registration. Since the sensor’s outputs are used for initial estimation of image registration, it is possible to make the tracking system robust and to reduce computation time.” (p.549, first paragraph in 5. Discussion).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the CAMARILLO system by generating the one or more virtual images at virtual camera poses determined based on the position and orientation of the EM sensor, as taught in MORI. One of ordinary skill in the art would have been motivated to use the position and orientation of the EM sensor when determining how to generate the virtual images in order to reduce the computation time as taught in MORI. There would have been a reasonable expectation of success as CAMARILLO and MORI demonstrate that virtual images can be generated from 3D representations.
With respect to claim 16, CAMARILLO teaches that the method further comprises determining a distance between the camera and an identified anatomical landmark of the identified second anatomical landmarks within the received real-time images, wherein the location of the camera within the reference coordinate frame is identified using the determined distance. “The machine learning model 430 can identify features, such as, for example, branchings, in real-time from images of the anatomical luminal network and may map or connect these features with pre-computed feature(s) extracted from the preoperative features described above…The machine learning model 430 can include one or more types of machine learning models, and can output information regarding a location and/or orientation of the imaging device. The location and/or orientation may be absolute, or may be relative to a preoperative model described. For example, when the information regarding location may be a positional relationship to a branching or other identified feature in the preoperative model.” (emphasis added) ([0162]).
With respect to claim 18, CAMARILLO discloses that the method further comprises advancing the catheter within an extended working channel (EWC) to gain access to the patient’s luminal network. “The medical instrument 202 may comprise a working channel 204 having a first diameter that may be larger than certain of the passageways 214, and thus a distal end of the working channel 204 may not able to be positioned through the smaller-diameter airways around the target 212. Accordingly, an elongate body 208 of the medical instrument 202 extends from the working channel 204 of the medical instrument 202 and to the remaining distance to the target 212.” ([0123]).
With respect to claim 19, CAMARILLO also discloses wherein receiving a plurality of real-time images captured by the camera includes continuously receiving the plurality of real-time images from the camera as the catheter is navigated through the luminal network of the patient. “The machine learning model 430 can identify features, such as, for example, branchings, in real-time from images of the anatomical luminal network and may map or connect these features with pre-computed feature(s) extracted from the preoperative features described above.” ([0162]).
With respect to claim 20, CAMARILLO discloses wherein identifying second anatomical landmarks within the received real-time images includes continuously analyzing the continuously received plurality of real-time images to identify second anatomical landmarks with the received real-time images corresponding to the identified first anatomical landmarks within the generated 3D representation of the patient’s anatomy. “The image analyzer 428 can access volume-rendered or surface-rendered images of the luminal network from the model scans and can compare the rendered images with the real-time image and/or video frames from the imaging device 440.” ([0161]).
Claims 3, 4, 8-14 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Appl. Publ. No. 2020/0297444 A1 to Camarillo et al. (hereinafter “CAMARILLO”) and U.S. Patent Appl. Publ. No. 2018/0256263 A1 to Krimsky et al. (hereinafter “KRIMSKY”) Mori, Kensaku, et al. “Hybrid bronchoscope tracking using a magnetic tracking sensor and image registration.” International conference on medical image computing and computer-assisted intervention. Berlin, Heidelberg: Springer Berlin Heidelberg, 2005 (hereinafter “MORI”).
With respect to claim 3 (depending from claim 2), CAMARILLO does not explicitly teach that the EM sensor of the catheter is disposed on the catheter at a predetermined distance from the camera, although this is strongly implied with CAMARILLO’s frequent reference to the EM and imaging have the same coordinate frame. (see, e.g., [0143]: “The registration calculator 365 is a module, for example, an algorithm, configured to identify a registration or mapping between the coordinate frame of the 3D model (e.g., a coordinate frame of the CT scanner used to generate the model) and the coordinate frame developed from the imaging and/or EM sensors.”; see also [0145] and [0146]: “In order to translate the initial position of the sensors (e.g., imaging device, EM sensor) into the model coordinate frame, the location calculator 330 can access the mapping between the EM/imaging device coordinate frame and the model coordinate frame….”).
Nonetheless, in the same field of endeavor, KRIMSKY teaches various embodiments in which the working channel (EWC) or access instrument (e.g. trocar or needle) permit a tool (e.g., needle, guidewire, biopsy tool, dilator, ablation device, or camera) to be inserted therethrough. (see, e.g., [0040] and [0047]). More specifically, KRIMSKY teaches a catheter guide assembly that “includes a control handle 91 and an extended working channel (EWC) 96 that is configured to receive a tool 120. The EWC 96 includes an electromagnetic (EM) sensor 94 located on a distal end 93 of the EWC 96 and a locking mechanism 99. Once inserted in EWC 96, tool 120 can be locked to EWC 96 with locking mechanism 99. Tool 120 may be any one of a variety of medical devices including, but not limited to, a needle, a guide wire, a biopsy tool, a dilator, or an ablation device.” ([0040]). Notably, the tool may also have an EM sensor. “In an embodiment, tool 120 also includes an EM sensor 94 and can be used independent of EWC 96.” In other embodiments, KRIMSKY teaches that the tool may have one or more “indicators” that are also considered EM sensors as they can “possess ferromagnetic properties that are detected” by the EM sensor of the working channel. ([0042]).
Figure 4B of KRIMSKY is shown here. “In FIG. 4B, EM sensor 94 remains positioned at the distal-most portion of EWC 96, but indicator 122 is shown located a distance from the distal-most portion of tool 120. The distance between the distal-most portion of tool 120 and indicator 122 may be a known distance that can establish a location of the distal-most portion of tool 120 outside of EWC 96.” (emphasis added) ([0050]). KRIMSKY also contemplates a video camera
PNG
media_image1.png
320
339
media_image1.png
Greyscale
being positioned at a distal end of the tool. (see, e.g., [0003], [0055], and [0062]).
It would have been obvious to modify the CAMARILLO so that the EM sensor of the catheter is disposed on the catheter at a predetermined distance from the camera as taught in KRIMSKY. One would have been motivated to do this because knowing the distance between the EM sensor and the distal tip of the tool enables better control of the tool with respect to the target and facilitates showing the tool within a 3D visualization. There would have been a reasonable expectation of success as KRIMSKY teaches that an EM sensor can be positioned a known distance away from the distal tip.
With respect to claim 4 (depending from claim 3), CAMARILLO teaches a memory storing thereon further instructions, which when executed by the processor cause the processor to perform operations of the procedure. CAMARILLO does not explicitly teach determining a distance between the camera and the identified anatomical landmark of the identified second anatomical landmarks within the received real-time images using the predetermined distance between the EM sensor and the camera.
PNG
media_image2.png
465
653
media_image2.png
Greyscale
In the same field of endeavor, KRIMSKY teaches using two EM sensors, one disposed along the EWC and another positioned at or near a distal tip of the catheter that moves through the EWC. For example, in Figure 5 below, the EM sensor or indicator 122 is located at a distal tip of a tool 120. The EM sensor 94 is located at a distal end of an EWC 96 that the tool 120 is permitted to move through. The EM sensor 94 detects the indicator 122, thereby allowing tracking of the tool 120. (see, e.g., [0054]).
Moreover, KRIMSKY teaches “[o]nce a distance or a position of indicator 122 is known in relation to EM sensor 94, a position of tool 120, to which indicator 122 is coupled, within the of the region of interest on monitoring equipment is determined and displayed on monitoring equipment 60 to guide the clinician's approach to the target. Upon establishing a position of tool 120 in the model of the region of interest, application 81 may determine the relative distance between tool 120 and the target on the model in order to count down the distance between the distal portion of tool 120 and the target.” (emphasis added) ([0068]).
It would have been obvious to modify the instructions of the CAMARILLO system so that the processors were configured to determine a distance between the camera and an identified anatomical landmark of the identified second anatomical landmarks within the received real-time images using the predetermined distance between the EM sensor and the camera. One would have been motivated to do this because using the combination of EM sensors to determine a distance between the camera (at a distal tip of the tool) and the EM sensor on the EWC “allows for a precise locating of the tool” as taught in KRIMSKY and, as such, the tool could approach the anatomical landmark. There would have been a reasonable expectation of success as KRIMSKY teaches that two EM sensors can be used in which one of them is tracked using the other.
With respect to claim 8, CAMARILLO discloses a system for performing a surgical procedure. Figure 20 shows a block diagram of a “localization system 90” that estimates a location of a medical instrument within a patient during a medical procedure. ([0108]). The system includes a catheter having a camera configured to capture images of a patient’s anatomy. “[T]he distal end of the elongate body 208 can [have]…an imaging device 210…disposed at a distal end of the elongate body 208 of the medical instrument 202.” ([0124], see also Figure 26 showing an imaging device 804 at different states within a luminal network 800). “the instrument may be equipped with a camera to provide vision data (or image data) 92. The localization module 95 may process the vision data 92 to enable one or more vision-based (or image-based) location tracking modules or features… Intraoperatively, this library may be referenced by the robotic system in order to compare real-time images captured at the camera (e.g., a camera at a distal end of the endoscope) to those in the image library to assist localization.” ([0111]).
The system also includes an extended working channel (EWC), the EWC configured to selectively receive the catheter and permit the catheter to access a luminal network of the patient, wherein the EWC includes an electromagnetic (EM) sensor. “The medical instrument 202 may comprise a working channel 204 having a first diameter that may be larger than certain of the passageways 214, and thus a distal end of the working channel 204 may not able to be positioned through the smaller-diameter airways around the target 212. Accordingly, an elongate body 208 of the medical instrument 202 extends from the working channel 204 of the medical instrument 202 and to the remaining distance to the target 212.” (emphasis added) ([0123]). “In such implementations,…the distal end of the working channel 204…can be provided with EM sensors for tracking their position within the passageways 214.” ([0124]).
a workstation (“control system 402”) operably coupled to the catheter, the workstation including a memory (“memory 412”) and a processor (“one or more processors 410”), the memory storing instructions. ([0156], see also [0190] for “instructions”). “The controller 406 can be coupled to the control system 402 and/or the medical instrument 408 to support guidance and control of the medical instrument 408. The controller 406 may correspond to the cart 11, the tower 30, the console 31, the controller 182 described above, or component(s) thereof.” ([0157]).
CAMARILLO also discloses that the instructions, when executed by the processor, cause the processor to:
generate a 3-dimensional (3D) representation of the patient's anatomy based on pre-procedure images of the patient's anatomy. “[L]ocalization module 95… processes input data 91-94,” which includes preoperative model data 91 and vision or image data 92. ([0109]-[0111]). Preoperative model data 91 includes “[p]reoperative CT scans [that] are reconstructed into three-dimensional images, which are visualized, e.g. as “slices” of a cutaway view of the patient's internal anatomy… Techniques such as center-line geometry may be determined and approximated from the CT images to develop a three-dimensional volume of the patient's anatomy….” (emphasis added) ([0110]).
identify first anatomical landmarks within the generated 3D representation of the patient's anatomy. “Some features of the localization module 95 may identify circular geometries in the preoperative model data 91 that correspond to anatomical lumens….” ([0112]) (see also [0130]: “The model generator 340 can generate a 3D volume of data from the series of 2D images, and can form the virtual 3D model of the internal surfaces of the anatomical luminal network from the 3D volume of data. For example, the model generator can apply segmentation to identify portions of the data corresponding to the tissue of the anatomical luminal network. As such, the resulting model can represent the luminal network, including interior surfaces of the tissue of the anatomical luminal network.”).
identify a position and orientation of the EM sensor of the EWC within a reference coordinate frame using the EM sensor. “As shown in FIG. 20, the localization system 90 may include a localization module 95 that processes input data 91-94 to generate location data 96 for the distal tip of a medical instrument. The location data 96 may be data or logic that represents a location and/or orientation of the distal end of the instrument relative to a frame of reference.” ([0109]). “The localization module 95 may use real-time EM tracking to generate a real-time location of the endoscope in a global coordinate system [i.e., reference coordinate frame] that may be registered to the patient's anatomy, represented by the preoperative model. In EM tracking, an EM sensor (or tracker) comprising one or more sensor coils embedded in one or more locations and orientations in a medical instrument (e.g., an endoscopic tool) ….” ([0114]). Input data includes “EM data 93.” ([0114]). Notably, both the catheter and the working channel may include EM sensors in order to track movement of both through the passageways. (see, e.g., [0123] and [0124]). “[O]ne or both of the distal end of the working channel 204 and/or the distal end of the elongate body 208 can be provided with EM sensors for tracking their position within the passageways 214.” ([0124]).
generate one or more virtual images from the 3D representation at virtual camera poses. CAMARILLO teaches “computationally position[ing] a virtual imaging device” at different locations within the model and generating “a virtual image at each location.” ([0133]). “A ‘virtual imaging device’ as described herein is not a physical imaging device, but rather a computational simulation of an image capture device. The simulation can generate virtual images based on virtual imaging device parameters including field of view, lens distortion, focal length, and brightness shading, which can in turn be based on parameters of an actual imaging device.” ([0133]). CAMARILLO does not explicitly teach that the poses are based on the position and orientation of the EM sensor.
receive real-time images of the patient’s anatomy from the camera of the catheter. “In some embodiments, the instrument may be equipped with a camera to provide vision data (or image data) 92. The localization module 95 may process the vision data 92 to enable one or more vision-based (or image-based) location tracking modules or features… Intraoperatively, this library may be referenced by the robotic system in order to compare real-time images captured at the camera (e.g., a camera at a distal end of the endoscope) to those in the image library to assist localization.” ([0111]).
identify second anatomical landmarks within the received real-time images, the second anatomical landmarks including branching structures of the luminal network visible in the real-time images. “The machine learning model 310 is a module configured to identify features, such as, for example, branchings, in real-time from images of the anatomical luminal network and compare them to the pre-computed feature(s) extracted from virtual images. The machine learning model 310 can scan the scope imaging data repository 380 for a match of a virtual feature to the feature extracted from an actual image….” (emphasis added) ([0136]) (see also, e.g., [0141] and [0162]).
[in light of the Section 112 rejection] determine an anatomical location of the camera within the 3D representation by comparing the branching structures identified in the real-time images with branching structures represented in the 3D representation. “The machine learning model 310 is a module configured to identify features, such as, for example, branchings, in real-time from images of the anatomical luminal network and compare them to the pre-computed feature(s) extracted from virtual images. The machine learning model 310 can scan the scope imaging data repository 380 for a match of a virtual feature to the feature extracted from an actual image, and can use the location corresponding to the match as the position of the instrument (e.g., an endoscope) within the anatomical luminal network.” (emphasis added) ([0136]). “The machine learning model 310 can receive data input (e.g., from the scope imaging data 380) and identify features therein, such as branchings, as an output.” ([0137]). “Additionally or alternatively, the machine learning model 310 can output a location (e.g., a coordinate or relative location within the 3D model described above), a relative or absolute orientation relative to the 3D model described above, a distance or angular relationship to a branching or other identified feature in the 3D model.” ([0137]). (see also, e.g., [0141], [0162], [0177]).
[in light of the Section 112 rejection] determine a registration between the reference coordinate frame and the 3D representation using the anatomical location of the camera and the position and orientation of the EM sensor of the catheter within the reference coordinate frame when the anatomical location was determined. “The machine learning model 310 can scan the scope imaging data repository 380 for a match of a virtual feature to the feature extracted from an actual image, and can use the location corresponding to the match as the position of the instrument (e.g., an endoscope) within the anatomical luminal network.” (emphasis added) ([0136]). “The machine learning model 310 can output the position to the registration calculator 365 for use in generating an initial registration between the model and an EM field disposed around the patient and/or an updated registration.” (emphasis added) ([0136]).
register a location of the catheter to the 3D representation using the registration between the reference coordinate frame and the 3D representation. “The machine learning model 310 can output the position to the registration calculator 365 for use in generating an initial registration between the model and an EM field disposed around the patient and/or an updated registration.” (emphasis added) ([0136]). See generally [0143]-[0146]: “The registration calculator 365 is a module, for example, an algorithm, configured to identify a registration or mapping between the coordinate frame of the 3D model (e.g., a coordinate frame of the CT scanner used to generate the model) and the coordinate frame developed from the imaging and/or EM sensors.” (emphasis added) ([0143]). “Based on the received data, the location calculator 330 may perform, e.g., on-the-fly transformation between the imaging/EM sensor position data to a position in the 3D model. This can represent a preliminary estimate of the position of the distal end of the scope within the topography of the 3D model and can be provided as one input to the state estimator 342 for generating a final estimate of the scope position.” ([0146]).
CAMARILLO does not explicitly teach a processor that is configured to generate one or more virtual images from the 3D representation at virtual camera poses determined based on the position and orientation of the EM sensor of the EWC. Nevertheless, as discussed above, CAMARILLO teaches generating one or more virtual images from the 3D representation at virtual camera poses. For example, CAMARILLO teaches “computationally position[ing] a virtual imaging device” at different locations within the model and generating “a virtual image at each location.” ([0133]). “A ‘virtual imaging device’ as described herein is not a physical imaging device, but rather a computational simulation of an image capture device.” ([0133]).
Moreover, CAMARILLO suggests circumstances in which both a working channel and a catheter having a camera each have corresponding EM sensors. ([0123] and [0124]). More specifically, a working channel 204 may have a diameter that is larger than certain passageways. ([0123]). In this case, the catheter may extend from the working channel into the smaller passageways and move toward a target. ([0123]). “In such implementations, one or both of the distal end of the working channel 204 and/or the distal end of the elongate body 208 can be provided with EM sensors for tracking their position within the passageways 214.” ([0124]).
In the same field of endeavor, KRIMSKY teaches various embodiments in which the working channel (EWC) or access instrument (e.g. trocar or needle) permit a tool (e.g., needle, guidewire, biopsy tool, dilator, ablation device, or camera) to be inserted therethrough. (see, e.g., [0040] and [0047]). More specifically, KRIMSKY teaches a catheter guide assembly that “includes a control handle 91 and an extended working channel (EWC) 96 that is configured to receive a tool 120. The EWC 96 includes an electromagnetic (EM) sensor 94 located on a distal end 93 of the EWC 96 and a locking mechanism 99. Once inserted in EWC 96, tool 120 can be locked to EWC 96 with locking mechanism 99. Tool 120 may be any one of a variety of medical devices including, but not limited to, a needle, a guide wire, a biopsy tool, a dilator, or an ablation device.” ([0040]). KRIMSKY also contemplates a video camera being positioned at a distal end of the tool. (see, e.g., [0003], [0055], and [0062]).
Moreover, the tool 120 may also have an EM sensor. “In an embodiment, tool 120 also includes an EM sensor 94 and can be used independent of EWC 96.” In other embodiments, KRIMSKY teaches that the tool may have one or more “indicators” that are also considered EM sensors as they can “possess ferromagnetic properties that are detected” by the EM sensor of the working channel. ([0042]).
KRIMSKY teaches registering the EWC and the tool during a procedure. In particular, KRIMSKY teaches identifying a location of the EM sensor of the EWC within the reference coordinate frame. “During registration, the location of EM sensor 94 within the patient's airways is tracked, and a plurality of points denoting the location of EM sensor 94 within the EM field generated by EM generator 76 is generated…As a result, detected movement of the EM sensor 94 within the patient can be accurately depicted on the display of the workstation 80 as a sensor 94 traversing the 3D model or a 2D image from which the 3D model was generated.” ([0061]).
KRIMSKY also teaches wherein the location of the tool is registered to the 3D representation of the patient’s anatomy using both the identified location of the EM sensor and the identified location of the catheter. “At step S626, the distance or relative position between indicator 122 and EM sensor 94 is used to update or establish a location of tool 120 in the model of the region of interest containing the target. As EM sensor 94 travels through the electromagnetic field, application 81 determines a location of EM sensor 94 and displays the location of EM sensor 94 in the model of the region of interest on monitoring equipment 60. Accordingly, a location of the item to which EM sensor 94 is coupled (e.g., bronchoscope 50, EWC 96, access instrument 200 or another device), can be shown within the model. Once a distance or a position of indicator 122 is known in relation to EM sensor 94, a position of tool 120, to which indicator 122 is coupled, within the of the region of interest on monitoring equipment is determined and displayed on monitoring equipment 60 to guide the clinician's approach to the target.” ([0068]).
KRIMSKY also teaches that it is desirable to know the distance between a distal end of the tool and the EWC. More specifically, KRIMSKY teaches that a distal end of a tool 120 may have an indicator 122 and may be positioned a predetermined distance (i.e., D1) beyond the EM sensor of the EWC 96. “As the distal portion of tool 120 exits EWC 96, a distance D1 may be tracked by application 81…[I]ndicator 122 may be located a distance from the distal-most portion of tool 120. In this scenario, a distance D1 could represent a known distance between indicator 122 and the distal-most portion of tool 120.” ([0054]).
KRIMSKY also teaches that the tool and the EWC may be locked into position with respect to each other. “Once tool 120 is locked within EWC 96, tool 120 and EWC 96 may be advanced toward the target together.” ([0049]). KRIMSKY also teaches “[o]nce a distance or a position of indicator 122 is known in relation to EM sensor 94, a position of tool 120, to which indicator 122 is coupled, within the of the region of interest on monitoring equipment is determined and displayed on monitoring equipment 60 to guide the clinician's approach to the target. Upon establishing a position of tool 120 in the model of the region of interest, application 81 may determine the relative distance between tool 120 and the target on the model in order to count down the distance between the distal portion of tool 120 and the target.” (emphasis added) ([0068]).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the CAMARILLO system, wherein each of the EWC and the catheter having a respective EM sensor, to further include a locking mechanism that locks the EWC and the catheter in position with respect to each other. One would have been motivated to make this modification so that the EWC and the catheter would have a known fixed relationship when locked, which would enable a more reliable registration of the catheter assembly. There would have been a reasonable expectation of success as both CAMARILLO and KRIMSKY teach that both the EWC and the tool/catheter may have an EM sensor for tracking.
In the same field of endeavor, MORI teaches “a hybrid method for tracking a bronchoscope that uses a combination of magnetic sensor tracking and image registration.” (Abstract). “Because of respiratory motion, the magnetic sensor provides only the approximate position and orientation of the bronchoscope in the coordinate system of a CT image acquired before the examination. The sensor position and orientation is used as the starting point for an intensity-based registration between real bronchoscopic video images and virtual bronchoscopic images generated from the CT image. The output transformation of the image registration process is the position and orientation of the bronchoscope in the CT image.” (Abstract).
MORI teaches that one approach to tracking a bronchoscope is call “image-based tracking” in which “the position of the bronchoscope is determined by registration of real bronchoscopic (RB) video images and virtual bronchoscopic (VB) images generated from a CT image acquired before the examination.” (p.544, first full paragraph). MORI notes that “[i]mage-based tracking generally works very well, but one limitation is that when mistracking occurs in one frame, tracking of subsequent frames is difficult and the method often fails.” (p.544, second full paragraph). “To address this issue and make tracking more robust, we propose a hybrid method for tracking a bronchoscope that uses a combination of magnetic sensor tracking and image-based tracking (image registration).” (Id).
Notably, the virtual images in MORI are at virtual camera poses that are determined based on the position and orientation of the EM sensor. “Synthetic VB images are generated from the CT image using the viewpoint and view direction of the camera.” (p.545, top paragraph). “VB views are rendered by using only the sensor’s outputs.” (p.549, Figure 2 caption). The MORI system reduces computation time. “Although the sensor gives us only a rough estimation, precise estimation is done by image registration. Since the sensor’s outputs are used for initial estimation of image registration, it is possible to make the tracking system robust and to reduce computation time.” (p.549, first paragraph in 5. Discussion).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the CAMARILLO-KRIMSKY system by generating the one or more virtual images at virtual camera poses determined based on the position and orientation of the EM sensor of the EWC. In order to determine the viewpoint of the camera, the position of the EM sensor on the EWC would be used to calculate the position of the camera/viewpoint. One of ordinary skill in the art would have been motivated to use the position and orientation of the EM sensor when determining how to generate the virtual images in order to reduce the computation time as taught in MORI. There would have been a reasonable expectation of success as CAMARILLO and MORI demonstrate that virtual images can be generated from 3D representations.
With respect to claim 9, CAMARILLO teaches memory storing thereon further instructions, which when executed by the processor cause the processor perform operations related to the procedure.
However, CAMARILLO does not explicitly teach identifying a location of the EM sensor of the EWC within the reference coordinate frame, wherein the location of the catheter is registered to the 3D representation of the patient’s anatomy using both the identified location of the EM sensor and the identified location of the catheter.
In the same field of endeavor, KRIMSKY teaches various embodiments in which the working channel (EWC) or access instrument (e.g. trocar or needle) permit a tool (e.g., needle, guidewire, biopsy tool, dilator, ablation device, or camera) to be inserted therethrough. (see, e.g., [0040] and [0047]). More specifically, KRIMSKY teaches a catheter guide assembly that “includes a control handle 91 and an extended working channel (EWC) 96 that is configured to receive a tool 120. The EWC 96 includes an electromagnetic (EM) sensor 94 located on a distal end 93 of the EWC 96 and a locking mechanism 99. Once inserted in EWC 96, tool 120 can be locked to EWC 96 with locking mechanism 99. Tool 120 may be any one of a variety of medical devices including, but not limited to, a needle, a guide wire, a biopsy tool, a dilator, or an ablation device.” ([0040]). Notably, the tool may also have an EM sensor. “In an embodiment, tool 120 also includes an EM sensor 94 and can be used independent of EWC 96.” In other embodiments, KRIMSKY teaches that the tool may have one or more “indicators” that are also considered EM sensors as they can “possess ferromagnetic properties that are detected” by the EM sensor of the working channel. ([0042]).
KRIMSKY teaches registering the EWC and the tool during a procedure. In particular, KRIMSKY teaches identifying a location of the EM sensor of the EWC within the reference coordinate frame. “During registration, the location of EM sensor 94 within the patient's airways is tracked, and a plurality of points denoting the location of EM sensor 94 within the EM field generated by EM generator 76 is generated…As a result, detected movement of the EM sensor 94 within the patient can be accurately depicted on the display of the workstation 80 as a sensor 94 traversing the 3D model or a 2D image from which the 3D model was generated.” ([0061]).
KRIMSKY also teaches wherein the location of the catheter is registered to the 3D representation of the patient’s anatomy using both the identified location of the EM sensor and the identified location of the catheter. “At step S626, the distance or relative position between indicator 122 and EM sensor 94 is used to update or establish a location of tool 120 in the model of the region of interest containing the target. As EM sensor 94 travels through the electromagnetic field, application 81 determines a location of EM sensor 94 and displays the location of EM sensor 94 in the model of the region of interest on monitoring equipment 60. Accordingly, a location of the item to which EM sensor 94 is coupled (e.g., bronchoscope 50, EWC 96, access instrument 200 or another device), can be shown within the model. Once a distance or a position of indicator 122 is known in relation to EM sensor 94, a position of tool 120, to which indicator 122 is coupled, within the of the region of interest on monitoring equipment is determined and displayed on monitoring equipment 60 to guide the clinician's approach to the target.” ([0068]).
It would have been obvious to modify the instructions of CAMARILLO so that the processors were configured to identify a location of the EM sensor of the EWC within the reference coordinate frame, wherein the location of the catheter is registered to the 3D representation of the patient’s anatomy using both the identified location of the EM sensor and the identified location of the catheter. One would have been motivated to do this because using the combination of EM sensors (one on the EWC and one on the tool that moves through the EWC) “allows for a precise locating of the tool” as taught in KRIMSKY. There would have been a reasonable expectation of success as KRIMSKY teaches that two EM sensors can be used in which one of them is tracked using the other.
With respect to claim 10, CAMARILLO does not explicitly teach wherein the camera is disposed a predetermined distance beyond the EM sensor of the EWC.
In the same field of endeavor, KRIMSKY teaches that a distal end of a tool 120 may have an indicator 122 and may be positioned a predetermined distance (i.e., D1) beyond the EM sensor of the EWC 96. “As the distal portion of tool 120 exits EWC 96, a distance D1 may be tracked by application 81…[I]ndicator 122 may be located a distance from the distal-most portion of tool 120. In this scenario, a distance D1 could represent a known distance between indicator 122 and the distal-most portion of tool 120.” ([0054]). KRIMSKY also contemplates a video camera being positioned at a distal end of the tool. (see, e.g., [0003], [0055], and [0062]). As such, KRIMSKY teaches wherein the camera is disposed a predetermined distance beyond the EM sensor of the EWC.
It would have been obvious to modify the CAMARILLO system such that the camera can be disposed a predetermined position beyond the EM sensor of the EWC. One would have been motivated to do this because using the combination of EM sensors (one on an end of the EWC and one on the tool that moves through the EWC and which includes a camera) “allows for a precise locating of the tool” as taught in KRIMSKY. There would have been a reasonable expectation of success as KRIMSKY teaches that two EM sensors can be used in which one of them is tracked using the other.
With respect to claim 11 (depending from claim 9), CAMARILLO does not explicitly teach wherein the catheter is configured to transition between a first, locked position where the catheter is inhibited from moving relative to the EWC and a second, unlocked position where the catheter is permitted to move relative to the EWC.
In the same field of endeavor, KRIMSKY teaches that “[i]n some embodiments of the present disclosure, tool 120 is inserted into EWC 96 and locked into place proximate a distal tip of EWC 96 via a locked mechanism (not explicitly shown). Tool 120 may be locked into place flush with the distal tip of EWC 96 or tool 120 may be locked into place extending beyond the distal tip of EWC 96 at a known distance.” (emphasis added) ([0049]).
It would have been obvious to modify the CAMARILLO system such that the catheter is configured to transition between a first, locked position where the catheter is inhibited from moving relative to the EWC and a second, unlocked position where the catheter is permitted to move relative to the EWC. One would have been motivated to do this because locking the tool (i.e., catheter) enables more precise control of the assembly as it is moved through the anatomy and enables holding the two with respect to each other so that position of the tool can be more precisely tracked. There would have been a reasonable expectation of success as KRIMSKY teaches that two EM sensors can be used in which one of them is tracked using the other.
With respect to claim 12 (depending from claim 10), CAMARILLO teaches memory storing thereon further instructions, which when executed by the processor cause the processor perform operations related to the procedure.
However, CAMARILLO does not explicitly teach determining a distance between the camera and an identified anatomical landmark of the identified second anatomical landmarks within the received real-time images using the predetermined distance between the EM sensor and the camera.
In the same field of endeavor, KRIMSKY teaches that the tool and the EWC may be locked into position with respect to each other. “Once tool 120 is locked within EWC 96, tool 120 and EWC 96 may be advanced toward the target together.” ([0049]). KRIMSKY also teaches “[o]nce a distance or a position of indicator 122 is known in relation to EM sensor 94, a position of tool 120, to which indicator 122 is coupled, within the of the region of interest on monitoring equipment is determined and displayed on monitoring equipment 60 to guide the clinician's approach to the target. Upon establishing a position of tool 120 in the model of the region of interest, application 81 may determine the relative distance between tool 120 and the target on the model in order to count down the distance between the distal portion of tool 120 and the target.” (emphasis added) ([0068]).
It would have been obvious to modify the instructions of the CAMARILLO system so that the processors were configured to determine a distance between the camera and an identified anatomical landmark of the identified second anatomical landmarks within the received real-time images using the predetermined distance between the EM sensor and the camera. One would have been motivated to do this because using the combination of EM sensors to determine a distance between the camera (at a distal tip of the tool) and the EM sensor on the EWC “allows for a precise locating of the tool” as taught in KRIMSKY and, as such, the tool could approach the anatomical landmark. There would have been a reasonable expectation of success as KRIMSKY teaches that two EM sensors can be used in which one of them is tracked using the other.
With respect to claim 13, CAMARILLO also discloses that the processor is configure to continuously receive the plurality of real-time images from the camera as the catheter is navigated through the luminal network of the patient. “The machine learning model 430 can identify features, such as, for example, branchings, in real-time from images of the anatomical luminal network and may map or connect these features with pre-computed feature(s) extracted from the preoperative features described above.” ([0162]).
With respect to claim 14, CAMARILLO discloses wherein identifying second anatomical landmarks within the received real-time images includes continuously identify second anatomical landmarks within the received real-time images corresponding to the identified first anatomical landmarks within the generated 3D representation of the patient’s anatomy as the catheter is navigated through the luminal network of the patient. “The image analyzer 428 can access volume-rendered or surface-rendered images of the luminal network from the model scans and can compare the rendered images with the real-time image and/or video frames from the imaging device 440.” ([0161]).
With respect to claim 17 (depending from claim 16), CAMARILLO does not explicitly teach wherein the distance between the camera and the identified anatomical landmark of the identified second anatomical landmarks within the received real-time images is determined using a pre-determined distance between the EM sensor and the camera.
In the same field of endeavor, KRIMSKY teaches that the tool and the EWC may be locked into position with respect to each other. “Once tool 120 is locked within EWC 96, tool 120 and EWC 96 may be advanced toward the target together.” ([0049]). KRIMSKY also teaches “[o]nce a distance or a position of indicator 122 is known in relation to EM sensor 94, a position of tool 120, to which indicator 122 is coupled, within the of the region of interest on monitoring equipment is determined and displayed on monitoring equipment 60 to guide the clinician's approach to the target. Upon establishing a position of tool 120 in the model of the region of interest, application 81 may determine the relative distance between tool 120 and the target on the model in order to count down the distance between the distal portion of tool 120 and the target.” (emphasis added) ([0068]).
It would have been obvious to modify the instructions of the CAMARILLO system so that the processors were configured to determine a distance between the camera and an identified anatomical landmark of the identified second anatomical landmarks within the received real-time images using the predetermined distance between the EM sensor and the camera. One would have been motivated to do this because using the combination of EM sensors to determine a distance between the camera (at a distal tip of the tool) and the EM sensor on the EWC “allows for a precise locating of the tool” as taught in KRIMSKY and, as such, the tool could approach the anatomical landmark. There would have been a reasonable expectation of success as KRIMSKY teaches that two EM sensors can be used in which one of them is tracked using the other.
RESPONSE TO APPLICANT’S ARGUMENTS
Applicant's arguments filed on March 4, 2026 have been fully considered but they are not persuasive. As an initial matter, Examiner is relying upon a newly cited reference, MORI, for teaching that the one or more virtual images from the 3D representation are generated at virtual camera poses determined based on the position and orientation of the EM sensor.
With respect to CAMARILLO’s disclosure, Applicant alleges that the following claim limitations are not disclosed in CAMARILLO. Examiner disagrees and addresses each in turn:
generate one or more virtual images from the 3D representation at virtual camera poses determined based on the position and orientation.
As discussed above, CAMARILLO explicitly describes generating one or more virtual images from the 3D representation at virtual camera poses. (see, e.g., [0133]). However, CAMARILLO does not explicitly describe generating the virtual images based on the position and orientation of the EM sensor. The Office Action relies upon the newly cited MORI for that limitation as explained above.
determine a refined anatomical location of the camera within the 3D representation by comparing branching structures identified in the real-time images with branching structures represented in the 3D representation.
As discussed above and in light of the Section 112(b) rejection, CAMARILLO explicitly describes this limitation in [0136] and elsewhere.
update an EM-sensor-based estimate of the position and orientation of the catheter within the reference coordinate frame based on the refined anatomical location.
As discussed above and in light of the Section 112(b) rejection, CAMARILLO describes this limitation in [0136] and [0143]-[0146].
refine a registration between the reference coordinate frame and the 3D representation using the updated position and orientation.
As discussed above and in light of the Section 112(b) rejection, CAMARILLO explicitly describes this limitation in [0136] and [0143]-[0146].
Prior Art Made of Record
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US-20190223760-A1 teaches “a registration method” in which “upon the commencement of navigating an endoscope, image-based registration methods are used in order to more accurately maintain the registration between the endoscope location and previously-acquired images. A six-degree-of-freedom location sensor is placed on the probe in order to reduce the number of previously-acquired images that must be compared to a real-time image obtained from the endoscope.” (Abstract; see also claims 2 and 9).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASON P GROSS whose telephone number is (571)272-1386. The examiner can normally be reached Monday-Friday 9:00-5:00CT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anne M. Kozak can be reached at (571) 270-5284. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JASON P GROSS/Examiner, Art Unit 3797
/SERKAN AKAR/Primary Examiner, Art Unit 3797
Prosecution Insights