DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 3-14, 16-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Slawinski (US 20230072879 A1).
Regarding claim 1, Slawinski teaches A computer-implemented method for an endoscopic device, comprising:
(a) in a navigation mode of a graphical user interface (GUI), navigating the endoscopic device towards a target within a subject, wherein the GUI displays a virtual view with visual elements to guide navigating the endoscopic device ([0062-65] kinematic catheter tip frame is expressed and calculated, and the error angle is projected);
(b) upon switching to a tomosynthesis mode of the GUI, i) receiving a sequence of fluoroscopic image frames containing a portion of the endoscopic device and the target, wherein the sequence of fluoroscopic image frames correspond to various poses of an imaging system acquiring the sequence of fluoroscopic image frames, ii) generating a reconstructed 3D tomosynthesis image based at least in part on the poses of the imaging system and iii) determining a location of the target based at least in part on the reconstructed 3D tomosynthesis image ([0086], In some cases, when an electromagnetic (EM) system is used, real-time imaging (e.g., tomosynthesis, EBUS, live camera) may be employed to provide corrections to EM navigation thereby enhancing the localization accuracy); and
(c) upon switching to a fluoroscopic view mode of the GUI, i) obtaining a pose of the imaging system associated with a fluoroscopic image frame acquired in the fluoroscopic view mode, and ii) generating an overlay of the target displayed onto the fluoroscopic image frame based at least in part on the pose of the imaging system and the location of the target determined in (b) ([0070], a relative orientation between the endoscope kinematic frame and a reference fluoroscopic coordinate system may be computed. For instance, by mapping motion from the fluoroscopic image data to motion in the kinematics obtained from the driving mechanism motion (e.g., compute the kinematics data and scope tip position based on the fluoroscopic image data).
Regarding claim 3, Slawinski teaches The computer-implemented method of claim 1,
wherein a location of the target displayed in the navigation mode is updated based on the location of the target determined in (b) ([0086] The multimodal sensing feature of the present disclosure may include combining the multiple sensing modalities using a unique fusion framework. The bronchoscope may combine electromagnetic (EM) sensor, direct imaging device, tomosynthesis, kinematics data and ultrasound imaging using a dynamic fusion framework allowing for small lung modules to be identified specifically outside the airways and automatically steer the bronchoscope towards the target. ).
Regarding claim 4, Slawinski teaches The computer-implemented method of claim 1,
wherein the poses of the imaging system in the tomosynthesis mode are estimated using a marker contained in the sequence of fluoroscopic image frames ([0073], radioplaque markers within the imaging field).
Regarding claim 5, Slawinski teaches The computer-implemented method of claim 1,
wherein the poses of the imaging system in the tomosynthesis mode are measured by one or more sensors ([0007] collecting positional sensor data and kinematics data).
Regarding claim 6, Slawinski teaches The computer-implemented method of claim 1,
wherein the pose of the imaging system associated with the fluoroscopic image frame in the fluoroscopic view mode is estimated using a marker contained in the fluoroscopic image frame ([0074] FIG. 5 shows an example of a radiopaque marker 503 attached to the catheter tip 501 for pose estimation. As shown in the figure, a radiopaque pattern is placed on the tip of an endoscope and imaged by fluoroscopic imaging).
Regarding claim 7, Slawinski teaches The computer-implemented method of claim 6,
wherein the marker has a 3D pattern ([0075] The radiopaque marker may have any pattern, shape or geometrics that is useful for recovering the 3D orientation of the catheter tip. For instance, the pattern may be non-symmetrical with at least three points).
Regarding claim 8, Slawinski teaches The computer-implemented method of claim 7,
wherein the marker comprises a plurality of features placed on at least two different planes ([0075] The radiopaque marker may have any pattern, shape or geometrics that is useful for recovering the 3D orientation of the catheter tip. For instance, the pattern may be non-symmetrical with at least three points).
Regarding claim 9, Slawinski teaches he computer-implemented method of claim 7,
wherein the marker has a plurality of features of different sizes arranged in a coded pattern ([0075] The radiopaque marker may have any pattern, shape or geometrics that is useful for recovering the 3D orientation of the catheter tip. For instance, the pattern may be non-symmetrical with at least three points).
Regarding claim 10, Slawinski teaches The computer-implemented method of claim 9,
wherein the coded pattern comprises a plurality of sub-areas each has a unique pattern ([0075] Markers of many shapes and sizes can be employed. In some cases, the markers may have a non-symmetrical shape or pattern with at least three distinguishable points).
Regarding claim 11, Slawinski teaches The computer-implemented method of claim 9,
wherein the pose of the imaging system is estimated by matching a patch of the plurality of features in the fluoroscopic image frame to the coded pattern ([0074] The fluoroscopic image data may be captured while the endoscopic device is in motion. The radiopaque pattern is visible in the fluoroscopic image data. The fluoroscopic image data may be processed for recovering the orientation of the catheter tip such as using computer vision, machine learning, or other object recognition methods to recognize and analyze the shape of the marker in the fluoroscopic image.).
Regarding claim 12, Slawinski teaches The computer-implemented method of claim 1,
wherein the pose of the imaging system associated with the fluoroscopic image frame in the fluoroscopic view mode is measured by one or more sensors ([0007] collecting positional sensor data and kinematics data).
Regarding claim 13, Slawinski teaches The computer-implemented method of claim 1,
wherein in the tomosynthesis mode, further comprising determining whether a fluoroscopic image frame from the sequence of fluoroscopic image frames is unique based at least in part on an intensity comparison ([0073] Fluoroscopy is an imaging modality that obtains real-time moving images of patient anatomy, medical instruments, and any radiopaque markers within the imaging field using X-rays. Fluoroscopic systems may include C-arm systems which provide positional flexibility and are capable of orbital, horizontal, and/or vertical movement via manual or automated control. Non-C-arm systems are stationary and provide less flexibility in movement. Fluoroscopy systems generally use either an image intensifier or a flat-panel detector to generate two dimensional real-time images of a patient anatomy. Bi-planar fluoroscopy systems simultaneously capture two fluoroscopic images, each from different (often orthogonal) viewpoints. In the presented methods, a radiopaque marker disposed at the tip of the catheter may be visible by the fluoroscopic imaging and is analyzed for estimating a pose of the catheter or the camera.).
Regarding claim 14, Slawinski teaches A non-transitory computer-readable media storing instructions which, when executed by at least one processor, cause the at least one processor to perform operations comprising:
(a) in a navigation mode of a graphical user interface (GUI), navigating an endoscopic device towards a target within a subject, wherein the GUI displays a virtual view with visual elements to guide navigating the endoscopic device ([0062-65] kinematic catheter tip frame is expressed and calculated, and the error angle is projected);
(b) upon switching to a tomosynthesis mode of the GUI, i) receiving a sequence of fluoroscopic image frames containing a portion of the endoscopic device and the target, wherein the sequence of fluoroscopic image frames correspond to various poses of an imaging system acquiring the sequence of fluoroscopic image frames, ii) generating a reconstructed 3D tomosynthesis image based at least in part on the poses of the imaging system and iii) determining a location of the target based at least in part on the reconstructed 3D tomosynthesis image ([0086], In some cases, when an electromagnetic (EM) system is used, real-time imaging (e.g., tomosynthesis, EBUS, live camera) may be employed to provide corrections to EM navigation thereby enhancing the localization accuracy); and
(c) upon switching to a fluoroscopic view mode of the GUI, i) obtaining a pose of the imaging system associated with a fluoroscopic image frame acquired in the fluoroscopic view mode, and ii) generating an overlay of the target displayed onto the fluoroscopic image frame based at least in part on the pose of the imaging system and the location of the target determined in (b) ([0070], a relative orientation between the endoscope kinematic frame and a reference fluoroscopic coordinate system may be computed. For instance, by mapping motion from the fluoroscopic image data to motion in the kinematics obtained from the driving mechanism motion (e.g., compute the kinematics data and scope tip position based on the fluoroscopic image data).
Regarding claim 16, Slawinski teaches The non-transitory computer-readable media of claim 14,
wherein a location of the target displayed in the navigation mode is updated based on the location of the target determined in (b) ([0086] The multimodal sensing feature of the present disclosure may include combining the multiple sensing modalities using a unique fusion framework. The bronchoscope may combine electromagnetic (EM) sensor, direct imaging device, tomosynthesis, kinematics data and ultrasound imaging using a dynamic fusion framework allowing for small lung modules to be identified specifically outside the airways and automatically steer the bronchoscope towards the target. ).
Regarding claim 17, Slawinski teaches The non-transitory computer-readable media of claim 14,
wherein the poses of the imaging system in the tomosynthesis mode are estimated using a marker contained in the sequence of fluoroscopic image frames, and wherein the marker has a 3D pattern ([0073], radioplaque markers within the imaging field).
Regarding claim 18, Slawinski teaches The non-transitory computer-readable media of claim 14,
wherein the poses of the imaging system in the tomosynthesis mode are measured by one or more sensors ([0007] collecting positional sensor data and kinematics data).
Regarding claim 19, Slawinski teaches The non-transitory computer-readable media of claim 14,
wherein the pose of the imaging system associated with the fluoroscopic image frame in the fluoroscopic view mode is estimated using a marker contained in the fluoroscopic image frame and wherein the marker comprises a plurality of features placed on at least two different planes or wherein the marker has a plurality of features of different sizes arranged in a coded pattern ([0074] FIG. 5 shows an example of a radiopaque marker 503 attached to the catheter tip 501 for pose estimation. As shown in the figure, a radiopaque pattern is placed on the tip of an endoscope and imaged by fluoroscopic imaging).
Regarding claim 20, Slawinski teaches The non-transitory computer-readable media of claim 14,
wherein in the tomosynthesis mode, the one or more operations further comprise determining whether a fluoroscopic image frame from the sequence of fluoroscopic image frames is unique based at least in part on an intensity comparison ([0073] Fluoroscopy is an imaging modality that obtains real-time moving images of patient anatomy, medical instruments, and any radiopaque markers within the imaging field using X-rays. Fluoroscopic systems may include C-arm systems which provide positional flexibility and are capable of orbital, horizontal, and/or vertical movement via manual or automated control. Non-C-arm systems are stationary and provide less flexibility in movement. Fluoroscopy systems generally use either an image intensifier or a flat-panel detector to generate two dimensional real-time images of a patient anatomy. Bi-planar fluoroscopy systems simultaneously capture two fluoroscopic images, each from different (often orthogonal) viewpoints. In the presented methods, a radiopaque marker disposed at the tip of the catheter may be visible by the fluoroscopic imaging and is analyzed for estimating a pose of the catheter or the camera.).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 2, 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Slawinski as applied to claims 1, 14 above, and further in view of Graetzel (US 20230072879 A1).
Regarding claim 2, Slawinski teaches The computer-implemented method of claim 1,
wherein the virtual view in the navigation mode comprises upon determining a distal tip of the endoscopic device is within a predetermined proximity of the target ([0086] The multimodal sensing feature of the present disclosure may include combining the multiple sensing modalities using a unique fusion framework. The bronchoscope may combine electromagnetic (EM) sensor, direct imaging device, tomosynthesis, kinematics data and ultrasound imaging using a dynamic fusion framework allowing for small lung modules to be identified specifically outside the airways and automatically steer the bronchoscope towards the target. ),
Slawinski does not explicitly teach rendering a graphical representation of the target and an indicator indicative of an angle of the target relative to an exit axis of a working channel of the endoscopic device.
However, Graetzel teaches rendering a graphical representation of the target and an indicator indicative of an angle of the target relative to an exit axis of a working channel of the endoscopic device ([0035] The system can also display a working channel indicator via the user interface, which indicates an angle of rotation of the working channel (e.g., a position of the working channel relative to the image representation)).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the display of Slawinski to include a working channel angle indicator as taught in Graetzel in order to assist a user in maintaining a frame of reference within the patient (Graetzel [0035]).
Regarding claim 15, Slawinski teaches The non-transitory computer-readable media of claim 14,
wherein the virtual view in the navigation mode comprises upon determining a distal tip of the endoscopic device is within a predetermined proximity of the target ([0086] The multimodal sensing feature of the present disclosure may include combining the multiple sensing modalities using a unique fusion framework. The bronchoscope may combine electromagnetic (EM) sensor, direct imaging device, tomosynthesis, kinematics data and ultrasound imaging using a dynamic fusion framework allowing for small lung modules to be identified specifically outside the airways and automatically steer the bronchoscope towards the target. ),
Slawinski does not explicitly teach rendering a graphical representation of the target and an indicator indicative of an angle of the target relative to an exit axis of a working channel of the endoscopic device.
However, Graetzel teaches rendering a graphical representation of the target and an indicator indicative of an angle of the target relative to an exit axis of a working channel of the endoscopic device ([0035] The system can also display a working channel indicator via the user interface, which indicates an angle of rotation of the working channel (e.g., a position of the working channel relative to the image representation)).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the display of Slawinski to include a working channel angle indicator as taught in Graetzel in order to assist a user in maintaining a frame of reference within the patient (Graetzel [0035]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TIMOTHY TUAN LUU whose telephone number is (703)756-4592. The examiner can normally be reached Monday-Tuesday, Thursday-Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Carey can be reached at 5712707235. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TIMOTHY TUAN LUU/ Examiner, Art Unit 3795
/MICHAEL J CAREY/ Supervisory Patent Examiner, Art Unit 3795
Prosecution Insights