DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Priority
Applicant's arguments filed 2/9/26 (“Remarks”) have been fully considered but they are not persuasive.
Applicant argues that
[s]upport for the feature ‘accessing, by a computing system from a database, a three-diemsnioanl representative model generated using a first tomogram derived from scanning a volume within a subject prior to an invasive procedure’ may be found
in at least paragraph [0022] of the 62/965,264 Application (the “‘264 App”). Remarks at 10.
First, Applicant fails to address the response to arguments set forth in the Non-Final Rejection mailed 11/7/2025 (“NF”) at 2-3 regarding similar claim features.
Second, contrary to Applicant’s arguments, paragraph [0022] does not provide support for “accessing, by a computing system from a database, a three-dimensional representative model generated using a first tomogram derived from scanning a volume within a subject prior to an invasive procedure.” At most, paragraph [0022] provides accessing, by a computing system (“the data acquirer 140 may retrieve”) a tomogram derived from scanning a volume within a subject (“previously generated biomedical image of the lung… a CT scan previously perform on the subject”). The generated model in paragraph [0022] is generated after “retriev[ing], identify[ing[, or receiv[ing] the one or more biomedical images from the circumferential imaging device.” There is no disclosure of accessing the first tomogram from a database or that the previously generated biomedical image was taken “prior to an invasive procedure.” Paragraph [0020] further appears to indicate that the retrieved biomedical image was acquired intraoperatively (“the generation and transmission of the biomedical image may be in real-time or near-real time… the subject 115 may be operated using biomedical images acquired of the patient in real-time”). Additionally, paragraph [0026] provides to “create or generate a virtual 3D position map of the robotic bronchoscope within the lung using a prior CT scan and real-time endoscopic visual landmarks.” There is no disclosure that the “prior CT scan” was accessed from a database or that the “prior CT scan” was taken “prior to an invasive procedure.”
Therefore, Applicant’s argument is not persuasive.
Applicant argues that
[s]upport for the feature ‘identifying, by the computing system, using a computer vision algorithm on the three-dimensional representative model generated using the first tomogram, a first region of interest (ROI) corresponding to a target within the volume of the subject prior to the invasive procedure’ may be found
in at least paragraph [0023] of the ‘264 App. Remarks at 10.
First, Applicant fails to address the response to arguments set forth in the NF at 3 regarding the absence of support for identifying a target in the first tomogram.
Second, contrary to Applicant’s arguments, paragraph [0023] does not provide support for “using a computer vision algorithm on the three-dimensional representative model generated using the first tomogram.” As discussed above, regarding paragraph [0022], the “biomedical image from the circumferential imaging device” and “the model generated using the biomedical image” disclose in paragraph [0023] does not disclose accessing the first tomogram from a database or that the previously generated biomedical image was taken “prior to the invasive procedure.” Therefore, paragraph [0023] cannot provide support for “the three-dimensional representative model generated using the first tomogram” as recited. Paragraph [0023] at most discloses “perform[ing] image pattern recognition… to correlate image features between the biomedical image and the sensory data.” Assuming that “image pattern recognition” discloses the asserted “computer vision algorithm,” using “image pattern recognition” to correlate/identify features this does not disclose identifying “a first region of interest (ROI) corresponding to a target within the volume of the subject.” In addition, while paragraphs [0024] and [0026] mention an “intended target” or “target lesion,” neither paragraph [0024] nor [0026] disclose that the “image pattern recognition” identifies the “intended target” or “target lesion,” that the “intended target” or “target lesion” is identified from the “three-dimensional representative model,” or that the “intended target” or “target lesion” is identified “within the volume of the subject prior to the invasive procedure.”
Therefore, Applicant’s argument is not persuasive.
Applicant argues that
[s]upport for the feature ‘acquiring by the computing system, via an endoscopic device at least partially disposed within a tract of the subject at a time instance during the invasive procedure, data comprising operational data identifying a movement of a distal end of the endoscopic device through the tract’ may be found
in at least paragraphs [0022] and [0024] of the ‘264 App. Remarks at 10.
First, Applicant fails to address the response to arguments set forth in the NF at 3-4 regarding similar claim features.
Second, contrary to Applicant’s arguments paragraphs [0022] and [0024] do not provide support for “acquiring… via an endoscopic device… data comprising operational data identifying a movement of a distal end of the endoscopic device through the tract.” At most, paragraph [0022] provides that “the data acquirer 140 may retrieve, identify, or receive the sensory data from the endoscope device 120 (e.g., from camera at distal end 125).” This does not provide support for intraoperative acquisition of operational data identifying a movement of a distal end of the endoscopic device through the tract as there is no disclosure that the sensory data identifies such movement.
At most, paragraph [0024] provides for “a video output from the endoscope device may include a virtual 3D rendering of the distal end 125 within the lung 305… and [t]he 3D model may allow for estimating the direction that the direction in 3D space that the endoscope device 120 to adjust the distal end 12.” This does not provide support for intraoperative acquisition of operational data identifying a movement of a distal end of the endoscopic device through the tract as there is no disclosure that the video output form the endoscopic device or virtual 3D rendering data identifies such movement.
Additionally, Applicant’s specification in the present application makes three mentions of “operational data” in paragraphs [0010], [0032], and [0036], respectively. Paragraph [0010] states “operational data identifying a translation of the endoscope through the subject.” Paragraph [0032] states
[i]n some embodiments, the data acquired via the endoscopic device 106 may include operational data of the endoscope device 106. The operational data may include, for example, information on movement (e.g., translation, curvature, and length) of the distal end 114 and the catheter 112 through the subject 110.
Paragraph [0035] states
[f]or example, the model mapper 118 may use the operational data from the endoscopic device 106 to estimate the point location of the distal end 114 within the cavity 405 of the organ 130.
From this limited disclosure, Applicant’s alleged claim feature of “operational data” appears to broadly encompass any information on movement of the distal end of the catheter through the subject such as information of the translation, curvature, and length. There is no disclosure of the manner by which the operational data/information is acquired by the endoscope device. Therefore, the plain meaning of the alleged claim feature of “operational data” appears to encompass any means by which the endoscope device acquires data/information on movement of the distal end of the catheter through the subject.
However, the disclosure of the ‘264 application, particularly, paragraphs [0022] and [0024] does not disclose any means by which the endoscope device acquires data/information on movement of the distal end of the catheter through the subject. Therefore, Applicant’s argument is not persuasive.
Applicant argues that
[s]upport for the feature ‘determining, by the computing system, in the first tomogram of the subject, a first relative location of the distal end of the endoscopic device to the target based on the data acquired via the endoscopic device’ may be found
in at least paragraph [0024] of the ‘264 App. Remarks at 10.
First, Applicant fails to address the response to arguments set forth in the NF at 4-5 regarding similar claim features.
Second, contrary to Applicant’s arguments, paragraph [0024] does not provide support for “determining… in the first tomogram of the subject.” As discussed above, regarding paragraph [0022], the “biomedical image from the circumferential imaging device” and “the 3D-rendering from the circumferential imaging device 110” disclosed in paragraph [0024] does not disclose accessing the first tomogram from a database or that the previously generated biomedical image was taken “prior to the invasive procedure.” Therefore, paragraph [0024] cannot provide support for “determining, in the first tomogram of the subject.”
Therefore, Applicant’s argument is not persuasive.
Applicant argues that
[s]upport for the feature ‘receiving, by the computing system using a tomograph, a second tomogram of the volume within the subject at the time instance during the invasive procedure, the second tomogram including the distal end of the endoscopic device and the target within the volume of the subject’ may be found
in at least paragraphs [0022]-[0025] of the ‘264 App. Remarks at 11.
There does not appear to be explicit support in paragraphs [0022]-[0025] for “the second tomogram including… the target within the volume of the subject.” Paragraph [0024] discloses “the real-time image of the distal end 125 within the lung 305 with 3D-rendering from the circumferential imaging device” and “[t]he 3D model may allow for estimating the direction that the direction in 3D space that the endoscope device 120 to adjust the distal end 125 is order to reach the intended target within the lung 305.” However, this does not appear to explictly disclose that the “intended target within the lung” is included in the 3D model derived from the intraoperative tomogram.
Additionally, paragraph [0026] discloses “a virtual 3D position map of the robotic bronchoscope in the lung using a prior CT scan and real-time endoscopic visual landmarks,” “calculate an estimate of virtual distances,” and “[t]he virtual distance may include, for example: the distance between the robotic catheter to the proximal end of the target lesion, distance of the robotic catheter to the distal end of the lesion.” However, this does not appear to explictly disclose that the “target lesion” or “lesion” is included in the 3D model derived from the intraoperative tomogram.
Therefore, Applicant’s argument is not persuasive.
Applicant argues that
[s]upport for the feature ‘identifying, by the computing system, using the computer vision algorithm on the second tomogram, (i) a second ROI corresponding to the target and (ii) the distal end of the endoscopic device within the volume of the subject at the time instance’ may be found
in at least paragraphs [0023]-[0025] of the ‘264 App. Remarks at 11.
There does not appear to be explicit support in paragraphs [0023]-[0025] for “identifying… (i) a second ROI corresponding to the target.” Paragraph [0024] discloses “the real-time image of the distal end 125 within the lung 305 with 3D-rendering from the circumferential imaging device” and “[t]he 3D model may allow for estimating the direction that the direction in 3D space that the endoscope device 120 to adjust the distal end 125 is order to reach the intended target within the lung 305.” However, this does not appear to explictly disclose that the “second ROI corresponding to the target” is included in the 3D model derived from the intraoperative tomogram.
Additionally, paragraph [0026] discloses “a virtual 3D position map of the robotic bronchoscope in the lung using a prior CT scan and real-time endoscopic visual landmarks,” “calculate an estimate of virtual distances,” and “[t]he virtual distance may include, for example: the distance between the robotic catheter to the proximal end of the target lesion, distance of the robotic catheter to the distal end of the lesion.” However, this does not appear to explictly disclose that the “target lesion” or “lesion” is included in the 3D model derived from the intraoperative tomogram.
Therefore, Applicant’s argument is not persuasive.
Applicant argues that
[s]upport for the feature ‘registering, by the computing system, the second tomogram received from the tomograph during the invasive procedure with the first tomogram obtained prior to the invasive procedure based on the first ROI identified in the first tomogram and the second ROI identified in the second tomogram’ may be found
in at least paragraphs [0023] and [0024] of the ‘264 App. Remarks at 11.
First, Applicant fails to address the response to arguments set forth in the NF at 6-7 regarding similar claim features.
Second, contrary to Applicant’s arguments, paragraphs [0023] and [0024] do not provide support for “the first tomogram obtain prior to the invasive procedure.” As discussed above, regarding paragraph [0022], the “biomedical image from the circumferential imaging device”, “the model generated using the biomedical image” disclose in paragraph [0023], and “the 3D-rendering from the circumferential imaging device 110” disclosed in paragraph [0024] do not disclose accessing the first tomogram from a database or that the previously generated biomedical image was taken “prior to the invasive procedure.” Therefore, paragraphs [0023] and [0024] cannot provide support for “determining, in the first tomogram of the subject.”
Third, paragraphs [0023] and [0024] do not provide support for “the first ROI identified in the first tomogram.” As discussed above, paragraph [0023] at most discloses “perform[ing] image pattern recognition… to correlate image features between the biomedical image and the sensory data.” Assuming that “image pattern recognition” discloses the asserted “computer vision algorithm,” using “image pattern recognition” to correlate/identify features this does not disclose identifying “a first region of interest (ROI) corresponding to a target within the volume of the subject.” In addition, while paragraphs [0024] and [0026] mention an “intended target” or “target lesion,” neither paragraph [0024] nor [0026] disclose that the “image pattern recognition” identifies the “intended target” or “target lesion,” that the “intended target” or “target lesion” is identified from the “three-dimensional representative model,” or that the “intended target” or “target lesion” is identified “within the volume of the subject prior to the invasive procedure.”
Fourth, paragraphs [0023] and [0024] do not provide explicit support for “the second ROI identified in the second tomogram. As discussed above, paragraph [0024] discloses “the real-time image of the distal end 125 within the lung 305 with 3D-rendering from the circumferential imaging device” and “[t]he 3D model may allow for estimating the direction that the direction in 3D space that the endoscope device 120 to adjust the distal end 125 is order to reach the intended target within the lung 305.” However, this does not appear to explictly disclose that the “second ROI corresponding to the target” is included in the 3D model derived from the intraoperative tomogram.
Additionally, paragraph [0026] discloses “a virtual 3D position map of the robotic bronchoscope in the lung using a prior CT scan and real-time endoscopic visual landmarks,” “calculate an estimate of virtual distances,” and “[t]he virtual distance may include, for example: the distance between the robotic catheter to the proximal end of the target lesion, distance of the robotic catheter to the distal end of the lesion.” However, this does not appear to explictly disclose that the “target lesion” or “lesion” is included in the 3D model derived from the intraoperative tomogram.
Therefore, Applicant’s argument is not persuasive.
Applicant argues that
[s]upport for the feature ‘determining, by the computing system, from registering the second tomogram with the first tomogram…’ may be found
in at least paragraphs [0024] and [0025] of the ‘264 App. Remarks at 11.
First, Applicant fails to address the response to arguments set forth in the NF at 6-7 regarding similar claim features.
Second, as discussed above regarding the disclosure of paragraphs [0024] and [0025], the ‘264 app contains no disclosure of acquiring/retrieving a preoperative tomogram. Thus, the ‘264 app cannot provide support for the alleged claim features. On this ground alone, Applicant’s argument is not persuasive.
Third, at most paragraph [0024] discloses “creat[ing] a 3D model of the lung 305 by merging the images and identifying image overlap using a registration process… confirm, match, or correlate anatomically recognized points.” The images being merged here via registration is an intraoperative tomogram as discussed above regarding paragraphs [0022]-[0023] with “sensory data” which in paragraph [0021] refers to “an image… acquired via camera on the distal end.” Thus, neither the intraoperative tomogram or the intraoperative endoscopic camera images being merged/registered provide support for a “preoperative tomogram” let alone provide support for registering the second, intraoperative tomogram with the first, pre-operative tomogram.
Fourth, at most paragraph [0025] discloses “[w]ith the identification of the portion within the biomedical image, the tool interface 145 may determine a location of the distal end 125 of the endoscope device 120 within the lung 305 of the subject 115.” In summary, paragraph [0025] discloses that the registration between the second, intraoperative tomogram with the sensory image data from the endoscope camera allows for identification of the distal end of the endoscope within the second, intraoperative tomogram. There is no support in paragraph [0025] for registering the second, intraoperative tomogram with the first, pre-operative tomogram.
Therefore, Applicant’s argument is not persuasive.
Applicant argues that
[s]upport for the feature ‘determining, by the computing system, based on registering the second tomogram with the three-dimensional representative model, a second relative location of the distal end of the endoscopic device to the target within the subject’ may be found
in at least paragraph [0025] of the ‘264 App. Remarks at 11.
First, as discussed above, at most paragraph [0025] discloses “[w]ith the identification of the portion within the biomedical image, the tool interface 145 may determine a location of the distal end 125 of the endoscope device 120 within the lung 305 of the subject 115.” In summary, paragraph [0025] discloses that the registration between the second, intraoperative tomogram with the sensory image data from the endoscope camera allows for identification of the distal end of the endoscope within the second, intraoperative tomogram. There is no support in paragraph [0025] for registering the second, intraoperative tomogram with the first, pre-operative tomogram, let alone registering the second, intraoperative tomogram with a three-dimensional representative model derived from the first, pre-operative tomogram. Further, as discussed above, the generated model in paragraph [0022] is generated after “retriev[ing], identify[ing[, or receiv[ing] the one or more biomedical images from the circumferential imaging device.” There is no disclosure of accessing the first tomogram from a database or that the previously generated biomedical image was taken “prior to an invasive procedure.” Therefore, there is no support for a three-dimensional representative model derived from the first, pre-operative tomogram.
Second, there is no disclosure in paragraph [0025] of “determining… a second relative location of the distal end of the endoscopic device to the target within the subject.” Paragraph [0025] at most discloses using the registration between the second, intraoperative tomogram and the images from the endoscope to identify the location of the distal end of the endoscope in the second, intraoperative tomogram. There is no support for identifying a target within the subject.
Additionally, as discussed above, paragraph [0026] discloses “a virtual 3D position map of the robotic bronchoscope in the lung using a prior CT scan and real-time endoscopic visual landmarks,” “calculate an estimate of virtual distances,” and “[t]he virtual distance may include, for example: the distance between the robotic catheter to the proximal end of the target lesion, distance of the robotic catheter to the distal end of the lesion.” However, this does not disclose that the “virtual distance” is “based on registering the second tomogram with the three-dimensional representative model” derived from the pre-operative tomogram.
Applicant argues that
[s]upport for the feature ‘providing, by the computing system for display, a user interface to visualize the movement of the distal end of the endoscopic device through the tract of the subject, the user interface comprising an indicator corresponding to the second relative location of the distal end to the target within the tract of the subject during the invasive procedure’ may be found
in at least paragraph [0026] and Figs. 4A-C of the ‘264 App. Remarks at 11.
First, there is no support in paragraph [0026] or Figs. 4A-C of “visualiz[ing] the movement of the distal end of the endoscopic device through the tract of the subject.” Paragraph [0026] makes no mention of visualizing movement of the distal end of the endoscopic device. Figs. 4A-C show a position of the distal end of the endoscopic device, but do not show a visualization of the movement of the distal end of the endoscopic device.
In addition, as discussed above, there is no support in paragraphs [0022] and [0024] for “acquiring… via an endoscopic device… data comprising operational data identifying a movement of a distal end of the endoscopic device through the tract.” At most, paragraph [0022] provides that “the data acquirer 140 may retrieve, identify, or receive the sensory data from the endoscope device 120 (e.g., from camera at distal end 125).” This does not provide support for intraoperative acquisition of operational data identifying a movement of a distal end of the endoscopic device through the tract as there is no disclosure that the sensory data identifies such movement.
At most, paragraph [0024] provides for “a video output from the endoscope device may include a virtual 3D rendering of the distal end 125 within the lung 305… and [t]he 3D model may allow for estimating the direction that the direction in 3D space that the endoscope device 120 to adjust the distal end 12.” This does not provide support for intraoperative acquisition of operational data identifying a movement of a distal end of the endoscopic device through the tract as there is no disclosure that the video output form the endoscopic device or virtual 3D rendering data identifies such movement.
Additionally, Applicant’s specification in the present application makes three mentions of “operational data” in paragraphs [0010], [0032], and [0036], respectively. Paragraph [0010] states “operational data identifying a translation of the endoscope through the subject.” Paragraph [0032] states
[i]n some embodiments, the data acquired via the endoscopic device 106 may include operational data of the endoscope device 106. The operational data may include, for example, information on movement (e.g., translation, curvature, and length) of the distal end 114 and the catheter 112 through the subject 110.
Paragraph [0035] states
[f]or example, the model mapper 118 may use the operational data from the endoscopic device 106 to estimate the point location of the distal end 114 within the cavity 405 of the organ 130.
From this limited disclosure, Applicant’s alleged claim feature of “operational data” appears to broadly encompass any information on movement of the distal end of the catheter through the subject such as information of the translation, curvature, and length. There is no disclosure of the manner by which the operational data/information is acquired by the endoscope device. Therefore, the plain meaning of the alleged claim feature of “operational data” appears to encompass any means by which the endoscope device acquires data/information on movement of the distal end of the catheter through the subject.
However, the disclosure of the ‘264 application, particularly, paragraphs [0022] and [0024] does not disclose any means by which the endoscope device acquires data/information on movement of the distal end of the catheter through the subject.
Second, as discussed above, paragraph [0026] discloses “a virtual 3D position map of the robotic bronchoscope in the lung using a prior CT scan and real-time endoscopic visual landmarks,” “calculate an estimate of virtual distances,” and “[t]he virtual distance may include, for example: the distance between the robotic catheter to the proximal end of the target lesion, distance of the robotic catheter to the distal end of the lesion.” However, this does not disclose that the “virtual distance” is “based on registering the second tomogram with the three-dimensional representative model” derived from the pre-operative tomogram.
Therefore, Applicant’s arguments are not persuasive.
Therefore, as the alleged disclosures from the ‘264 app do not provide support for the alleged claim features Applicant’s arguments are not persuasive.
103 Rejections
Applicant’s arguments filed 2/9/26 have been fully considered but they are not persuasive.
First, Applicant argues that
the combination of Duindam and Krimsky fails to teach or suggest a computing system using a computer vision algorithm to identify (1) a first ROI corresponding to a target in a three-dimensional representative model prior to the invasive procedure and (2) a second ROI corresponding to the target in a tomogram at a time instance during the invasive procedure.’
Remarks at 14.
Contrary to Applicant’s arguments, as detailed in infra rejections, Duindam discloses the alleged claim features.
Regarding the first alleged feature, at least paragraph [0037] discloses a three-dimensional model created from a three-dimensional preoperative image data set. At least paragraph [0062] discloses a three-dimensional preoperative image data set. At least paragraphs [0065]-[0070] disclose creating a three-dimensional model from the three-dimensional preoperative image data set of paragraph [0062] and using computer vision algorithms including segmentation, filtering, and image characteristics analysis to identify anatomical structures, instruments, and targets such as a tumor within the three-dimensional model. At least paragraphs [0083]-[0084] discloses a three-dimensional model created from a three-dimensional preoperative image data set. At least paragraphs [0084]-[0091] disclose using computer vision algorithms including image characteristics analysis, segmentation, and filtering to identify anatomical structures, instruments, and targets such as a tumor within the three-dimensional model. At least paragraph [0111] discloses the patient anatomy, target such as a tumor, and instrument are identified in the three-dimensional model.
Regarding the second alleged feature, at least paragraph [0037] discloses a 3D intraoperative image data set. At least paragraph [0057] discloses a 3D intraoperative image data set. At least paragraph [0062] discloses a 3D intraoperative image data set. At least paragraph [0062] discloses a 3D intraoperative image data set. At least paragraphs [0065]-[0070] disclose using computer vision algorithms including segmentation, filtering, and image characteristics analysis to identify anatomical structures, instruments, and targets such as a tumor within the 3D intraoperative image data set of paragraph [0062]. At least paragraphs [0092]-[0103] discloses using computer vision algorithms including image characteristics to identify the instrument and the target in the 3D intraoperative image data set. At least paragraph [0111] discloses the target such as a tumor, instrument, and patient tissue are identified in the intraoperative image data.
Further, paragraphs [0006], [0008], and [0082] disclose that the above is performed on a computing system using a control system processor.
Applicant further asserts that Duindam does not employ computer vision algorithms to identify a target and is limited to a navigation path. Remarks at 14. As discussed above, at least paragraphs [0065]-[0070], [0083]-[0091], and [0111] disclose computer vision algorithms to identify a target in the 3D preoperative model.
Applicant further asserts that Duindam only discloses identifying structures and instruments. Remarks at 14. As discussed above, at least paragraphs [0065]-[0070], [0083]-[0091], and [0111] disclose computer vision algorithms to identify a target in the 3D preoperative model.
Applicant further asserts that segmentation in Duindam refers only to intraoperative image data. Remarks at 14-15. As discussed above, at least paragraphs [0062], [0065]-[0070], [0083]-[0091], and [0111] disclose that the target is identified in the 3D preoperative model.
Therefore, Duindam discloses the asserted claim features, ergo, Applicant’s arguments are not persuasive.
Second, Applicant argues that
the combination of Duindam and Krimsky fails to teach or suggest a computing system to determine, from registering the second tomogram with a three-dimensional representative model, a set of transformation parameters defining differences in visual representations between (1) a first ROI corresponding to a target in the three-dimensional representative model and (2) a second ROI corresponding to the target in the second tomogram.’
Remarks at 15.
Contrary to Applicant’s arguments, as detailed in infra rejections, Krimsky teaches the alleged claim features.
Regarding the first alleged feature, at least paragraph [0048] teaches a preoperative (pre-procedural scan) image data. At least paragraph [0051] teaches generating a 3D model of the patient’s chest from the preoperative image data. At least paragraphs [0049]-[0052] teaches using computer vision algorithms including automatic image processing and analysis to identify anatomical structures and target locations in the 3D model.
Regarding the second alleged feature, at least paragraph [0055]-[0059] teaches acquiring additional intraoperative (during the medical procedure or real-time) image data. At least paragraphs [0055]-[0067] teaches using computer vision algorithms including automatic image processing and analysis to identify anatomical structures and target locations in the intraoperative image data.
Further, paragraphs [0037] and [0068]-[0069] teach that the above is performed on a computing device for executive functions of the application.
Applicant further asserts that Krimsky only registers the intraoperative and preoperative image data using landmarks and/or fiducial markers. Remarks at 15-16. Contrary to Applicant’s contention, paragraphs [0058]-[0059] explictly provides that the intraoperative image data and preoperative 3D model are registered using alignment of the target locations identified in each including the difference between the target locations.
Applicant further asserts that Krimsky does not use computer vision to identify the target ROI. Remarks at 16. As discussed above, at least paragraphs [0049]-[0052] teaches using computer vision algorithms including automatic image processing and analysis to identify anatomical structures and target locations in the 3D model; and at least paragraphs [0055]-[0067] teaches using computer vision algorithms including automatic image processing and analysis to identify anatomical structures and target locations in the intraoperative image data.
Applicant further asserts that Krimsky relies on manual identification of the target ROI. Remarks at 16. As discussed above, at least paragraphs [0049]-[0052] teaches using computer vision algorithms including automatic image processing and analysis to identify anatomical structures and target locations in the 3D model; and at least paragraphs [0055]-[0067] teaches using computer vision algorithms including automatic image processing and analysis to identify anatomical structures and target locations in the intraoperative image data. Further, in citing paragraph [0052], Applicant appears to have overlooked the last sentence of that paragraph stating “[a]lternatively, or in addition, application 81 may automatically select one or more target locations for diagnosis and/or treatment, for example, by performing image processing and analysis.”
Therefore, Krimsky teaches the asserted claim features, ergo, Applicant’s arguments are not persuasive.
Priority
Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C. 120, 121, 365(c), or 386(c) is acknowledged. Applicant has not complied with one or more conditions for receiving the benefit of an earlier filing date under 35 U.S.C. 119(e) as follows:
The later-filed application must be an application for a patent for an invention which is also disclosed in the prior application (the parent or original nonprovisional application or provisional application). The disclosure of the invention in the parent application and in the later-filed application must be sufficient to comply with the requirements of 35 U.S.C. 112(a) or the first paragraph of pre-AIA 35 U.S.C. 112, except for the best mode requirement. See Transco Products, Inc. v. Performance Contracting, Inc., 38 F.3d 551, 32 USPQ2d 1077 (Fed. Cir. 1994).
The disclosure of the prior-filed application, Application No. 62/965,264 fails to provide adequate support or enablement in the manner provided by 35 U.S.C. 112(a) for one or more claims of this application. A review of the disclosure cited by the applicant and the entirety of the disclosure of the ‘264 app reveals that there is not support for at least the subject matter of independent claim 1, lines 2-49. The disclosure of the ‘264 app is limited to determining and visualizing by the computing system in one or more intraoperative tomograms of the subject a relative location of the distal end of an endoscopic device to the target based on the data acquired intraoperatively via the endoscopic device.
For example, claim 1 being illustrative, there is no disclosure in the ‘264 app of accessing, by a computing system from a database, a three-dimensional representative model generated using a first tomogram corresponding to a scan of a volume within a subject prior to an invasive procedure, (claim 1, lines 2-4), i.e., a 3D model generated using a preoperative tomogram. Further, there is no disclosure of using a computer vision algorithm on the 3D model to identify a target ROI (claim 1, lines 5-8). Further, there is no disclosure of acquiring intraoperatively via a endoscopic device operational data identifying a movement of a distal end of the endoscopic device through the tract (claim 1, lines 9-12). Further, there is no disclosure of determining a first relative location of the distal end of the endoscopic device to the target using the preoperative tomogram and the intraoperative endoscope data (claim 1, lines 13-15). Further, there is no explicit disclosure of receiving an intraoperative tomogram that includes a target within the volume of the subject (claim 1, lines 16-19). Further, there is no explicit disclosure of using a computer vision algorithm on the intraoperative tomogram to identify a target ROI (claim 1, lines 20-22). Further, there is no disclosure of registering the preoperative tomogram with an intraoperative tomogram based on the target identified in the pre- and intraoperative tomograms (claim 1, lines 23-26). Further, there is no disclosure of using the registration to determine a set of transformation parameters defining differences in visual representations between the distal end in the three-dimensional representative model and the intraoperative tomogram (claim 1, lines 29-32). Further, there is no disclosure of using the registration to determine a second set of transformation parameters defining differences in visual representations between the first ROI in the three-dimensional representative model and the second ROI in the intraoperative tomogram (claim 1, lines 33-35). Further, there is no disclosure of using the registration to determine a first deviation, based on the first set of transformation parameters for the distal end, between the distal end of the endoscopic device in the three-dimensional representative model and the distal end of the endoscopic device in the intraoperative tomogram (claim 1, lines 36-39). Further, there is no disclosure of using the registration to determine a second deviation, based on the second set of transformation parameters for the target, between the first ROI in the three-dimensional representative model and the second ROI in the intraoperative tomogram (claim 1, lines 40-42). Further, there is no disclosure of using the registration to determine a second relative location of the distal end of the endoscopic device to the target (claim 1, lines 43-45). Further, there is no disclosure of visualizing an indicator corresponding to the second relative location of the distal end to the target within the tract of the subject during the invasive procedure (claim 1, lines 46-49).
It is noted that in discussing the technical challenges of the prior art overcome by the solution of the ‘264 app, there is mention of using only a preoperative tomogram to guide bronchoscopy (paragraph [0016]) and using some combination of a preoperative tomogram and a robotic endoscope device to calculate a navigation path for the endoscope (paragraph [0017]). However, the discussions are not part of the solution of the ‘264 app and are expressly distinguished from and criticized as posing specific technical challenges. Moreover, neither paragraph [0016] or [0017] disclose the features of claim 1, lines 2-49.
Thus, as the disclosure of the ‘264 does not disclose the subject matter of independent claim 1, lines 2-49, the ‘264 app fails to provide adequate support or enablement in the manner provided by 35 U.S.C. 112(a) necessary to receive the benefit of the earlier filing date of 1/24/2020 for claims 1-10. Similar limitations are present in independent claim 11, ergo, the ‘264 app fails to provide adequate support or enablement in the manner provided by 35 U.S.C. 112(a) necessary to receive the benefit of the earlier filing date of 1/24/2020 for claims 11-20. See MPEP 211.05.
The earliest support for the entirety of the subject matter for each one of claims 1-20 appears to be in PCT/US21/14853 filed 01/25/2021. Therefore, the effective filing date of each of claims 1-20 is 01/25/2021.
Claim Objections
Claim 11 is objected to because of the following informalities:
Claim 11, lines 27-30 recites “(i) a first set of transformation parameters defining differences in visual representations of distal end in the three-dimensional representative model and the second tomogram.” This should read “(i) a first set of transformation parameters defining differences in visual representations between the
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 11-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 11 recites the limitation "the first ROI" in lines 22-23, 32, and 39. There is insufficient antecedent basis for this limitation in the claim. Applicant appears to have unintentionally omitted a limitation for claim 11, line 6-8 with language similar to “a first region of interest (ROI) corresponding to a target” as recited in claim 1, lines 6-7.
Dependent claims 12-20 depend from and, therefore, incorporate the limitations of claim 11 and are rejected on the same grounds as above.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-10 are rejected under 35 U.S.C. 103 as being unpatentable over Duindam et al. (U.S. Pub. No. 2019/0320878), hereinafter “Duindam,” in further view of Krimsky et al. (U.S. Pub. No. 2018/0235713), hereinafter “Krimsky.”
Regarding claim 1, Duindam discloses a method for intraoperative medical imaging (method of using registered real-time images, i.e., intraoperative images, and prior-time, i.e., preoperative images, of patient anatomy during an image-guided procedure that uses a medical instrument, Abstract), comprising:
accessing, by a computing system from a database, a three-dimensional representative model generated using a first tomogram corresponding to a scan of a volume within a subject prior to an invasive procedure (computing system, [0006], [0008]; control system processor(s), [0082]; retrieve recorded preoperative image dataset from an imaging system, [0037], [0042], [0043], [0083]; create a three-dimensional pre-operative model using a three-dimensional preoperative image data set, [0037], [0062], [0065]-[0070], [0083]-[0084], [0084]-[0091], [0111]; image-guided procedure using a medical instrument, Abstract; see also medical instrument system being inserted into the body, [0052], Figs. 3-4, 6-8, 11);
identifying, by the computing system, using a computer vision algorithm on the three-dimensional representative model generated using the first tomogram, a first region of interest (ROI) corresponding to a target within the volume of the subject prior to the invasive procedure (computing system, [0006], [0008]; control system processor(s), [0082]; create a three-dimensional model from the three-dimensional preoperative image data set and use segmentation, filtering, and image characteristics analysis to identify targets of regions of interest such as a tumor within the three-dimensional pre-operative model, [0037], [0062], [0065]-[0070], [0083]-[0084], [0084]-[0091], [0111]);
acquiring, by the computing system, via an endoscopic device at least partially disposed within a tract of the subject at a time instance during the invasive procedure (computing system, [0006], [0008]; control system processor(s), [0082]; medical instrument is a broncho/endoscope for insertion into passageways such as that of the lungs, [0035], [0045], [0050], [0052], [0059], [0060], [0076], Figs. 3-4, 6-8, 11), data comprising operational data identifying a movement of a distal end of the endoscopic device through the tract (medical instrument is a broncho/endoscope that acquires shape/position/location data through passageways such as that of the lungs, [0033], [0035], [0043], [0047]-[0048], [0071], [0094]);
determining, by the computing system, in the first tomogram of the subject, a first relative location of the distal end of the endoscopic device to the target based on the data acquired via the endoscopic device (computing system, [0006], [0008]; control system processor(s), [0082]; determine vector between the distal tip/end of the medical instrument and the target in the three-dimensional pre-operative model and display the vector, target, and distal tip/end based on the shape/position/location data/information obtained from the sensor systems of the medical instrument, [0070], [0078], [0086]-[0091], Figs. 6A and 8; see also use segmentation, filtering, and image characteristics analysis to identify the instrument distal end and targets of regions of interest such as a tumor within the three-dimensional pre-operative model, [0037], [0062], [0065]-[0070], [0083]-[0084], [0084]-[0091], [0111]);
receiving, by the computing system using a tomograph, a second tomogram of the volume within the subject at the time instance during the invasive procedure, the second tomogram including the distal end of the endoscopic device and the target within the volume of the subject (computing system, [0006], [0008]; control system processor(s), [0082]; acquire three-dimensional intraoperative/concurrent/real-time anatomic images during insertion of the medical instrument into the body including the distal end of the medical instrument and the target within the body, Abstract, [0035], [0037], [0038], [0039], [0042], [0057], [0062], [0065]-[0070], [0071]-[0075], [0092]-[0103], [0111]);
identifying, by the computing system, using the computer vision algorithm on the second tomogram, (i) a second ROI corresponding to the target (computing system, [0006], [0008]; control system processor(s), [0082]; use segmentation, filtering, and image characteristics analysis to identify targets of regions of interest such as a tumor within the three-dimensional intraoperative image data set of the body, [0037], [0057], [0062], [0065]-[0070], [0092]-[0103], [0111]) and (ii) the distal end of the endoscope device within the volume of the subject at the time instance (computing system, [0006], [0008]; control system processor(s), [0082]; use segmentation, filtering, and image characteristics analysis to identify the distal end of the instrument within the three-dimensional intraoperative image data set of the body, [0037], [0057], [0062], [0065]-[0070], [0092]-[0103], [0111]);
registering, by the computing system, the second tomogram received from the tomograph during the invasive procedure with the first tomogram obtained prior to the invasive procedure based on a first feature identified in the first tomogram and a second feature identified in the second tomogram (computing system, [0006], [0008]; control system processor(s), [0082]; register the preoperative and intraoperative image datasets based on the features detected in the preoperative and intraoperative image datasets, [0105]; repeatedly registering and determining a shift in the target between a previously acquired tomogram image dataset, such as the preoperative model, and a subsequently acquired tomogram image dataset, such as the intraoperative images, [0115]-[0116]);
determining, by the computing system (computing system, [0006], [0008]; control system processor(s), [0082]), from registering the second tomogram with the first tomogram (register the pre-operative model and the intraoperative image dataset based on one or more features detected in the three-dimensional pre-operative model and the intraoperative image dataset, [0105]; registering the three-dimensional pre-operative model and the intraoperative image dataset allows for determination of the shift in location, change in size, or change in shape of the tumor/target and of the shift in location/position or orientation of the distal tip of the instrument model/medical instrument between the preoperative model and the intraoperative image data [0111]-[0114]; repeatedly registering and determining a shift in the target and instrument location, i.e., the distal tip/end effector, between a previously acquired tomogram image dataset, such as the preoperative model, and a subsequently acquired tomogram image dataset, such as the intraoperative images, [0115]-[0116]):
(i) a set of transformation parameters defining differences in visual representations between the distal end in the three-dimensional representative model and the second tomogram (registering the three-dimensional pre-operative model and the intraoperative image dataset allows for determination of the shift in location/position or orientation of the distal tip of the instrument model/medical instrument between the three-dimensional pre-operative model and the intraoperative image data, [0111]-[0114]; repeatedly registering and determining a shift in the instrument location, i.e., the distal tip/end effector, between a previously acquired tomogram image dataset, such as the three-dimensional pre-operative model, and a subsequently acquired tomogram image dataset, such as the intraoperative images, [0115]-[0116]);
(ii) a second set of transformation parameters defining differences in visual representations between the first ROI in the three-dimensional representative model and the second ROI in the second tomogram (registering the three-dimensional pre-operative model and the intraoperative image dataset allows for determination of the shift in location, change in size, or change in shape of the tumor/target between the three-dimensional pre-operative model and the intraoperative image data, [0111]-[0114]; repeatedly registering and determining a shift in the target, between a previously acquired tomogram image dataset, such as the three-dimensional model, and a subsequently acquired tomogram image dataset, such as the intraoperative images, [0115]-[0116]);
(iii) a first deviation, based on the first set of transformation parameters for the distal end, between the distal end of the endoscopic device in the three-dimensional representative model and the distal end of the endoscopic device in the second tomogram (registering the three-dimensional pre-operative model and the intraoperative image dataset allows for determination of the shift in location/position or orientation of the distal tip of the instrument model/medical instrument between the three-dimensional pre-operative model and the intraoperative image data, [0111]-[0114]; repeatedly registering and determining a shift in the instrument location, i.e., the distal tip/end effector, between a previously acquired tomogram image dataset, such as the three-dimensional pre-operative model, and a subsequently acquired tomogram image dataset, such as the intraoperative images, [0115]-[0116]); and
(iv) a second deviation, based on the second set of transformation parameters for the target, between the first ROI in the three-dimensional representative model and the second ROI in the second tomogram (registering the preoperative model and the intraoperative image dataset allows for determination of the shift in location, change in size, or change in shape of the tumor/target between the preoperative model and the intraoperative image data, [0111]-[0114]; repeatedly registering and determining a shift in the target between a previously acquired tomogram image dataset, such as the preoperative model, and a subsequently acquired tomogram image dataset, such as the intraoperative images, [0115]-[0116]);
determining, by the computing system (computing system, [0006], [0008]; control system processor(s), [0082]), based on registering the second tomogram with the three-dimensional representative model, a second relative location of the distal end of the endoscopic device to the target within the subject (register the three-dimensional pre-operative model and the intraoperative image dataset based on one or more features detected in the three-dimensional pre-operative model and the intraoperative image dataset, [0105]; registering the three-dimensional pre-operative model and the intraoperative image dataset allows for determination of an updated location of the tumor/target relative to the distal tip of the instrument model/medical instrument within the patient including the relative distance, position, orientation, and trajectory between the tumor/target and the distal tip of the instrument model/medical instrument, [0111]-[0114]; see also repeatedly registering and determining a shift in the target and instrument location, i.e., the distal tip/end effector, between a previously acquired tomogram image dataset, such as the three-dimensional pre-operative model, and a subsequently acquired tomogram image dataset, such as the intraoperative image dataset, [0115]-[0116]); and
providing, by the computing system for display, a user interface to visualize the movement of the distal end of the endoscopic device through the tract of the subject (computing system, [0006], [0008]; control system processor(s), [0082]; display, [0034]-[0040]; user interface visualizes the real-time movement of the distal end of the endoscopic device through the passageways such as that of the lungs, [0039], [0078], [0081], [0090]), the user interface comprising an indicator corresponding to the second relative location of the distal end to the target within the tract of the subject during the invasive procedure (registering the three-dimensional pre-operative model and the intraoperative image dataset allows for determination of an updated location of the tumor/target relative to the distal tip of the instrument model/medical instrument within the patient including the relative distance, position, orientation, and trajectory between the tumor/target and the distal tip of the instrument model/medical instrument, wherein the updated relative distance, position, orientation, and trajectory between the tumor/target and the distal tip of the instrument model/medical instrument is displayed as the medical instrument is navigated through passageways such as that of the patient’s lungs, [0111]-[0114], Figs. 8 and 11).
However, while Duindam discloses the first tomogram identifying a target within the volume of the subject, the second tomogram including the target within the volume of the subject, and registering, by the computing system, the second tomogram received from the tomograph during the invasive procedure with the first tomogram obtain prior to the invasive procedure based on a first feature identified in the first tomogram and a second feature identified in the second tomogram as detailed above, Duindam does not appear to explictly disclose the registering is based on the first ROI identified in the first tomogram and the second ROI identified in the second tomogram.
However, in the same field of endeavor of image guided navigation of endoscopes, Krimsky teaches a method for intraoperative medical imaging (a method of navigating a tool inside a luminal network in the patient’s lungs using intra-procedural computed tomography image data, [0006]), comprising:
accessing, by a computing system from a database, a three-dimensional representative model generated using a first tomogram corresponding to a scan of a volume within a subject prior to an invasive procedure (computing device receives previously acquired pre-procedural computed tomography scan image dataset such as from a DICOM standard storage, [0048]; generate a three-dimensional pre-operative model from the preoperative image data, [0049]-[0052]; see also storage of image data in memory or in a database, [0059], [0068]);
identifying, by the computing system, using a computer vision algorithm on the three-dimensional representative model generated using the first tomogram, a first region of interest (ROI) corresponding to a target within the volume of the subject prior to the invasive procedure (generate a three-dimensional pre-operative model from the preoperative image data and use automatic image processing and analysis to identify anatomical structures and target locations in the three-dimensinoal pre-operative model, [0049]-[0052]);
acquiring, by the computing system (application receives EM sensor tracking, [0054]; computing device for executing functions of the application, [0037], [0068]-[0069]), via an endoscopic device at least partially disposed within a tract of the subject at a time instance during the invasive procedure, data comprising operational data identifying a movement of a distal end of the endoscopic device through the tract (determine tracked location of the EM sensor on the distal end of the locatable guide (LG) intraoperatively as the locatable guide (LG) is moved through the lungs/airway, [0006], [0033], [0035]-[0036], [0039]-[0040], [0054], [0058], [0060], [0063], [0065]);
determining, by the computing system (image processing performed by an application, [0051]-[0054], [0056]; computing device for executing functions of the application, [0037], [0068]-[0069]), in the first tomogram of the subject, a first relative location of the distal end of the endoscopic device to the target based on the data acquired via the endoscopic device (determine tracked location of the EM sensor on the distal end of the locatable guide (LG) relative to the position of the target location in the pre-procedural image dataset/3D model, [0051]-[0054], [0058], Fig. 1);
receiving, by the computing system using a tomograph, a second tomogram of the volume within the subject at the time instance during the invasive procedure (application receives an additional computed tomography scan image data set of the patient’s body such as of the airways intra-procedurally, [0055]-[0059]; computing device for executing functions of the application, [0037], [0068]-[0069]), the second tomogram including the distal end of the endoscopic device and the target within the volume of the subject (use automatic image processing and analysis in the intraoperative image data to produce intra-procedural image dataset includes the position of the EM sensor on the distal end of the locatable guide (LG) and the position of the target within the patient’s body such as the airways, [0055]-[0067]);
identifying, by the computing system (application receives an additional computed tomography scan image data set of the patient’s body such as of the airways intra-procedurally, [0055]-[0059]; computing device for executing functions of the application, [0037], [0068]-[0069]), using the computer vision algorithm on the second tomogram, (i) a second ROI corresponding to the target and (ii) the distal end of the endoscope device within the volume of the subject at the time instance (use automatic image processing and analysis in the intraoperative image data to produce intra-procedural image dataset includes the position of the EM sensor on the distal end of the locatable guide (LG) and the position of the target within the patient’s body such as the airways, [0055]-[0067]);
registering, by the computing system (image processing performed by an application, [0056]; computing device for executing functions of the application, [0037], [0068]-[0069]), the second tomogram received from the tomograph during the invasive procedure with the first tomogram obtained prior to the invasive procedure based on the first ROI identified in the first tomogram and the second ROI identified in the second tomogram (registration of additional intra-procedural image data to the pre-procedural image data/3D model based on the target location identified in each of the pre-procedural image data/3D model and the additional intra-procedural image data, [0058]-[0059]);
determining, by the computing system (image processing performed by an application, [0056]; computing device for executing functions of the application, [0037], [0068]-[0069]), from registering the second tomogram with the first tomogram (registration of additional intra-procedural image data to the pre-procedural image data/3D model based on the target location identified in each of the pre-procedural image data/3D model and the additional intra-procedural image data, [0058]-[0059]; registering the intra-procedural image data to the pre-procedural image data/3D model allows for determination of the difference in position/movement of the target location and the difference in position/movement of the EM sensor on the distal end of the locatable guide (LG) between the preoperative model and the intraoperative image data, [0059]-[0061]):
(i) a set of transformation parameters defining differences in visual representations between the distal end in the three-dimensional representative model and the second tomogram (registering the intra-procedural image data to the pre-procedural image data/3D model allows for determination of the difference in position/movement of the EM sensor on the distal end of the locatable guide (LG) between the three-dimensional preoperative model and the intraoperative image data, [0059]-[0061]);
(ii) a second set of transformation parameters defining differences in visual representations between the first ROI in the three-dimensional representative model and the second ROI in the second tomogram (registering the intra-procedural image data to the pre-procedural image data/3D model allows for determination of the difference in position/movement of the target location between the three-dimensional preoperative model and the intraoperative image data, [0059]-[0061]);
(iii) a first deviation, based on the first set of transformation parameters for the distal end, between the distal end of the endoscopic device in the three-dimensional representative model and the distal end of the endoscopic device in the second tomogram (registering the intra-procedural image data to the pre-procedural image data/3D model allows for determination of the difference in position/movement of the EM sensor on the distal end of the locatable guide (LG) between the three-dimensional preoperative model and the intraoperative image data, [0059]-[0061]); and
(iv) a second deviation, based on the second set of transformation parameters for the target, between the first ROI in the three-dimensional representative model and the second ROI in the second tomogram (registering the intra-procedural image data to the pre-procedural image data/3D model allows for determination of the difference in position/movement of the target location between the three-dimensional preoperative model and the intraoperative image data, [0059]-[0061]);
determining, by the computing system (image processing performed by an application, [0056]; computing device for executing functions of the application, [0037], [0068]-[0069]), based on registering the second tomogram with the three-dimensional representative model, a second relative location of the distal end of the endoscopic device to the target within the subject (registration of additional intra-procedural image data to the pre-procedural image data/3D model based on the target location identified in each of the pre-procedural image data/3D model and the additional intra-procedural image data, [0058]-[0059]; registering the additional intra-procedural image data to the pre-procedural image data/3D model allows for determination of an updated tracked location of the EM sensor on the distal end of the locatable guide (LG) relative to the position of the target within the patient’s body such as within the patient’s airways including the relative position and pathway between the tracked location of the EM sensor on the distal end of the locatable guide (LG) and the target location, [0058], [0063]);
providing, by the computing system for display, a user interface to visualize the movement of the distal end of the endoscopic device through the tract of the subject (computing device includes a display device, [0014], [0016]-[0017], [0020], [0041], [0068]; display provides a user interface, [0068]; display/user interface displays the real-time movement of the location of the EM sensor on the distal end of the locatable guide (LG), [0006], [0008], [0012], [0014], [0016]-[0017], [0020], [0029], [0040], [0054], [0057], [0061]-[0062], [0065]), the user interface comprising an indicator corresponding to the second relative location of the distal end to the target within the tract of the subject during the invasive procedure (registration of additional intra-procedural image data to the pre-procedural image data/3D model based on the target location identified in each of the pre-procedural image data/3D model and the additional intra-procedural image data, [0058]-[0059]; registering the additional intra-procedural image data to the pre-procedural image data/3D model allows for determination of an updated tracked location of the EM sensor on the distal end of the locatable guide (LG) relative to the position of the target within the patient’s body such as within the patient’s airways including the relative position and pathway between the tracked location of the EM sensor on the distal end of the locatable guide (LG) and the target location, wherein the updated relative position and pathway between the tracked location of the EM sensor on the distal end of the locatable guide (LG) and the target location is displayed using the display/user interface as the medical instrument is navigated through the patient’s body such as through the patient’s airways, [0011]-[0012], [0019], [0058], [0063]).
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to have applied Krimsky’s known technique of performing registration using the target position in the pre-procedural and intra-procedural images to Duindam’s known process of performing registration using features in the pre-procedural and intra-procedural images to achieve the predictable result that intra-procedural movement such as that of the respiratory cycle may be accounted and/or compensated for in updating the location of the LG/EWC/EM sensor in relation to the target location. See, e.g., Krimsky, [0055], [0067].
Regarding claim 2, Duindam discloses receiving, by the computing system using the tomograph, a third tomogram of the volume within the subject at a second time instance during the invasive procedure after the time instance, the third tomogram including the distal end of the endoscopic device moved subsequent to provision of the second relative location (computing system, [0006], [0008]; control system processor(s), [0082]; acquire second set of intraoperative anatomic images at a second time during insertion of the medical instrument into the body including any shift in the anatomy, target, and instrument location, [0115]-[0116]; see also repeated real-time intraoperative images taken for real-time updates of the displayed trajectory vector, target, and distal tip/end, [0002], [0078], [0090], [0092], [0111]- [0114]);
registering, by the computing system, the third tomogram received from the tomograph at the second time instance with the first tomogram received prior to the invasive procedure (computing system, [0006], [0008]; control system processor(s), [0082]; register the preoperative, first intraoperative, and second intraoperative image datasets to one another to determine any shift in the anatomy, target, and instrument location, [0115]-[0116]);
determining a third relative location of the distal end of the endoscopic device and the target within the subject (register the preoperative, first intraoperative, and second intraoperative image datasets to one another to determine any shift in the anatomy, target, and instrument location, [0115]-[0116]; see also capturing a plurality of intraoperative images as a continuous/real-time process during the medical instrument navigation to continuously update the shifted relative position between the distal tip/end of the medical instrument and the target in the image model using the second set of intraoperative images and display the vector, target, and distal tip/end, [0115]-[0116]; see also repeated real-time intraoperative images taken for real-time updates of the displayed trajectory vector, target, and distal tip/end, [0002], [0078], [0090], [0092], [0111]- [0114]); and
providing, by the computing system for display, the user interface (computing system, [0006], [0008]; control system processor(s), [0082]; display, [0034]-[0040]; user interface visualizes the real-time movement of the distal end of the endoscopic device through the passageways such as that of the lungs, [0039], [0078], [0081], [0090]) comprising the indicator corresponding to the third relative location of the distal end to the target within the tract of the subject (update the shifted relative position between the distal tip/end of the medical instrument and the target in the image model using the second set of intraoperative images and display the vector, target, and distal tip/end, [0115]-[0116]; see also repeated real-time intraoperative images taken for real-time updates of the displayed trajectory vector, target, and distal tip/end, [0002], [0078], [0090], [0092], [0111]- [0114]; user interface visualizes the real-time movement of the distal end of the endoscopic device through the passageways such as that of the lungs, [0039], [0078], [0081], [0090]).
Regarding claim 3, Duindam discloses providing, by the computing system, a user interface for display of one or more of: the first tomogram, the first relative location or the second relative location of the distal end in the first tomogram, a first location of the target in the first tomogram, the second tomogram, the second relative location of the distal end in the second tomogram, and a second location of the target in the second tomogram (computing system, [0006], [0008]; control system processor(s), [0082]; display graphical user interface depicting preoperative and/or intraoperative images including the relative positions of the target and the distal end/tip of the medical instrument such as the vector between them, [0039], [0078], [0081], [0090], Figs. 3-4, 6-8, 11).
Regarding claim 4, Duindam discloses accessing the three-dimensional representative model further comprises retrieving the three-dimensional representative model identifying an organ within the subject, one or more cavities within the organ, and the target (three-dimensional preoperative model identifies organs such as the lungs, cavities/voids such as those of the lungs, and targets such as a tumor, [0037], [0047], [0062], [0065]-[0070], [0092]-[0103], [0111]).
Regarding claim 5, Duindam discloses acquiring the data further comprises acquiring, via the endoscopic device, the data further comprising image data acquired via the distal end of the endoscopic device (medical instrument is a broncho/endoscope that acquires image data via the distal end of the broncho/endoscope, [0033], [0035], [0049]).
Regarding claim 6, Duindam discloses receiving the second tomogram further comprises receiving, using the tomograph, the second tomogram in at least one of a two-dimensional space or a three-dimensional space, the second tomogram in an imaging modality different from an imaging modality of the first tomogram (intraoperative image dataset can be acquired in 2D or 3D space using an imaging modality different from the preoperative image dataset, [0037], [0042], [0057], [0062], [0075], [0083]).
Regarding claim 7, Duindam discloses registering the second tomogram with the thre-dimensional representative model further comprises determining a second displacement between the distal end of the endoscopic device and the target within the subject (registering the reference frames between the three-dimensional preoperative model and the intraoperative image dataset determines an updated real-time location of the target/tumor relative to other anatomical structures and/or the instrument including a shift in location/distance, [0111]).
Regarding claim 8, Duindam discloses determining, by the computing system (computing system, [0006], [0008]; control system processor(s), [0082]), a correspondence between the target in the three-dimensional representative model and the target in the second tomogram (registering the reference frames between the three-dimensional preoperative model and the intraoperative image dataset determines an updated real-time location of the target/tumor relative to other anatomical structures and/or the instrument including a shift in location/distance, [0105], [0111], [0115]).
Regarding claim 9, Duindam discloses registering the second tomogram with the first tomogram further comprises determining a difference in size between the target in the three-dimensional representative model and the target in the second tomogram within the subject (registering the reference frames between the three-dimensional preoperative model and the intraoperative image dataset determines an updated real-time size of the target/tumor, [0111]).
Regarding claim 10, Duindam discloses the invasive procedure further comprises a bronchoscopy, the distal end of the endoscopic device is inserted through a tract in a lung of the subject, and the volume of the subject scanned at least partially includes the lung (the medical instrument is a bronchoscope/endoscope inserted into the tract of a subject’s lung, [0052], [0060], [0069], [0070], [0084], [0090], [0095], Figs. 3-4, 6-8, 11).
Claims 11-20 are rejected under 35 U.S.C. 103 as being unpatentable over Duindam in further view of Krimsky.
Regarding claim 11, Duindam discloses a system for intraoperative medical imaging (system using registered real-time images, i.e., intraoperative images, and prior-time, i.e., preoperative images, of patient anatomy during an image-guided procedure that uses a medical instrument, [0002], [0006]-[0008], [0029]), comprising:
a computing system having one or more processors coupled with memory (control system including at least one memory and at least one computer processor, [0040], [0118]), configured to:
access, from a database, a three-dimensional representative model generated using a first tomogram corresponding to a scan of a volume within a subject prior to an invasive procedure (retrieve recorded preoperative image dataset from an imaging system, [0037], [0042], [0043], [0083]; create a three-dimensional pre-operative model using a three-dimensional preoperative image data set, [0037], [0062], [0065]-[0070], [0083]-[0084], [0084]-[0091], [0111]; image-guided procedure using a medical instrument, Abstract; see also medical instrument system being inserted into the body, [0052], Figs. 3-4, 6-8, 11);
identify, using a computer vision algorithm on the three-dimensional representative model generated using the first tomogram identifying a target within the volume of the subject (create a three-dimensional model from the three-dimensional preoperative image data set and use segmentation, filtering, and image characteristics analysis to identify targets of regions of interest such as a tumor within the three-dimensional pre-operative model, [0037], [0062], [0065]-[0070], [0083]-[0084], [0084]-[0091], [0111]);
acquire, via an endoscopic device at least partially disposed within a tract of the subject at a time instance during the invasive procedure (medical instrument is a broncho/endoscope for insertion into passageways such as that of the lungs, [0035], [0045], [0050], [0052], [0059], [0060], [0076], Figs. 3-4, 6-8, 11), data comprising operational data identifying a movement of a distal end of the endoscopic device through the tract (medical instrument is a broncho/endoscope that acquires shape/position/location data through passageways such as that of the lungs, [0033], [0035], [0043], [0047]-[0048], [0071], [0094]);
determine, in the first tomogram the subject, a first relative location of the distal end of the endoscopic device to the target based on the data acquired via the endoscopic device (determine vector between the distal tip/end of the medical instrument and the target in the three-dimensional pre-operative model and display the vector, target, and distal tip/end based on the shape/position/location data/information obtained from the sensor systems of the medical instrument, [0070], [0078], [0086]-[0091], Figs. 6A and 8; see also use segmentation, filtering, and image characteristics analysis to identify the instrument distal end and targets of regions of interest such as a tumor within the three-dimensional pre-operative model, [0037], [0062], [0065]-[0070], [0083]-[0084], [0084]-[0091], [0111]);
receive, using a tomograph, a second tomogram of the volume within the subject at the time instance during the invasive procedure, the second tomogram including the distal end of the endoscopic device and the target within the volume of the subject (acquire three-dimensional intraoperative/concurrent/real-time anatomic images during insertion of the medical instrument into the body including the distal end of the medical instrument and the target within the body, Abstract, [0035], [0037], [0038], [0039], [0042], [0057], [0062], [0065]-[0070], [0071]-[0075], [0092]-[0103], [0111]);
identify, using the computer vision algorithm on the second tomogram, (i) a second ROI corresponding to the target (use segmentation, filtering, and image characteristics analysis to identify targets of regions of interest such as a tumor within the three-dimensional intraoperative image data set of the body, [0037], [0057], [0062], [0065]-[0070], [0092]-[0103], [0111]) and (ii) the distal end of the endoscope device within the volume of the subject at the time instance (use segmentation, filtering, and image characteristics analysis to identify the distal end of the instrument within the three-dimensional intraoperative image data set of the body, [0037], [0057], [0062], [0065]-[0070], [0092]-[0103], [0111]);
register the second tomogram received from the tomograph during the invasive procedure with the first tomogram obtained prior to the invasive procedure based on the a first feature identified in the first tomogram and a second feature identified in the second tomogram (register the preoperative and intraoperative image datasets based on the features detected in the preoperative and intraoperative image datasets, [0105]; repeatedly registering and determining a shift in the target between a previously acquired tomogram image dataset, such as the preoperative model, and a subsequently acquired tomogram image dataset, such as the intraoperative images, [0115]-[0116]);
determine, from registering the second tomogram with the three-dimensional representative model (register the pre-operative model and the intraoperative image dataset based on one or more features detected in the three-dimensional pre-operative model and the intraoperative image dataset, [0105]; registering the three-dimensional pre-operative model and the intraoperative image dataset allows for determination of the shift in location, change in size, or change in shape of the tumor/target and of the shift in location/position or orientation of the distal tip of the instrument model/medical instrument between the preoperative model and the intraoperative image data [0111]-[0114]; repeatedly registering and determining a shift in the target and instrument location, i.e., the distal tip/end effector, between a previously acquired tomogram image dataset, such as the preoperative model, and a subsequently acquired tomogram image dataset, such as the intraoperative images, [0115]-[0116]):
(i) a first set of transformation parameters defining differences in visual representations of distal end in the three-dimensional representative model and the second tomogram (registering the three-dimensional pre-operative model and the intraoperative image dataset allows for determination of the shift in location/position or orientation of the distal tip of the instrument model/medical instrument between the three-dimensional pre-operative model and the intraoperative image data, [0111]-[0114]; repeatedly registering and determining a shift in the instrument location, i.e., the distal tip/end effector, between a previously acquired tomogram image dataset, such as the three-dimensional pre-operative model, and a subsequently acquired tomogram image dataset, such as the intraoperative images, [0115]-[0116]);
(ii) a second set of transformation parameters defining differences in visual representations between the first ROI in the three-dimensional representative model and the second ROI in the second tomogram (registering the three-dimensional pre-operative model and the intraoperative image dataset allows for determination of the shift in location, change in size, or change in shape of the tumor/target between the three-dimensional pre-operative model and the intraoperative image data, [0111]-[0114]; repeatedly registering and determining a shift in the target, between a previously acquired tomogram image dataset, such as the three-dimensional model, and a subsequently acquired tomogram image dataset, such as the intraoperative images, [0115]-[0116]);
(iii) a first deviation, based on the first set of transformation parameters for the distal end, between the distal end of the endoscopic device in the three-dimensional representative model and the distal end of the endoscopic device in the second tomogram (registering the three-dimensional pre-operative model and the intraoperative image dataset allows for determination of the shift in location/position or orientation of the distal tip of the instrument model/medical instrument between the three-dimensional pre-operative model and the intraoperative image data, [0111]-[0114]; repeatedly registering and determining a shift in the instrument location, i.e., the distal tip/end effector, between a previously acquired tomogram image dataset, such as the three-dimensional pre-operative model, and a subsequently acquired tomogram image dataset, such as the intraoperative images, [0115]-[0116]); and
(iv) a second deviation, based on the second set of transformation parameters for the target, between the first ROI in the three-dimensional representative model and the second ROI in the second tomogram (registering the preoperative model and the intraoperative image dataset allows for determination of the shift in location, change in size, or change in shape of the tumor/target between the preoperative model and the intraoperative image data, [0111]-[0114]; repeatedly registering and determining a shift in the target between a previously acquired tomogram image dataset, such as the preoperative model, and a subsequently acquired tomogram image dataset, such as the intraoperative images, [0115]-[0116]);
determine, based on registering the second tomogram with the three-dimensional representative model, a second relative location of the distal end of the endoscopic device and the target within the subject (register the three-dimensional pre-operative model and the intraoperative image dataset based on one or more features detected in the three-dimensional pre-operative model and the intraoperative image dataset, [0105]; registering the three-dimensional pre-operative model and the intraoperative image dataset allows for determination of an updated location of the tumor/target relative to the distal tip of the instrument model/medical instrument within the patient including the relative distance, position, orientation, and trajectory between the tumor/target and the distal tip of the instrument model/medical instrument, [0111]-[0114]; see also repeatedly registering and determining a shift in the target and instrument location, i.e., the distal tip/end effector, between a previously acquired tomogram image dataset, such as the three-dimensional pre-operative model, and a subsequently acquired tomogram image dataset, such as the intraoperative image dataset, [0115]-[0116]); and
provide, for display, a user interface comprising the indicator corresponding to the third relative location of the distal end to the target within the tract of the subject (registering the three-dimensional pre-operative model and the intraoperative image dataset allows for determination of an updated location of the tumor/target relative to the distal tip of the instrument model/medical instrument within the patient including the relative distance, position, orientation, and trajectory between the tumor/target and the distal tip of the instrument model/medical instrument, wherein the updated relative distance, position, orientation, and trajectory between the tumor/target and the distal tip of the instrument model/medical instrument is displayed as the medical instrument is navigated through passageways such as that of the patient’s lungs, [0111]-[0114], Figs. 8 and 11).
However, while Duindam discloses the first tomogram identifying a target within the volume of the subject, the second tomogram including the target within the volume of the subject, and registering, by the computing system, the second tomogram received from the tomograph during the invasive procedure with the first tomogram obtain prior to the invasive procedure based on a first feature identified in the first tomogram and a second feature identified in the second tomogram as detailed above, Duindam does not appear to explictly disclose the registering is based on the first ROI identified in the first tomogram and the second ROI identified in the second tomogram.
However, in the same field of endeavor of image guided navigation of endoscopes, Krimsky teaches a system for intraoperative medical imaging (a computer system implementing a method of navigating a tool inside a luminal network in the patient’s lungs using intra-procedural computed tomography image data, [0014]; method and computing device may be implemented via an electromagnetic navigation (EMN) system, [0031]-[0032], [0037]), comprising:
a computing system having one or more processors coupled with memory (computing device including a memory and a processor, [0068]-[0069]), configured to:
access, from a database, a three-dimensional representative model generated using a first tomogram corresponding to a scan of a volume within a subject prior to an invasive procedure (computing device receives previously acquired pre-procedural computed tomography scan image dataset such as from a DICOM standard storage, [0048]; generate a three-dimensional pre-operative model from the preoperative image data, [0049]-[0052]; see also storage of image data in memory or in a database, [0059], [0068]);
identify, using a computer vision algorithm on the three-dimensional representative model generated using the first tomogram identifying a target within the volume of the subject (generate a three-dimensional pre-operative model from the preoperative image data and use automatic image processing and analysis to identify anatomical structures and target locations in the three-dimensional pre-operative model, [0049]-[0052]);
acquire, via an endoscopic device at least partially disposed within a tract of the subject at a time instance during the invasive procedure, data comprising operational data identifying a movement of a distal end of the endoscopic device through the tract (determine tracked location of the EM sensor on the distal end of the locatable guide (LG) intraoperatively as the locatable guide (LG) is moved through the lungs/airway, [0006], [0033], [0035]-[0036], [0039]-[0040], [0054], [0058], [0060], [0063], [0065]);
determine, in the first tomogram of the subject, a first relative location of the distal end of the endoscopic device to the target based on the data acquired via the endoscopic device (determine tracked location of the EM sensor on the distal end of the locatable guide (LG) relative to the position of the target location in the pre-procedural image dataset/3D model, [0051]-[0054], [0058], Fig. 1);
receive, using a tomograph, a second tomogram of the volume within the subject at the time instance during the invasive procedure (application receives an additional computed tomography scan image data set of the patient’s body such as of the airways intra-procedurally, [0055]-[0059]; computing device for executing functions of the application, [0037], [0068]-[0069]), the second tomogram including the distal end of the endoscopic device and the target within the volume of the subject (use automatic image processing and analysis in the intraoperative image data to produce intra-procedural image dataset includes the position of the EM sensor on the distal end of the locatable guide (LG) and the position of the target within the patient’s body such as the airways, [0055]-[0067]);
identify, using the computer vision algorithm on the second tomogram, (i) a second ROI corresponding to the target and (ii) the distal end of the endoscope device within the volume of the subject at the time instance (use automatic image processing and analysis in the intraoperative image data to produce intra-procedural image dataset includes the position of the EM sensor on the distal end of the locatable guide (LG) and the position of the target within the patient’s body such as the airways, [0055]-[0067]);
register the second tomogram received from the tomograph during the invasive procedure with the first tomogram obtained prior to the invasive procedure based on the first ROI identified in the first tomogram and the second ROI identified in the second tomogram (registration of additional intra-procedural image data to the pre-procedural image data/3D model based on the target location identified in each of the pre-procedural image data/3D model and the additional intra-procedural image data, [0058]-[0059]);
determine, from registering the second tomogram with the three-dimensional representative model (registration of additional intra-procedural image data to the pre-procedural image data/3D model based on the target location identified in each of the pre-procedural image data/3D model and the additional intra-procedural image data, [0058]-[0059]; registering the intra-procedural image data to the pre-procedural image data/3D model allows for determination of the difference in position/movement of the target location and the difference in position/movement of the EM sensor on the distal end of the locatable guide (LG) between the preoperative model and the intraoperative image data, [0059]-[0061]):
(i) a first set of transformation parameters defining differences in visual representations of distal end in the three-dimensional representative model and the second tomogram (registering the intra-procedural image data to the pre-procedural image data/3D model allows for determination of the difference in position/movement of the EM sensor on the distal end of the locatable guide (LG) between the three-dimensional preoperative model and the intraoperative image data, [0059]-[0061]);
(ii) a second set of transformation parameters defining differences in visual representations between the first ROI in the three-dimensional representative model and the second ROI in the second tomogram (registering the intra-procedural image data to the pre-procedural image data/3D model allows for determination of the difference in position/movement of the target location between the three-dimensional preoperative model and the intraoperative image data, [0059]-[0061]);
(iii) a first deviation, based on the first set of transformation parameters for the distal end, between the distal end of the endoscopic device in the three-dimensional representative model and the distal end of the endoscopic device in the second tomogram (registering the intra-procedural image data to the pre-procedural image data/3D model allows for determination of the difference in position/movement of the EM sensor on the distal end of the locatable guide (LG) between the three-dimensional preoperative model and the intraoperative image data, [0059]-[0061]); and
(iv) a second deviation, based on the second set of transformation parameters for the target, between the first ROI in the three-dimensional representative model and the second ROI in the second tomogram (registering the intra-procedural image data to the pre-procedural image data/3D model allows for determination of the difference in position/movement of the target location between the three-dimensional preoperative model and the intraoperative image data, [0059]-[0061]);
determine, based on registering the second tomogram with the three-dimensional representative model, a second relative location of the distal end of the endoscopic device to the target within the subject (registration of additional intra-procedural image data to the pre-procedural image data/3D model based on the target location identified in each of the pre-procedural image data/3D model and the additional intra-procedural image data, [0058]-[0059]; registering the additional intra-procedural image data to the pre-procedural image data/3D model allows for determination of an updated tracked location of the EM sensor on the distal end of the locatable guide (LG) relative to the position of the target within the patient’s body such as within the patient’s airways including the relative position and pathway between the tracked location of the EM sensor on the distal end of the locatable guide (LG) and the target location, [0058], [0063]); and
provide, for display, a user interface to visualize the movement of the distal end of the endoscopic device through the tract of the subject (computing device includes a display device, [0014], [0016]-[0017], [0020], [0041], [0068]; display provides a user interface, [0068]; display/user interface displays the real-time movement of the location of the EM sensor on the distal end of the locatable guide (LG), [0006], [0008], [0012], [0014], [0016]-[0017], [0020], [0029], [0040], [0054], [0057], [0061]-[0062], [0065]), the user interface comprising an indicator corresponding to the second relative location of the distal end to the target within the tract of the subject during the invasive procedure (registration of additional intra-procedural image data to the pre-procedural image data/3D model based on the target location identified in each of the pre-procedural image data/3D model and the additional intra-procedural image data, [0058]-[0059]; registering the additional intra-procedural image data to the pre-procedural image data/3D model allows for determination of an updated tracked location of the EM sensor on the distal end of the locatable guide (LG) relative to the position of the target within the patient’s body such as within the patient’s airways including the relative position and pathway between the tracked location of the EM sensor on the distal end of the locatable guide (LG) and the target location, wherein the updated relative position and pathway between the tracked location of the EM sensor on the distal end of the locatable guide (LG) and the target location is displayed using the display/user interface as the medical instrument is navigated through the patient’s body such as through the patient’s airways, [0011]-[0012], [0019], [0058], [0063]).
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to have applied Krimsky’s known technique of performing registration using the target position in the pre-procedural and intra-procedural images to Duindam’s known apparatus for performing registration using features in the pre-procedural and intra-procedural images to achieve the predictable result that intra-procedural movement such as that of the respiratory cycle may be accounted and/or compensated for in updating the location of the LG/EWC/EM sensor in relation to the target location. See, e.g., Krimsky, [0055], [0067].
Regarding claim 12, Duindam discloses the computing system (computing system, [0006], [0008]; control system processor(s), [0082]) is further configured to:
receive, using the tomograph, a third tomogram of the volume within the subject at a second time instance during the invasive procedure after the time instance, the third tomogram including the distal end of the endoscopic device moved subsequent to provision of the second relative location (acquire second set of intraoperative anatomic images at a second time during insertion of the medical instrument into the body including any shift in the anatomy, target, and instrument location, [0115]-[0116]);
register the third tomogram received from the tomograph at the second time instance with the first tomogram received prior to the invasive procedure (register the preoperative, first intraoperative, and second intraoperative image datasets to one another to determine any shift in the anatomy, target, and instrument location, [0115]-[0116]);
determine a third relative location of the distal end of the endoscopic device and the target within the subject (register the preoperative, first intraoperative, and second intraoperative image datasets to one another to determine any shift in the anatomy, target, and instrument location, [0115]-[0116]; see also capturing a plurality of intraoperative images as a continuous/real-time process during the medical instrument navigation to continuously update the shifted relative position between the distal tip/end of the medical instrument and the target in the image model using the second set of intraoperative images and display the vector, target, and distal tip/end, [0115]-[0116]; see also repeated real-time intraoperative images taken for real-time updates of the displayed trajectory vector, target, and distal tip/end, [0002], [0078], [0090], [0092], [0111]- [0114]); and
provide, for display, the user interface (computing system, [0006], [0008]; control system processor(s), [0082]; display, [0034]-[0040]; user interface visualizes the real-time movement of the distal end of the endoscopic device through the passageways such as that of the lungs, [0039], [0078], [0081], [0090]) comprising the indicator corresponding to the third relative location of the distal end to the target within the tract of the subject (update the shifted relative position between the distal tip/end of the medical instrument and the target in the image model using the second set of intraoperative images and display the vector, target, and distal tip/end, [0115]-[0116]; see also repeated real-time intraoperative images taken for real-time updates of the displayed trajectory vector, target, and distal tip/end, [0002], [0078], [0090], [0092], [0111]- [0114]; user interface visualizes the real-time movement of the distal end of the endoscopic device through the passageways such as that of the lungs, [0039], [0078], [0081], [0090]).
Regarding claim 13, Duindam discloses the computing system is further configured to provide a user interface for display of one or more of: the first tomogram, the first relative location or the second relative location of the distal end in the first tomogram, a first location of the target in the first tomogram, the second tomogram, the second relative location of the distal end in the second tomogram, and a second location of the target in the second tomogram (computing system, [0006], [0008]; control system processor(s), [0082]; display graphical user interface depicting preoperative and/or intraoperative images including the relative positions of the target and the distal end/tip of the medical instrument such as the vector between them, [0039], [0078], [0081], [0090], Figs. 3-4, 6-8, 11).
Regarding claim 14, Duindam discloses the computing system is further configured to identify a three-dimensional representative model derived from scanning the volume within the subject prior to the invasive procedure, the three-dimensional representative model identifying an organ within the subject, one or more cavities within the organ, and the target (computing system, [0006], [0008]; control system processor(s), [0082]; preoperative image dataset forms a 3D model of the subject that identifies the lung, airways, and target, [0037]-[0038], [0084]-[0091]).
Regarding claim 15, Duindam discloses the computing system is further configured to acquire, via the endoscopic device, the data further comprising image data acquired via the distal end of the endoscopic device (computing system, [0006], [0008]; control system processor(s), [0082]; medical instrument is a broncho/endoscope that acquires image data via the distal end of the broncho/endoscope, [0033], [0035], [0049]).
Regarding claim 16, Duindam discloses the computing system is further configured to receive, using the tomograph, the second tomogram in at least one of a two-dimensional space or a three-dimensional space, the second tomogram in an imaging modality different from an imaging modality of the first tomogram (computing system, [0006], [0008]; control system processor(s), [0082]; intraoperative image dataset can be acquired in 2D or 3D space using an imaging modality different from the preoperative image dataset, [0037], [0042], [0057], [0062], [0075], [0083]).
Regarding claim 17, Duindam discloses the computing system is further configured to register
the second tomogram with the three-dimensional representative model to determine a displacement between the distal end of the endoscopic device and the target within the subject (computing system, [0006], [0008]; control system processor(s), [0082]; registering the reference frames between the three-dimensional preoperative model and the intraoperative image dataset determines an updated real-time location of the target/tumor relative to other anatomical structures and/or the instrument including a shift in location/distance, [0111]).
Regarding claim 18, Duindam discloses the computing system is further configured to determine a correspondence between the target in the three-dimensional representative model and the target in the second tomogram (computing system, [0006], [0008]; control system processor(s), [0082]; registering the reference frames between the three-dimensional preoperative model and the intraoperative image dataset determines an updated real-time location of the target/tumor relative to other anatomical structures and/or the instrument including a shift in location/distance, [0105], [0111], [0115]).
Regarding claim 19, Duindam discloses the computing system is further configured to register the second tomogram with the three-dimensional representative model to determine a difference in size between the target in the three-dimensional representative model and the target in the second tomogram within the subject (computing system, [0006], [0008]; control system processor(s), [0082]; registering the reference frames between the three-dimensional preoperative model and the intraoperative image dataset determines an updated real-time size of the target/tumor, [0111]).
Regarding claim 20, Duindam discloses the invasive procedure further comprises a bronchoscopy, the distal end of the endoscopic device is inserted through a tract in a lung of the subject, and the volume of the subject scanned at least partially includes the lungs (the medical instrument is a bronchoscope/endoscope inserted into the tract of a subject’s lung, [0052], [0060], [0069], [0070], [0084], [0090], [0095], Figs. 3-4, 6-8, 11).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Kopel et al. (U.S. Pub. No. 2020/0245982), Chopra et al. (U.S. Pub. No. 2012/0289777), Panescu et al. (U.S. Pub. No. 2014/0343416), Duindam et al. (U.S. Pub. No. 2018/0153621), and Wang et al. (U.S. Pub. No. 2020/0030044) disclose image guided interventions using a bronchoscope and a preoperative and one or more intraoperative (continuous) tomography imaging modality datasets for visualization of targets and the distal tip/end of the bronchoscope as it is navigated through the airways of a subject’s lungs.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Johnathan Maynard whose telephone number is (571)272-7977. The examiner can normally be reached 10 AM - 6 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Keith Raymond can be reached at 571-270-1790. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.M./Examiner, Art Unit 3798
/KEITH M RAYMOND/Supervisory Patent Examiner, Art Unit 3798
Prosecution Insights