Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The claim amendments filed 4/24/26 overcome the previous 112(b) rejections.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 3, 10-13, 16, 17 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Graetzel et al. [US 2021/0393338 A1, hereinafter “Graetzel”].
Re. claim 1, Graetzel discloses a system [Fig. 1] for displaying and controlling the progress of an interventional device configured for insertion into an anatomical structure of a subject [Par. 0005], the system comprising:
at least one processor [part of control system 150, Pars. 0072 and 0245] coupled to (i) a display [152] and (ii) a user interface [154 and 156, together] to provide control inputs for controlling movements [Par. 0067] of the interventional device [130],
the at least one processor configured to:
read a determinate coordinate system associated with the user interface [vertical/horizontal axes of image data within an interface, Par. 0085];
receive current image data of a current image of the interventional device in the anatomical structure displayed or to be displayed on a display, the current image showing a current position of the interventional device [Par. 0068];
receive a control input from the user interface for controlling a movement of the interventional device from the current position, the control input being representative of a control direction in the determinate coordinate system of the user interface [Par. 0078];
estimate from at least the current image data a movement direction of the interventional device in the current image on the display based on the control input [“the control system 150 can instruct the physician 160 to move the catheter 130 in a particular direction with respect to the interface(s) 154” Par. 0101, emphasis added; note that the interface appears on the display, Fig. 1];
estimate a mismatch between (i) the movement direction of the interventional device in the current image on the display and (ii) the control direction represented by the control input in the determinate coordinate system of the user interface [a “difference between the direction indicated by the physician 160 in which the catheter 130 moved (with respect to a control/coordinate frame) and the direction in which the control system 150 estimated that the catheter 130 moved (with respect to the control/coordinate frame),” and “control system…can instruct the physician…to move the catheter…in a particular direction (e.g., a right arrow…can be displayed via the interface(s) 154” Par. 0101. Note that the directions are shown with respect to the display.];
determine a change of orientation of the current image displayed or to be displayed or a change of orientation of the coordinate system of the user interface to align the movement direction of the interventional device in the current image on the display and the control direction represented by the control input in the determinate coordinate system of the user interface; and implement the change of orientation [“the control system 150 can update the control frame/scheme for the catheter,” Par. 0101, which changes the display orientation; see e.g. Pars. 0092-0093, which discuss the examples of direct and inverted control schemes].
Re. claim 3, Graetzel discloses determining a change of orientation of the coordinate system of the user interface comprises controlling the alteration of functionality of the user interface such that a control input corresponding to the control direction represented by the control input matches the movement direction of the interventional device displayed or to be displayed [Par. 0087].
Re. claim 10, Graetzel discloses the user interface for controlling movements of the interventional device based on control inputs, associated with said the stored determinate coordinate system, coupled to the at least one processor [Fig. 1, Par. 0067], and the display configured to display images of the interventional device in the anatomical structure of the subject, coupled to the at least one processor [Par. 0068].
Re. claim 11, Graetzel discloses the user interface comprises a control console [156] comprising an input device operable by a user for controlling movement of the interventional device [Fig. 1], the input device being optionally a joy stick or a thumb stick [this optional limitation need not be met, though Fig. 1 does show a thumb stick].
Re. claim 12, Graetzel discloses the at least one processor is further configured to: determine the control direction of the input device relative to the control console based on the control input [Pars. 0134-0136 disclose the two control directions being determined based on control input].
Re. claim 13, Graetzel discloses an imaging system [imaging device 180] configured to acquire the current image of the anatomical structure [Par. 0076], and a robot controller [control electronics in 114 of robotic system 110, Par. 0074] configured to enable control of the robot in accordance with the control input provided through the user interface [Par. 0074].
Re. claim 16, Graetzel discloses the processor configured to perform the steps as set forth with respect to claim 1 above. Thus, Graetzel also inherently discloses performing said steps as claimed in claim 16.
Re. claim 17, Graetzel discloses the processor configured to perform the steps as set forth with respect to claim 1 above. Because the processor must be provided with instructions to perform said steps, Graetzel inherently discloses the non-transitory computer readable medium storing instructions to cause the processor to perform the steps of claim 1, and thus teaches the limitations of claim 17.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Graetzel in view of Hunter et al. [US 2004/0097806 A1, hereinafter “Hunter.”].
Re. claim 4, the primary reference teaches the system as set forth with respect to claim 1 above, but fails to teach the estimating the movement direction comprises inferring a shape of the interventional device. However, Hunter teaches estimating a movement direction of an interventional device (catheter) comprises: inferring a shape of the interventional device and surrounding anatomy of the anatomical structure from the current image [“Visualization of the shape and position of a distal portion of the catheter,” Par. 0010] and a plurality of recent past images [“The image 178 further includes a spline or curved projection 182, which is based upon the shape of the curved catheter 52,” Par. 0080]; and estimating the movement direction of the interventional device based on the shape of the interventional device and the surrounding anatomy of the anatomical structure [Par. 0080]. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the system of Graetzel as taught by Hunter in order to enable estimated curved trajectories of the catheter to be displayed to assist the user [Hunter Par. 0010].
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over in view of Gormley et al. [US 2023/0372032 A1, hereinafter “Gormley”].
Re. claim 5, Graetzel teaches the system of claim 1 above but fails to teach estimating the movement direction in the manner claimed in claim 5. However, Gormley teaches estimating the movement direction of an interventional device comprises:
establishing a motion vector for the interventional device using a plurality of recent past images and corresponding control inputs, wherein the motion vector represents a direction and a magnitude of displacement of the interventional device moving through the anatomical structure shown in the plurality of recent past images [Par. 0054];
predicting future motion vectors in corresponding next images indicative of a future direction of the movement of the interventional device using a first neural network model [Par. 0054],
wherein a length of the future motion vector indicates a number of future frames of the next images are needed for the movement of the interventional device to be fully realized [see 112(b) above; Gormley, Pars. 0054-0056]; and
estimating the movement direction of the interventional device based on the predicted future motion vectors [Par. 0055].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the system of Graetzel by configuring the system such that estimating the movement direction comprises establishing a motion vector, predicting future motion vectors using a first neural network model, and estimating the movement direction, as taught by Gormley, because this allows for automatic driving of the instrument [Gormley, abstract].
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Graetzel in view of Gormley and further in view of Yeung et al. [US 2018/0296281 A1, hereinafter “Yeung”].
Re. claim 6, the modified Graetzel teaches training the first neural network, initially train the first neural network model using motion vectors and corresponding control inputs associated with a plurality of training images [Pars 0054-0056] but fails to teach recurrent convolutional layers or transformer architectures. However, Yeung teaches, in an automated steering system for an endoscope, at least one processor is further configured to: initially train the first neural network model using motion vectors and corresponding control inputs associated with a plurality of training images, wherein the first neural network model includes recurrent convolutional layers [Par. 0007, 0231]. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the system of the modified Graetzel such that the first neural network model includes recurrent convolutional layers as taught by Yeung because this allows image data to be used directly as input and “and the neural network is allowed to formulate the logical processing steps that provide optimal mapping of the input data to an output navigational direction and/or set of steering control instructions” [Yeung Par. 0231].
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Graetzel in view of Gormley and further in view of Sen et al. [US 2021/0290317 A1, hereinafter “Sen”].
Re. claim 7, the modified Graetzel teaches estimating the mismatch between the movement direction of the interventional device and the control direction of the input device of the control console but fails to teach training a second neural network model. However, Sen teaches, in a system for tracking a position of a surgical instrument, a processor configured to initially train a “second” neural network model for estimating the mismatch between a movement direction of the interventional device and a control direction using a current image, a plurality of recent past images and corresponding control inputs, and an estimated movement direction of the interventional device [Par. 0061]. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the system of Graetzel such that a second neural network is trained as taught by Sen because this allows for more precise, accurate, efficient, and reliable position tracking of the system [Sen, Par. 0003].
Claim(s) 8-9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Graetzel in view of Gormley and Sen, as applied to claim 7, and further in view of Yeung.
Re. claims 8 and 9, Graetzel fail to teach the neural networks, which are taught by secondary references. Yeung teaches, in a neural network model, that it may be supervised or unsupervised, or any combination thereof [Par. 0091]. Therefore, selecting each of the first neural network model and the second neural network model to be supervised as in claim 8, or unsupervised as in claim 9, would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, because this amounts to selecting one from a limited list of options re. the type of machine learning training used.
Claim(s) 14-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Graetzel in view of Yeung.
Re. claim 14. Graetzel fail to teach predicting future movement. However, Yeung teaches estimating the movement direction of the interventional device comprises predicting future movement [“predicted steering direction,” Par. 0084] of the interventional device. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the system of Graetzel by predicting future movement as taught by Yeung in order to allow the position of the device to be determined in real time [Yeung, Par. 0009]
Re. claim 15. Graetzel fail to teach estimating the movement direction based on movement in recent past images. However, Yeung teaches the movement direction of the interventional device is estimated based on movement of the interventional device in the plurality of recent past images [“a predicted position for the center of the lumen is calculated based on motion vectors derived from the center positions of the lumen in two or more images previously captured by the first or at least second image sensor,” Par. 0011]. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the system of Graetzel by configuring the movement direction of the interventional device is estimated based on movement of the interventional device in the plurality of recent past images as taught by Yeung in order to allow the movement direction to be accurately estimated based on historical data.
Response to Arguments
Applicant's arguments filed 4/24/26 have been fully considered but they are not persuasive. Applicant argues that Graetzel fails to teach the amended limitations, specifically the mismatch between the movement direction of the device in the current image on the display and the control direction represented by the control input. However, the examiner respectfully disagrees. While Graetzel does teach the orientation of one instrument relative to another, this is not all that is taught. See above-cited paragraphs and additionally, this is explained in Par. 0078: “The control system 150 can determine an orientation/position of the medical instrument…and/or an orientation of image data displayed through the interface(s) 154. The control system…can use such information to generate a control signal to move the medical instrument in the appropriate direction relative to a coordinate/control frame of the medical instrument,” and Par. 0080: “In the case of first-person driving, the control system…can generally control a medical instrument to move in a correlated manner to an orientation of image data from the medical instrument displayed via the interface(s) 154” and Par. 0081:
To illustrate, if the interface(s) 154 is displaying a static view for the scope 120 (e.g., up in the interface(s) 154 corresponds to the positive y-vector of the coordinate frame for the scope 120), the control system 150 can cause the scope 120 to move along the positive y-vector of the coordinate frame for the scope 120 in response to up input on the I/O device(s) 156. Further, if the interface(s) 154 is displaying a rotated image view for the scope 120 (e.g., up in the interface(s) 154 does not always correspond to the positive y-vector of the coordinate frame for the scope 120), the control system 150 can determine an offset of a frame of reference of the image data to the coordinate frame for the scope 120 to cause the scope 120 to move in the appropriate direction that appears as the scope 120 moving upward in the interface(s) 154.
Thus, the examiner finds that Graetzel discloses determining a mismatch between (i)the movement direction of the interventional device in the current image on the display and (ii) the control direction represented by the control input in the determinate coordinate system of the user interface as claimed.
Arguments against Masaki are moot as this reference is no longer applied in light of the amendments.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ERIN MCGRATH whose telephone number is (571)270-0674. The examiner can normally be reached M-Th 9 am to 3 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JACKIE HO can be reached at (571) 272-4696. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ERIN MCGRATH/Primary Examiner, Art Unit 3771
Prosecution Insights