DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant's submission filed on 04/08/2026 has been entered. Accordingly, claims 32-51 remain pending, and claims 35, 39 and 45, 49 have been amended.
Response to Arguments
Rejections under 35 USC 112
In light of applicant’s amendments to the claims filed 04/08/2026, the previous rejections under 112(b) have been withdrawn.
Rejections under 35 USC 103
Applicant's arguments filed 04/08/2026 have been fully considered but they are not persuasive.
Applicant argues in the second paragraph of page 9 through the first paragraph of page 10
“First, Moll does not teach or suggest ‘determine an instrument position error relative to the imaging device,’ as recited by claim 32. The Office Action on page 6 acknowledges that ‘the determined instrument tip position error disclosed by Zhao does not explicitly disclosed as being determined relative to the imaging device,’ and instead relies on Moll, paragraph [0277] and FIG. 151, as allegedly teaching this feature. However, the error in Moll, is not determined ‘relative to the imaging device.’ Rather, Moll's error is a comparison between a localization-measured tip position and a kinematics-predicted tip position. Moll's paragraph [0277] describes a closed-loop control system in which an instrument tip is driven through a cyclic range of motion, and the
error detected between a localization-measured tip position and a kinematics-predicted tip position serves as "an indicator of tissue contact."…
No reasonable reading of Moll, paragraph [0277] teaches or suggests ‘determine an instrument position error relative to the imaging device,’ as recited by claim 32.
Second, the cited portions of Zhao do not teach or suggest ‘in response to determining that at least a portion of the instrument is outside the field of view of the imaging device and based on a mode of operation of the teleoperational system, cause an out-of-view indication for the instrument to be presented,’ as recited by claim 32. Zhao's paragraph [0090] discusses ‘adaptively fus[ing] visual information and robot kinematics in order to achieve robust, accurate and efficient tool tracking,’ Zhao's paragraph [0166] states that ‘the robotic kinematics information may indicate that the one or both of the robotic surgical tools are outside the field of view,’ and Zhao's paragraph [0216] generally states that ‘[if] the robotic surgical tool is out of the field of view, then one of a plurality of compass icons 420 may be displayed in the field of view to show a direction of tool reentrance.’ None of cited paragraphs [0090], [0166], and [0216] teach or suggest that the presentation of the out-of-view indication is ‘based on a mode of operation of the teleoperational system,’ as recited by claim 32. In that regard, cited paragraph [0216] of Zhao fails to teach that the compass icons 420 are presented ‘based on a mode of operation of the teleoperational system.’”
In response, it is noted that the error of the tip position disclosed in [0277] of secondary reference Moll is not solely reliant upon the localization system, the predicted tip location, and determined via the inverse kinematics of the instrument structure, the cited disclosure dependent upon the experimentally determined systematic error between the predicted and measured tip locations for a given particular instrument structure thresholds may be utilized were error is considered an indication of tissue contact which may be detected by observing directly the directionality of error between the predicted and the measured instrument position. Direct observation is made via the imaging device, which may be located at the distal tip of the device shown in FIGS. 151, with the distal tip having the imaging device as illustrated in FIGS. 162. Further, secondary reference Moll additionally discloses in [0304], but not limited thereto, that a rough model is acquired utilizing conventional imaging modalities such as ultrasound or fluoroscopy as a starting point for the tissue structure model acquisition processes described above. Therefore, secondary reference Moll reads on the limitations as presently claimed.
In response to applicant’s second argument, it is noted that applicant’s limitation “based on a mode of operation of the teleoperational system” has been interpreted under the broadest reasonable interpretation to read on any mode for performing any operation that is performed by the teleoperational system. Therefore, the cited disclosure of primary reference Zhao which outlines multiple modes of multiple operations of the teleoperational system as cited in [0090], [0166], [0216] reads on the limitation as presently claimed. Further, applicant is additionally directed to the related improves the cited disclosure of primary reference Zhao also found in [0047]-[0048].
Therefore the claims remain rejected and the rejection has been made final.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to co click form Wizard click form Wizard click add page move mouse to display number three insider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 32-34, 38-39, 41, 42-44, 48-49, and 51 are rejected under 35 U.S.C. 103 as being unpatentable over Zhao et al. (US20090088634, hereafter “Zhao”), in view of Moll et al. (US20070197896, hereafter “Moll”).
Regarding claims 32, 42, and 51, Zhao discloses a method for controlling a teleoperational (FIG. 6A, claim 13) and a teleoperational system (FIG. 1, claim 1), a non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a processing unit of a teleoperational system ([0049] computer having a hard drive stores the instructions to be executed), comprising:
a teleoperational assembly configured to support an instrument (endoscope FIG. 2) and an imaging device ([0052] stereo viewer, three dimensional maps (a depth map with respect to a camera coordinate system derived from alternative imaging modalities (e.g. CT scan, XRAY, or MRI), the instrument having an instrument tip (endoscope in FIG. 2); and
a processing unit including one or more processors ([0049] the computer having one or more microprocessors), wherein the processing unit is configured to:
determine an instrument position of the instrument ([0204] determine an estimation error (EET) between the actual tip location 1520 and the estimated tip location 1518 using tool tracking methods);
determine an instrument position error ([0061], FIG. 5A, tool tracking is determined by determinig a pose of a robotic instrument 101 including its position or location (Xt,Yt,Zt) and its orientation in the camera coordinate system as it moves in, around, and out of the surgical site, a full pose description may not only include the location and orientation of a robotic instrument in a three-dimensional space but may further include the pose of an end-effector, if any [a lack of indicator of a pose of an end-effector corresponds to the device with an end-effector being outside of the field of view], then positional information or pose as used herein may be used to refer to one or both the location and orientation [being within or outside of the field of view] of a robotic instrument); and
determine, based on the instrument position and the instrument position error, that at least a portion of the instrument is outside a field of view of the imaging device ([0204], FIGS. 5A, 9A-9B, image sequences 1514 along with kinematics information may be adaptively fused together to translate a model of the tool tip (MET) into the endoscopic camera frame of reference (ECM)…The operational error EOP between the actual location of the tumor and the actual location of tool tips [position error] is a sum of errors); and
in response to determining that at least a portion of the instrument is outside the field of view of the imaging device and based on a mode of operation of the teleoperational system ([0090], [0166], FIG. 8, tool tracking system adaptively fuses visual information [based on a determination] and robot kinematics in order to achieve robust, accurate and efficient tool tracking, the unknown full pose of a robotic instrument, at a time instant t, is represented as a state in a Bayesian state-space model, the state-space model uses a plurality of posed states to perform tool tracking in the surgical site. The state-space model generates the corrected kinematics information of the robotic instrument, and the robotic kinematics information [including bounding volume of the instrument tip] indicates that the one or both of the robotic surgical tools are outside the field of view), cause an out-of-view indication for the instrument to be presented (see FIG. 5A for determining the tip pose is outside the field of view, [0216] when the robotic surgical tool is out of the field of view, then one of a plurality of compass icons may be displayed [indicate when the instrument tip is outside of the field of view] in the field of view to show a direction of tool reentrance), but the determined instrument tip position error disclosed by Zhao does not explicitly disclosed as being determined relative to the imaging device.
However, in the same field of endeavor, Moll teaches determining an instrument tip position error relative to the imaging device ([0277], FIG. 151, the instrument operated by an instrument driver and a closed-loop control system incorporating a localization technology to measure actual instrument position is depicted so that when the instrument tip is driven through a range of motion, such as +pitch to -pitch, then back to neutral and +yaw to -yaw, at some cyclic interval, loads encountered by tissue structure contact, as opposed to free cavity space in blood, for example, will tend to increase the error detected between the measured tip position determined by the localization system, and the predicted tip location, determined via the inverse kinematics of the instrument structure, and other cyclic patterns of motion may also be utilized, e.g., repeated spiral motion, circular motion, etc. A threshold may be utilized, depending upon the experimentally determined systematic error between the predicted and measured tip locations in free space given a particular instrument structure, the threshold is beyond which error is considered an indicator of tissue contact. And, depending upon the cyclic motion pattern selected, the direction of contact between the instrument and another object may also be detected by observing the directionality of error between predicted and measured instrument position).
It would have been obvious to one ordinarily skilled in the art before the effective filing date of the claimed invention to modify the system and method disclosed by Zhao with the step to determine an instrument tip position error relative to the imaging device as taught by Moll in order to provide a real-time representation of where the computerized system expects the instrument to be relative to the surrounding anatomy to facilitate the user's comprehension of relative positioning of the various structures ([0306] of Moll).
Regarding claims 33 and 43, Zhao substantially discloses all the limitations of the claimed invention, specifically, Zhao discloses wherein the processing unit is configured to cause the out-of-view indication for the instrument to be presented based on the mode of operation of the teleoperational system being in a camera control mode in which a user of the teleoperational system controls the imaging device ([0047]-[0048] the operator's head sets off the viewing sensor to enable the control of the robotic instruments when the system operator aligns his or her eyes with the binocular eye pieces of the display to view a stereoscopic image [part of imaging system/device] of the surgical worksite and the operator O can use their feet to control the foot-pedals to change the configuration of the surgical system and generate additional control signals to control the robotic instruments as well as the endoscopic camera [imaging device]).
Regarding claims 34 and 44, Zhao substantially discloses all the limitations of the claimed invention, specifically, Zhao discloses wherein the processing unit is configured to cause the out-of-view indication for the instrument to be presented based on the mode of operation of the teleoperational system being in a following mode* in which a user of the teleoperational system controls the instrument ([0047]-[0048] the operator's head sets off the viewing sensor to enable the control of the robotic instruments when the system operator aligns his or her eyes with the binocular eye pieces of the display to view a stereoscopic image [part of imaging system/device] of the surgical worksite and the operator O can use their feet to control the foot-pedals to change the configuration of the surgical system and generate additional control signals to control the robotic instruments).
*The term has been interpreted as defined by the applicant in [0048] of the PG Pub to mean “the out-of-view indicator may be displayed for a few seconds after initiating a mode of the system in which the operator controls movement of the instruments, a mode which may be known as a following mode”, where the term “instruments” has been interpreted under the broadest reasonable interpretation to mean any device used by an operator during a surgical operation as known in the surgical arts.
Regarding claims 38 and 48, Zhao, in view of Moll, substantially discloses all the limitations of the claimed invention, specifically, Zhao discloses wherein the processing unit is further configured to, in response to determining that at least a portion of the instrument is outside the field of view of the imaging device and based on a mode of operation of the teleoperational system ([0047]-[0048] the operator's head sets off the viewing sensor to enable the control of the robotic instruments when the system operator aligns his or her eyes with the binocular eye pieces of the display to view a stereoscopic image [part of imaging system/device] of the surgical worksite and the operator O can use their feet to control the foot-pedals to change the configuration of the surgical system and generate additional control signals to control the robotic instruments as well as the endoscopic camera [imaging device]), and specifically, Moll discloses disable a function of the instrument (FIG. 2.1, [0131], a device disabling switch (7) configured to temporarily disable the activity of the instrument).
Regarding claims 39 and 49, Zhao substantially discloses all the limitations of the claimed invention, specifically, Zhao discloses wherein the processing unit is further configured to:
receiving, from an input device, of the teleoperation system, a signal indicating a user acknowledgement of an out-of-view instrument status for the instrument ([0046]-[0048] the operator gripping the ouch sensitive handles of the control input wrists to generate control signals [a signal indicating a user acknowledgement] to change the configuration of the surgical system and generate additional control signals to control the robotic instruments 101 as well as the endoscopic camera when the operator's head is removed the area of the display 312 [an out-of-view instrument status]); and
in response to receiving the signal indicating the user acknowledgment of the out-of-view instrument status, enabling the function of the instrument ([0047] the operator, via the operator's head, sets off the viewing sensor 320 to enable the control of the robotic instruments 101).
Regarding claim 41, Zhao substantially discloses all the limitations of the claimed invention, specifically, Zhao discloses wherein the out-of-view indication for the instrument is based on an instrument type* of the instrument ([0195], [0211] one type of tracked tool (e.g., an ultrasound tool) is used to draw marks to indicate regions of interest and then use a different type of tracked tool (e.g., a cautery tool) to operate or perform a surgical procedure, prior knowledge used in the tracking information includes the knowledge of the type of instrument).
*The limitation has been interpreted under the broadest reasonable interpretation to mean any type of instrument known in the surgical arts.
Claim(s) 35-37, 40, 45-47 and 50 are rejected under 35 U.S.C. 103 as being unpatentable over Zhao, in view of Moll, as applied to claims 32, 34 and 42, 44 above, further in view of Bendall (US20170337705).
Regarding claims 35 and 45, Zhao substantially discloses all the limitations of the claimed invention, but does not explicitly disclose wherein the processing unit is further configured to: in response to the user initiating the following mode, cause the out-of-view indication to be presented; and after a predetermined time period after initiating the following mode, cause the presentation of the out-of-view indication to be ceased.
However, in solving the same problem, Bendall teaches wherein the processing unit is further configured to:
upon initiation of the following mode, cause the out-of-view indication to be presented ([0159] the graphic overlay may be updated in real time as the cursors are moved by the user); and
after a predetermined time period after initiation of the following mode, cause the presentation of the out-of-view indication to be ceased ([0160] the graphic overlay is only displayed temporarily when a cursor is moved and may be removed a few seconds [predetermined period] after cursor movement stops).
It would have been obvious to one ordinarily skilled in the art before the effective filing date of the claimed invention to modify the method and system disclosed by Zhao with the processing unit is further configured to: in response to the user initiating the following mode, cause the out-of-view indication to be presented; and after a predetermined time period after initiating the following mode, cause the presentation of the out-of-view indication to be ceased as taught by Bendall in order to provide a visual indication of the orientation of the tip of the probe of the video inspection device with respect to the object ([0166] of Bendall).
Regarding claims 36 and 46, Zhao substantially discloses all the limitations of the claimed invention, but does not explicitly disclose wherein the processing unit is further configured to: receive a user input to view information relating to the instrument's location; and cause the out-of-view indication for the instrument to be presented based further on receiving the user input.
However, in solving the same problem, Bendall wherein the processing unit is further configured to:
receive a user input to view information relating to the instrument's location ([0077] the video inspection device monitors the movement of the cursor 234 and detect when the cursor 234 has stopped moving, the video inspection device 100 (e.g., the CPU 150) can determine the deepest surface point proximate to the cursor 234); and
cause the out-of-view indication for the instrument to be presented based further on receiving the user input ([0160] the graphic overlay may be shown temporarily and is removed a few seconds after cursor movement stops).
It would have been obvious to one ordinarily skilled in the art before the effective filing date of the claimed invention to modify the method and system disclosed by Zhao with receive a user input to view information relating to the instrument's location; and cause the out-of-view indication for the instrument to be presented based further on receiving the user input as taught by Bendall in order to provide a visual indication of the orientation of the tip of the probe of the video inspection device with respect to the object ([0166] of Bendall).
Regarding claims 37 and 47, Zhao substantially discloses all the limitations of the claimed invention, but does not explicitly disclose wherein the processing unit is further configured to: receive a user input to disable the out-of-view indication; and cause the presentation of the out-of-view indication to be ceased in response to receiving the user input.
However, in solving the same problem, Bendall teaches wherein the processing unit is further configured to:
receive a user input to disable the out-of-view indication ([0160] the graphic overlay may be shown temporarily and is removed a few seconds after cursor movement stops as input by a user); and
cause the presentation of the out-of-view indication to be ceased in response to receiving the user input ([0160] the graphic overlay may be hidden whenever a reference surface cursor is inactive).
It would have been obvious to one ordinarily skilled in the art before the effective filing date of the claimed invention to modify the method and system disclosed by Zhao with the processing unit is further configured to: receive a user input to disable the out-of-view indication; and cause the presentation of the out-of-view indication to be ceased in response to receiving the user input as taught by Bendall in order to provide a visual indication of the orientation of the tip of the probe of the video inspection device with respect to the object ([0166] of Bendall).
Regarding claims 40 and 50, Zhao substantially discloses all the limitations of the claimed invention, but does not explicitly disclose wherein the processing unit is further configured to present, based on an instrument type of the instrument, a warning message or an acknowledgment relating to the at least the portion of the instrument being outside of the field of view of the imaging device.
However, in solving the same problem, Bendall teaches wherein the processing unit is further configured to present, based on an instrument type of the instrument, a warning message or an acknowledgment relating to the at least the portion of the instrument being outside of the field of view of the imaging device ([0175] the video inspection device outputs a warning system to a user via a visual or audible warning when there is an undesirable (e.g., far from perpendicular) viewing perspective at the location where a measurement cursor is being placed on an edge of the viewing frame).
It would have been obvious to one ordinarily skilled in the art before the effective filing date of the claimed invention to modify the method and system disclosed by Zhao with the processing unit is further configured to present, based on an instrument type of the instrument, a warning message or an acknowledgment relating to the at least the portion of the instrument being outside of the field of view of the imaging device the user input as taught by Bendall in order to provide a visual indication of the orientation of the tip of the probe of the video inspection device with respect to the object ([0166] of Bendall).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AMY SHAFQAT whose telephone number is (571)272-4054. The examiner can normally be reached Monday-Friday 9:30AM-5:30PM MST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Keith Raymond can be reached at (571) 270-1790. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/A.S./Examiner, Art Unit 3798
/KEITH RAYMOND/Supervisory Patent Examiner, Art Unit 3798
Prosecution Insights