Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Joint Inventors
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on December 3rd, 2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
Claims 8-10 and 14 are objected to because of the following informalities:
Claim 8 Line 2: “generating an help-alignment graphic element” should be revised to “generating the help-alignment graphic element”.
Claim 9 Line 2: “generating an help-alignment graphic element” should be revised to “generating the help-alignment graphic element”.
Claim 10 Line 2: “generating an help-alignment graphic element” should be revised to “generating the help-alignment graphic element”.
Claim 14 Line 2: “said signal for enabling” should be revised to “said enabling signal” to avoid lacking antecedent basis.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 15 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because it is directed to “a computer program” which falls under “products that do not have a physical or tangible form, such as […] a computer program per se”. MPEP § 2106.03.
This rejection can be overcome by revising “A computer program […]” to “A non-transitory computer readable medium comprising a program […]”.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-19 are rejected under 35 U.S.C. 103 as being obvious over DiMaio et al. (US Patent Pub. No. 2020/0360097 A1), herein “DiMaio”, in view of Suresh et al. (US Patent Pub. No. 2017/0333139 A1), herein “Suresh”.
Regarding Claims 1 and 15-16, DiMaio discloses a control method of a robotic system, a computer program installable in an internal memory of a microprocessor unit, and a robotic system for medical or surgical teleoperation, wherein said robotic system comprises at least one master device, which can be handled and is adapted to be moved by an operator, at least one slave device configured to be moved by an actuator and to be controlled by the master device, a central microprocessor unit configured to control said slave device, and a display (See Abstract, “[…] the teleoperated system is a telemedical system such as a telesurgical system.” See also 0010, “[…] teleoperated system comprises a display, a master input device, and a control system. The control system comprises one or more processors and a memory […] commanding (based on the alignment relationship) a change in a pose of the end effector in response to a change in a pose of the master input device.” See also 0128-0130, “[…] user 540 holds master device 536 […] A centralized or distributed computer control system 550 receives information about the orientations of the various teleoperated system components […]”),
said method comprises performing the following operations:
determining with said central unit a first spatial orientation of said master device and a second spatial orientation of said slave device (See 0010 as referenced above and, “[…] determining an orientation of an end-effector reference frame relative to a field-of-view reference frame, determining an orientation of an input-device reference frame relative to a display reference frame, establishing an alignment relationship […]” Examiner notes the end effector reference frame comprises the slave device, as it is commanded in response to the master input device);
when said central unit receives a help-alignment request signal, opening with said central unit a graphic user interface on said display, said graphic user interface comprising at least one help-alignment graphic element displayed in a portion of said display, visually representing information on a relative spatial orientation of the master device updated in real time with respect to the slave device and/or instructions to allow said operator to reach an alignment condition between the master device and the slave device (See 0010 as referenced above. See also 0137, “[…] control alignment is automatically established as soon as the master device orientation is within the alignment tolerance, and optionally a visual, audio, haptic, or similar indication is output to the user as a signal that the master/slave control relationship is in effect […] in addition to the master orientation being within the alignment tolerance, the system must receive another event, such as a button press, verbal command, or similar input that requests the control relationship be established.” See also 0051, “[…] master devices may optionally have one or more inputs such as switches to control additional end effector or surgical system features […] engaging and disengaging the master/slave control relationship between the master device and the slave tool (“clutching”), changing system operating modes, changing master device control from one slave surgical tool to a second slave surgical tool, display menu selection, and the like.” Examiner notes the display clearly includes a menu selection and shows visual representations of the reference frames for the master device and end effectors. The system further receives user input as a request for initiating alignment control);
closing, with said central unit, said graphic user interface when at least one of the following conditions is met:
said relative spatial orientation is within a predetermined interval (See 0141, “[…] predetermined fixed spatial relationship with respect to the other manipulators […] establish and maintain the control relationship necessary for teleoperation based only on sensing the angular relation between the various kinematic pairs […] spatial relationships between units are determined in order to establish and maintain effective intuitive control.” Examiner notes the system is able to determine relative spatial orientation within a predefined interval, and simply closing the interface once a requirement is met, such as receiving a signal or timing out, is a common and widely understood function for medical displays, or any displays);
an enabling signal for a medical or surgical teleoperation has been generated;
a maximum time has elapsed wherein said graphical user interface has remained open since said help-alignment request signal was generated.
But does not explicitly disclose updating said help-alignment graphic element shown on the display in a manner corresponding to the relative spatial orientation of the master device with respect to the slave device.
Suresh, in a similar field of endeavor, teaches updating said help-alignment graphic element shown on the display in a manner corresponding to the relative spatial orientation of the master device with respect to the slave device (See 0046, “[…] the interface controller 608 to use the first input signal to update a graphical user interface presented by the display device 606.” See also 0051-0053, “[…] communicatively coupled to the teleoperated surgical system 602 to transmit a second input signal to the interface controller […] the second input signal used by the interface controller 608 in conjunction with the first input signal to control the graphical user interface […] receives a rotational signal indicating that a portion of the first master controller 604 is manually rotated by an amount of rotation. In such an embodiment, updating the graphical user interface comprises scrolling a portion of the graphical user interface based on the amount of rotation.”).
In view of Suresh’s teachings, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include, with the system and method for aligning a master and slave device for teleoperation including displaying their spatial orientations as disclosed by DiMaio, updating a graphical user interface corresponding to changes to the relative spatial orientation of the master and slave device, with a reasonable expectation of success, since the combination would have predictably allowed orientation-based alignment through larger ranges of the operation, and would provide a more efficient human-machine interface without needing to change any underlying teleoperation architecture.
Regarding Claims 2 and 18, DiMiao further discloses the method according to claim 1 and robotic system according to claim 16, wherein the master device is of the unactuated type (See 0128, “Master device 536 is optionally mechanically grounded or ungrounded […]”).
Regarding Claim 3, DiMiao further discloses the method according to claim 1, wherein said display is configured to display at least in its central portion a surgical site in which the slave device operates,
the method including the following steps:
displaying said surgical site and said slave device in said central portion of the display (See 0049, “[…] imaging device such as an endoscopic camera at the surgical site captures a moving image of tissue and a slave surgical tool's working end.” See also 0060, “[…] image of the internal surgical site is displayed to user 102 by a display 108 in user's control unit 112. The internal surgical site is optionally simultaneously shown to assistant 128 by an auxiliary display 130 (2D or 3D). As mentioned above and described in more detail below, however, in some teleoperated systems user 102 may be close to patient 104 during surgery […]”).
But does not explicitly disclose displaying said help-alignment graphic element in a peripheral portion of said display, not superimposed on said central portion of the display, which shows said slave device.
Suresh, in a similar field of endeavor, teaches displaying said help-alignment graphic element in a peripheral portion of said display, not superimposed on said central portion of the display, which shows said slave device (See 0046 as referenced above and Fig. 4B shown below. Examiner notes Fig. 4B shows the graphical interface with graphic elements in the peripheral of the display).
PNG
media_image1.png
486
516
media_image1.png
Greyscale
In view of Suresh’s teachings, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include, with the system and method for aligning a master and slave device for teleoperation including displaying their spatial orientations as disclosed by DiMaio, displaying the graphic elements in a peripheral portion of the display as to not overlap with the central portion of the display, with a reasonable expectation of success, since the combination would have predictably allowed common user interaction with the graphical elements without interfering with other elements within the display, while also improving perceivability of the various devices shown.
Regarding Claims 4 and 19, DiMaio does not explicitly disclose the method according to claim 1 and robotic system according to claim 16, comprising the operation of displaying the at least one help-alignment graphic element on said display, avoiding viewing the surgical site; or wherein said robotic system includes at least a first display and a second display, the method comprising the operations of viewing the surgical site on the first display and displaying the at least one help-alignment graphic element on the second display.
Suresh, in a similar field of endeavor, teaches the operation of displaying the at least one help-alignment graphic element on said display, avoiding viewing the surgical site; or wherein said robotic system includes at least a first display and a second display, the method comprising the operations of viewing the surgical site on the first display and displaying the at least one help-alignment graphic element on the second display (See 0060 as referenced above. Examiner notes the image if the internal surgical site along with the graphical interface is visualized on one display, while an auxiliary display shows the surgical site to a second user).
In view of Suresh’s teachings, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include, with the system and method for aligning a master and slave device for teleoperation including displaying their spatial orientations as disclosed by DiMaio, a system including more than one display to show the surgical site and the graphic elements, with a reasonable expectation of success, since the combination would have predictably allowed a clear depiction of the operative environment to multiple users, thus providing a more efficient and collaborative human-machine interface without changing the underlying teleoperation architecture.
Regarding Claim 5, DiMiao further discloses the method according to claim 1, comprising the operation of generating said help-alignment request signal with said central unit out of a teleoperation (See 0137 as referenced above. See also 0115, “[…] system events may be at a transition between teleoperation of one or more tools and teleoperation of one or more endoscopic cameras, exchanging a first tool for a second tool on a teleoperated manipulator, and other actions that are expected to occur throughout a procedure (e.g. through a surgical procedure in surgical examples). As a specific example, one event that optionally may be used to trigger the request to establish the control relationship […]”).
Regarding Claim 6, DiMiao further discloses the method according to claim 1, comprising the operation of generating said help-alignment request signal with said central unit in one of the following conditions, or their combination:
when the closing/opening command of the master device has been activated (See 0024, “[…] establishing the alignment relationship in response to an indication to begin teleoperation. In some aspects, the indication to begin teleoperation comprises receiving a user command to begin teleoperation […] change in a pose of the end effector in response to the change in the pose of the master input device.” See also 0130, “[…] When the alignment relationships are established and any other conditions necessary for entering a master/slave control mode are met, computer control system 550 outputs a command to operate in the teleoperated system in the master/slave control mode.”);
when the opening/closing command of the master device has been repeatedly activated;
when the master device has been overturned by a hand gesture in a defined time interval;
when a gaze detection device installed in said robotic system detects that the operator is looking in a predetermined direction;
when the master device is subjected to a predetermined hand gesture, in a predefined time interval.
Regarding Claims 7 and 17, DiMiao further discloses the method according to claim 1 and robotic system according to claim 16, comprising the operation of generating said help-alignment request signal with said central unit in one of the following conditions, or their combination:
when said central unit detects that the master device is in a predetermined workspace (See 0134, “[…] establishing the initial master/slave relationship between master device and end effector, the reference frame transform chain between master device and end effector is established for the master device for any master device position and orientation in space at which the user is holding the master device.”);
when said central unit detects that a predetermined movement has been performed with said master device;
when said central unit detects that said master device is in a predefined spatial position (See 0134, “[…] user may choose to visually align the positions and orientations of the master device and the displayed end effector images […]”);
when said central unit detects that a hand-activated switch of the robotic system has been pressed, said hand-activated switch being installed on a handle of said master device and being connected to said central unit;
when said central unit detects that a foot switch of the robotic system has been pressed, said foot switch being connected to said central unit.
Regarding Claim 8, DiMiao further discloses the method according to claim 1, comprising the operation of generating an help-alignment graphic element to:
visually represent the relative spatial orientation between said master device and said slave device (See 0010 and 0137 as referenced above. See also 0020, “[…] determining an orientation of an end effector reference frame relative to a field of view reference frame comprises: determining a complete orientation of the field of view reference frame, and determining a complete orientation of the end effector reference frame. In some aspects, determining an orientation of a master input device reference frame relative to a display reference frame comprises: determining a complete orientation of the display reference frame, and determining a complete orientation of the master input device reference frame.”).
Regarding Claim 9, DiMiao further discloses the method according to claim 1, comprising the operation of generating an help-alignment graphic element so as to:
visually represent said first spatial orientation of the master device and said second spatial orientation of the slave device (See 0010, 0020 and 0137 as referenced above).
Regarding Claim 10, DiMiao further discloses the method according to claim 1, comprising the operation of generating an help-alignment graphic element so as to:
visually represent information, such as a graphic indicator, to move the master device to achieve alignment with said second spatial orientation of the slave device (See 0057, “[…] master devices 110 are computationally aligned with tools 114. Based on this alignment, computer 113 generates commands that correlate the movement of the master devices and the end effectors […]” See also 0156-0157, “Spatial indicators 656 a-656 c also illustrate natural physical features that can be sensed […] Spatial indicators 656 a-656 c in FIG. 6A illustrate such targets or natural features (see also FIG. 5H, targets 512 and 528). A calibration establishes the spatial relationship between the coordinate frame of each target and its corresponding unit base frame. For example, spatial detecting unit 658 acts as an optical tracker, and the spatial relationship between spatial indicators 656 a and 656 b is determined.”).
Regarding Claim 11, DiMiao further discloses the method according to claim 1, wherein when said relative spatial orientation is within a predetermined interval, an enabling signal for a medical or surgical teleoperation is generated (See 0141 as referenced above).
Regarding Claim 12, DiMiao further discloses the method according to claim 1, wherein when said relative spatial orientation is within a predetermined range, a slave device motion enable signal is generated to align with said master device orientation; and
wherein when said relative spatial orientation is within a narrow range, contained in the predetermined range, an enabling signal for a medical or surgical teleoperation is generated (See 0071, “If the perceived spatial motions match, then the user can easily control the slave's movement by moving the master device. But if the perceived spatial motions do not match […] then slave control is difficult. The required alignment is done using known kinematic relationships and reference frame transforms in the teleoperated system […]” See also 0141, “[…] spatial relationships between units are determined in order to establish and maintain effective intuitive control.” Examiner notes it has been held that where the general conditions of a claim are disclosed in the prior art, discovering the optimum or working ranges involves only routine skill in the art. In re Aller, 220 F.2d 454, 456, 105 USPQ 233, 235).
Regarding Claim 13, DiMiao further discloses the method according to claim 1, wherein said enabling signal of a medical or surgical teleoperation is generated by a request from the operator (See 0115 and 0137 as referenced above).
Regarding Claim 14, DiMiao further discloses the method according to claim 1, wherein said signal for enabling a medical or surgical teleoperation is automatically generated when said relative spatial orientation is within a predetermined range (See 0130, “When the alignment relationships are established and any other conditions necessary for entering a master/slave control mode are met, computer control system 550 outputs a command to operate in the teleoperated system in the master/slave control mode.” Examiner notes it has been held that where the general conditions of a claim are disclosed in the prior art, discovering the optimum or working ranges involves only routine skill in the art. In re Aller, 220 F.2d 454, 456, 105 USPQ 233, 235).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Goswami et al. (WO Patent Pub. No. 2019/103954 A1), which is directed towards a robotic system with an operator interactive display for aligning an imaging device and a tool based on an alignment of the operator with an input device
Kralicky et al. (US Patent Pub. No. 2018/0271607 A1), which is directed towards the correction of alignment errors in a master-slave robotic system
Kilroy et al. (WO Patent Pub. No. 2016/137527 A1), which is directed towards a hyper dexterous user interface as a master device where an operator can control a slave device
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Bryant Tang whose telephone number is (571)270-0145. The examiner can normally be reached M-F 8-5 CST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Thomas Worden can be reached at (571)272-4876. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BRYANT TANG/Examiner, Art Unit 3658
/JASON HOLLOWAY/Primary Examiner, Art Unit 3658
Prosecution Insights