DETAILED ACTION
Notice of Pre-AIA or AIA Status
According to a preliminary amendment; claims 2, 7, 17-20, 22-26, 30, 32-40, and 42-43 were cancelled. Claims 1, 3-6, 8-16, 21, 27-29, 31, and 41 are pending in this application.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 3-6, 8-16, 21, 27-29, 31, and 41 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Faraji et al. (US PGPub US 20230110248 filed September 23, 2022 with priority to September 24, 2021).
Consider Claims 1 and 41.
Faraji teaches:
1. (Currently Amended) A camera calibration method for X-ray imaging implemented on a computing device having one or more processors and one or more storage devices, the method comprising: / 41. (Currently Amended) A system, comprising: at least one storage medium including a set of instructions; at least one processor in communication with the at least one storage medium, wherein when executing the set of instructions, the at least one processor is directed to cause the system to perform operations including: (Faraji: abstract, New and innovative systems and methods for calibrating and correcting sensors associated with a collaborative robot are disclosed. An example system comprises: at least one robotic arm; a sensor affixed to a location on the robotic arm, wherein the sensor measures force and torque across six degrees of freedom (6DOF); a processor; and memory. The system may receive, from the sensor, sensor input in real-time that indicate a measured force or torque. The system may generate, in real-time, sensor corrections that correspond to offset, linear, and non-linear deviations of the measured force in each sensor axis. The sensor corrections may correspond to offset, linear, and non-linear cross-coupling of the measured force between two or more sensor axes. The sensor corrections may be determined by applying offset, linear, and non nonlinear corrections to each degree of freedom (DOF) from every other DOF. [0034] Section I. . Exemplary Surgical Environment for the Force/Torque Sensor [0035] FIGS. 1 and 2 show diagrams of a surgical environment including one or more robotic arms having a force/torque sensor, according to example embodiments of the present disclosure. The surgical environment may include a patient 102 that is subjected to or intended to be subjected to a surgical procedure, surgical tools 104, surgical camera (e.g., a stereoscopic visualization camera 106), robotic controls (e.g., a foot pedals 108), at least one robotic arm 114, a surgeon 112, and a surgeon assistant 118. In at least one embodiment, the surgical camera may comprise a stereoscopic visualization camera 106 used within a microsurgical environment 100, according to example embodiments of the present disclosure. The small footprint and maneuverability of the stereoscopic visualization camera (especially when used in conjunction with a multiple-degree of freedom arm) enables flexible positioning with respect to a patient 102. A portion of the patient 102 in view of the stereoscopic visualization camera 106 includes a target site 110. A surgeon 112 can position the stereoscopic visualization camera 106 in virtually any orientation while leaving more than sufficient surgical space above the patient 102 (lying in the supine position). The stereoscopic visualization camera 106 accordingly is minimally intrusive (or not intrusive) to enable the surgeon 112 to perform a life-altering microsurgical procedure without distraction or hindrance.)
1. acquiring at least one image taken by a camera to be calibrated (160), wherein the at least one image includes a calibration target (510, 700, 810, 900), and the calibration target (510, 700, 810, 900) includes at least one calibration point; / 41. acquiring at least one image taken by a camera to be calibrated, wherein the at least one image includes a calibration target (510, 700, 810, 900), and the calibration target (510, 700, 810, 900) includes at least one calibration point; (Faraji: [0042] In some aspects, the computer 124 may process video (or stereoscopic video) signals (e.g., an image or frame stream) from the stereoscopic visualization camera 106 for display on the display monitors 122A and 122B. [0043] FIG. 2 shows a diagram of the microsurgical environment 100 with the patient 102 in a sitting position for a posterior-approach skull base neurosurgery. In the illustrated embodiment, the stereoscopic visualization camera 106 is placed into a horizontal position to face the back of the head of the patient 102. The mechanical robotic arm 114 includes joints that enable the stereoscopic visualization camera 106 to be positioned as shown. In addition, the cart 120 includes a monitor 122C, which may be aligned with the surgeon’s natural view direction.)
1. selecting any calibration point of the at least one calibration point as a target calibration point; / 41. selecting any calibration point of the at least one calibration point as a target calibration point; (Faraji: [0041] In some embodiments, the cart 120 may include a computer 124. In these embodiments, the computer 124 may control a robotic mechanical arm connected to the stereoscopic visualization camera 106. Additionally or alternatively, the computer 124 may process raw sensor data received from force/torque sensor 115 of the robotic arm 114. As will be described further, the computer 124 may perform calibration of the force/torque sensor across multiple dimensions in real-time to resolve any non-linearity in received sensor data, in order to provide accurate force/torque measurements in real-time. The computer 124 may also be used to store raw sensor data received over time into a file (stored to a memory). Such data may be used as training data for the force/torque sensor to learn correct calibration parameters. Further, the computer 124 may also send control signals to the stereoscopic visualization camera 106 to select settings and/or perform calibration. C. Robotic Arm and Stereoscopic Camera Calibration Embodiment [0112] As discussed above, the example stereoscopic visualization camera 106 is configured to provide high-resolution stereoscopic video images of a target surgical site at different magnifications. As part of the stereoscopic robotic platform 302, the stereoscopic visualization camera 106 operates in connection with the robotic arm 114 and/or the coupling plate 306 for precise and clear changes to image focus, working distance, magnification, etc. To accomplish the image acquisition flexibility, the stereoscopic robotic platform 302 is configured to operate one or more calibration, initialization, and/or reset routines. In some embodiments, the stereoscopic visualization camera 106, the robotic arm 114, the coupling plate 306, or more generally, the stereoscopic robotic platform 302 is calibrated during manufacture and/or after installation. Calibration of the camera 106 with the robotic arm 114 provides positioning information of the camera 106 relative to the robotic arm 114 and operator space. After power-up of the stereoscopic robotic platform 302, in some embodiments, the camera 106 and/or the robotic arm 114 is configured to perform further calibration/initialization to measure and verify a location and orientation of the camera 106 at that time.)
1. determining, based on the at least one image, image coordinates of the target calibration point; / 41. determining, based on the at least one image, image coordinates of the target calibration point; (Faraji: [0113] The example processor 1102 is configured to store results from the calibration (e.g., calibration data), in, for example, the memory 1150 and/or the memory 1120 of FIG. 11 . The calibration results may be stored to calibration registers and/or lookup tables (“LUTs”) in the memories 1150 and/or 1120. The stored calibration data relates or maps optical, functional, and/or performance characteristics to attributes of the camera 106, the robotic arm 114, and/or the coupling plate 306 that are adjustable, measurable, and/or verifiable by an operator or by the processor 1102. For example, a working distance actuator motor encoder position for the main objective assembly 702 (of FIG. 5 ) is mapped in a LUT to a working distance. In another example a zoom lens axial position along a linear encoder for the zoom lens assembly 716 is mapped in a LUT to the magnification level. For each of these examples, the example processor 1102 is configured to determine the proper level of an encoder characteristic, adjust, and verify that the characteristic provides the specified or desired working distance and/or magnification. In some embodiments, LUTs may be compound, where multiple performance characteristics are mapped to multiple platform 516 attributes for overall control of all relevant aspects of the camera 106, the robotic arm 114, and/or the coupling plate 306. [0114] )
1. obtaining a first position and a second position of an X-ray imaging device (110, 400) , wherein when the X-ray imaging device (110, 400) is located in the first position and the second position, / 41. obtaining a first position and a second position of an X-ray imaging device (110, 400), wherein when the X-ray imaging device (110, 400) is located in the first position and the second position, (Faraji: [0115] In some embodiments, the example processor 1102 of FIG. 11 is configured to integrate a registration of stereoscopic visualization camera 106, including its visual tip, precisely with a position, direction, and/or orientation calibration of the robotic arm 114 to define a unified position, direction, and/or orientation awareness of acquired stereoscopic images and all points therewithin, with respect to a prescribed coordinate frame. The example processor 1102 is configured to use intrinsic visual imaging data from the stereoscopic visualization camera 106 to coordinate or direct the physical positioning and/or orientating of the robotic arm 114 to provide visualization, as desired by an operator. In addition, such direction and coordination provided by the processor 1102 is provided to maintain preferred characteristics of the visualization such as focus, working distance, pre-defined positioning, orientating, etc. [0055] As will be described further, systems and methods are disclosed for calibrating the force/torque sensor 115 across multiple dimensions in real-time to resolve any non-linearity in received sensor data. The real-time calibration will assist in providing accurate force/torque measurements in real-time. [0056] In the illustrated embodiments, a first end of the robotic arm 114 may be mounted to the cart 120 while a second, opposite end of the robotic arm is mechanically connected to stereoscopic visualization camera 106 (e.g., a robot end effector). FIG. 3 shows the robotic arm 114 holding the stereoscopic visualization camera 106 in an extended position, such as positioning the stereoscopic visualization camera 106 above a surgical site while keeping the rest of the platform 302 out of the way of a surgeon. The cart 120 can be configured to securely hold the stereoscopic robotic platform 302 and can be weighted and balanced to prevent tipping under prescribed operating positions. A. Robotic Arm Embodiment [0057] FIG. 4 illustrates an embodiment of the example robotic arm 114, according to an example embodiment of the present disclosure. In some embodiments, the robotic arm 114 is similar to or comprises model UR5 from Universal Robots S/A. The exterior surfaces of the robotic arm 114 comprise aluminum and plastic materials, which are compatible for use in an operating room and easily cleaned. [0058] While the robotic arm 114 is described herein as being electromechanical, in other examples, the robotic arm 114 may be mechanical, hydraulic, or pneumatic. In some embodiments, the robotic arm 114 may have mixed actuation mechanisms, for example, using a vacuum chuck with a control valve to hold and manipulate the camera 106. Further, while the robotic arm 114 is described below as including a certain number of joints and links, it should be appreciated that the robotic arm 114 may include any number of joints, any lengths of links, and/or comprise any types of joints, or sensors.)
1. the target calibration point (512) is within an imaging field of view of the X-ray imaging device (110, 400) , / 41. the target calibration point (512) is within an imaging field of view of the X-ray imaging device (110, 400), (Faraji: [0059] As described herein, the robotic arm 114 is situated and the joints are oriented to provide an unrestricted view of an operating field while providing a 3D stereoscopic display for an operator for any surgical procedure for a patient. Movement of the robotic arm 114 in noncritical motions is provided to be fast enough for an operator to be convenient yet safe. Movement of the robotic arm 114 is controlled during surgery to be meticulous and accurate. In addition, movement of the robotic arm is controlled to be smooth and predictable through the entire range of motion required for a surgical procedure. As described herein, movement of the robotic arm 114 is controllable by remote control or via manual manipulation of the arm itself. In some embodiments, the robotic arm 114 is configured to be positionable with minimal force (e.g., via an assisted guidance feature) with just the use of, for example, a single auricular finger. [0060] In some embodiments, the robotic arm 114 may include mechanically or electronically locking brakes on the joints. The brakes may be engaged once the aim or “pose”, which is generally the location and direction, of the camera 106 after it is set by an operator. The robotic arm 114 may include a locking or unlocking switch or other input device to prevent undesired manual or accidental motion. When locked, the example robotic arm provides sufficient stability that enables the stereoscopic visualization camera 106 to provide a stable, clear image. The robotic arm 114 may additionally or alternatively include one or more dampening devices to absorb or attenuate vibrations following movement of the stereoscopic visualization camera 106 to a new pose. The dampening devices may include, for example, fluid-filled linear or rotational dampeners, rubber-based vibration isolation mounting dampeners, and/or tuned mass-spring dampeners. Alternatively, or in addition, the arm 114 may include electromechanical dampening, for example, through the use of a proportional integral derivative (“PID”) servo system.)
1. and a first line and a second line are not parallel, the first line connecting a radiation source (430, 520) and a detector (440, 530) of the X-ray imaging device (110, 400) at the first position, the second line connecting the radiation source (430, 520) and the detector (440, 530) of the X-ray imaging device (110, 400) at the second position; / 41. and a first line and a second line are not parallel, the first line connecting a radiation source and a detector (440, 530) of the X-ray imaging device (110, 400) at the first position, the second line connecting the radiation source (430, 520) and the detector (440, 530) of the X-ray imaging device (110, 400) at the second position; (Faraji: [0062] In the illustrated embodiment of FIG. 4 , the robotic arm 114 includes six joints, labeled R1, R2, R3, R4, R5, and R6. In other embodiments, the robotic arm 114 may include fewer or additional joints. Additionally, in some embodiments, at least some of the joints R1 to R6 have rotational motion capabilities of +/-360°. The rotational motion may be provided by an electromechanical subsystem that includes, for each joint, an electric motor configured to drive a mechanical rotational joint through one or more anti-backlash joint gearboxes. Each of the joints R1 to R6 may include one or more rotational sensors to detect joint position. Further, each joint may include a slip clutch and/or an electromechanical brake.
[0063] Each of the joints R1 to R6 may have an overall repeatability of motion (with the camera 106 attached) of approximately +/-⅒ of a millimeter (“mm”). The joints may be have variable rotational speeds that can be controlled between 0.5° to 180° per second. Together, this translates to camera movement between 1 mm per second to 1 meter per second. In some embodiments, the stereoscopic robotic platform 302 may have speed governors for one or more of the joints R1 to R6 that are in place during surgical procedures. Each of the joints R1 to R6 may be electrically connected to a power source and/or command line in a controller of the robotic arm 114. Wires for power and command signals may be routed internally within the joints and links. Further, one or more of the joints may include dampeners, such as o-rings for connection to links. The dampeners may, for example, reduce or absorb vibrations in the robotic arm 114, vibrations from the cart 120, and/or vibrations imparted via the stereoscopic visualization camera 106.)
1. determining, based on the first position and the second position, spatial coordinates of the target calibration point; / 41. (v) determining, based on the first position and the second position, spatial coordinates of the target calibration point; and (Faraji: 0064] Joint R1 includes a base joint that is mechanically coupled to a flange 402, which is secured to a stationary structure 404. The flange 402 may include any type of mechanical connector. The stationary structure 404 may include, for example, the cart 120 of FIG. 3 , a wall, a ceiling, a table, etc. The joint R1 is configured to rotate around a first axis 410, which may include the z-axis. [0065] Joint R1 is connected to joint R2 via a link 430. The example link 430 includes a cylinder or other tubular structure configured to provide structural support for the downstream sections of the robotic arm 114. The link 430 is configured to provide a rotational secure connection with joint R2 to enable joint R2 to rotate while the link 430 is held in place by its connection to the joint R1. Joint R2 may include, for example, a shoulder joint configured to rotate around an axis 412. The example axis 412 is configured to be perpendicular (or substantially perpendicular) to axis 410. The axis 412 is configured to be within an x-y plane given the rotation of the joint R1 around the z-axis. [0066] Joint R2 is mechanically coupled to joint R3 via link 432. The link 432 is configured to have a greater length than the link 430 and is configured to provide structural support for downstream portions of the robotic arm 114. Joint R3 may include, for example, an elbow joint. Together with joint R2, joint R3 provides extensible positioning and/or orientating of the robotic arm 114. The joint R3 is configured to rotate around an axis 414, which is perpendicular or orthogonal to the axis 410 and parallel to the axis 412. [0067] Joint R3 is connected to joint R4 via link 434, which provides structural support for downstream portions of the robotic arm 114. The example joint R4 may be, for example, a first wrist joint configured to provide rotation around axis 416, which may be orthogonal to the axes 412 and 414. Joint R4 is mechanically connected to joint R5 via link 436. Joint R5 may be a second wrist joint configured to provide rotation around an axis 418, which is orthogonal to axis 416. Joint R5 is mechanically connected to joint R6 via link 438. Joint R6 may be a third wrist joint configured to rotate around axis 420, which is orthogonal to the axis 418. Together, the wrist joints R4 to R6 provide precise flexibility in positioning the stereoscopic visualization camera 106 described herein. [0124])
1. and performing calibration on the camera to be calibrated (160) based on the image coordinates and the spatial coordinates./ 41. (vi) performing calibration on the camera to be calibrated (160) based on the image coordinates and the spatial coordinates. (Faraji: [0068] The example robotic arm 114 includes a connector 450. The example connector 3450 is connected to joint R6 via link 440. In some embodiments, the example link 440 may include a sleeve that enables joint R6 to rotate the connector 450. As discussed herein, the connector 3450 may be configured to mechanically couple to the coupling plate 306 or the stereoscopic visualization camera 106 directly when a coupling plate is not used. The connector 450 may include one or more screws to secure the robotic arm 114 to the coupling plate 304 and/or the stereoscopic visualization camera 106. [0071] FIG. 6 shows an embodiment where the robotic arm 114 is mounted to a stationary structure 404 that is a ceiling plate. The robotic arm may be suspended from the ceiling of an operating room via the flange 402 to reduce floor space clutter. The robotic arm 114, including the joints, can be positioned above and traversed from the area where surgical activity is being performed, out of the way of the surgeon and operating room staff, yet still providing functional positioning and/or orientating of the camera 106 while providing a clear view of the display monitors (e.g., display monitors 122A-E as previously described in relation to FIGS. 1-3 ). [0072] FIG. 7 shows an embodiment of the coupling plate 306. In the illustrated example, a first end 702 of the coupling plate 306 is connected to the connector 450 of the robotic arm 114. A second end 704 of the coupling plate 306 is connected to the stereoscopic visualization camera 106. The example coupling plate 306 is configured to provide additional degrees of freedom for moving the stereoscopic visualization camera 106. The coupling plate 306 also extends the maximum reach of the robotic arm 114. The coupling plate 306 may have a length between 10 cm to 100 cm. [0102] After the boundary checks are performed, the robotic arm controller 1106 uses the movement delta and the current position/orientation of each of joints R1 to R9 to determine an optimal or near optimal movement sequence for rotating one or more of the joints to cause the robotic arm 114 to move the camera 106 into the specified location. The robotic arm controller 1106 may use, for example, an optimization routine that determines a minimal amount of joint movement needed to satisfy the movement delta vector. After the amount of joint movement is determined, the example robotic arm controller 1106 is configured to send one or more messages (indicative of an amount of rotation and speed of rotation, taking into account any scale factors) to a motor controller 1124. The robotic arm controller 1106 may transmit a sequence of messages to cause the robotic arm 114 and/or coupling plate 306 to move in a defined or coordinated sequence. The sequence of messages may also cause a change in joint speed as, for example, the robotic arm 114 approaches a virtual or physical boundary.)
Consider Claim 31.
Faraji teaches:
31. (Original) A calibration target (510, 700, 810, 900), the calibration target (510, 700, 810, 900) includes at least one calibration unit (600, 711, 712, 721, 910, 920, 930, 940, 950, 960), (Faraji: abstract, New and innovative systems and methods for calibrating and correcting sensors associated with a collaborative robot are disclosed. An example system comprises: at least one robotic arm; a sensor affixed to a location on the robotic arm, wherein the sensor measures force and torque across six degrees of freedom (6DOF); a processor; and memory. The system may receive, from the sensor, sensor input in real-time that indicate a measured force or torque. The system may generate, in real-time, sensor corrections that correspond to offset, linear, and non-linear deviations of the measured force in each sensor axis. The sensor corrections may correspond to offset, linear, and non-linear cross-coupling of the measured force between two or more sensor axes. The sensor corrections may be determined by applying offset, linear, and non nonlinear corrections to each degree of freedom (DOF) from every other DOF. [0034] Section I. . Exemplary Surgical Environment for the Force/Torque Sensor [0035] FIGS. 1 and 2 show diagrams of a surgical environment including one or more robotic arms having a force/torque sensor, according to example embodiments of the present disclosure. The surgical environment may include a patient 102 that is subjected to or intended to be subjected to a surgical procedure, surgical tools 104, surgical camera (e.g., a stereoscopic visualization camera 106), robotic controls (e.g., a foot pedals 108), at least one robotic arm 114, a surgeon 112, and a surgeon assistant 118. In at least one embodiment, the surgical camera may comprise a stereoscopic visualization camera 106 used within a microsurgical environment 100, according to example embodiments of the present disclosure. The small footprint and maneuverability of the stereoscopic visualization camera (especially when used in conjunction with a multiple-degree of freedom arm) enables flexible positioning with respect to a patient 102. A portion of the patient 102 in view of the stereoscopic visualization camera 106 includes a target site 110. A surgeon 112 can position the stereoscopic visualization camera 106 in virtually any orientation while leaving more than sufficient surgical space above the patient 102 (lying in the supine position). The stereoscopic visualization camera 106 accordingly is minimally intrusive (or not intrusive) to enable the surgeon 112 to perform a life-altering microsurgical procedure without distraction or hindrance.)
31. each of the at least one calibration unit (600, 711,712, 721, 910, 920, 930, 940, 950, 960) includes a calibration part (620) and a base plate (610), (Faraji: [0042] In some aspects, the computer 124 may process video (or stereoscopic video) signals (e.g., an image or frame stream) from the stereoscopic visualization camera 106 for display on the display monitors 122A and 122B. [0043] FIG. 2 shows a diagram of the microsurgical environment 100 with the patient 102 in a sitting position for a posterior-approach skull base neurosurgery. In the illustrated embodiment, the stereoscopic visualization camera 106 is placed into a horizontal position to face the back of the head of the patient 102. The mechanical robotic arm 114 includes joints that enable the stereoscopic visualization camera 106 to be positioned as shown. In addition, the cart 120 includes a monitor 122C, which may be aligned with the surgeon’s natural view direction.)
and a geometric center (630) of the calibration part (620) is set as a calibration point, the calibration part (620) is disposed on the base plate (610), (Faraji: [0122] After the example stereoscopic visualization camera 106 is connected to the robotic arm 114, the example processor 1102 and/or the robotic arm controller 1106 are configured to calibrate the camera and its view vector into a coordinate system originated around the stationary base 404 of the robotic arm 114 (block 1206). The coordinate system is referred to herein as “robot space” or “robotic space”. During this calibration step, known movements to the robotic arm 114 are used by the processor 1102 and/or the robotic arm controller 1106 to determine an orientation and a location of a view vector and object plane of the camera 106 during visualization of a target surgical site. [0123] In some embodiments, the mechanical features of the camera 106, the coupling plate 306, and the robotic arm 114 exist such that, when mechanically connected together, the relationship between the camera 106, the coupling plate 306, and the robotic arm 114 is uniquely determined and known. In these embodiments, the processor 1102 and/or the robotic arm controller 1106 determine the position, direction, and/or orientation of the view vector from the known mechanical geometry of the camera 106, the coupling plate 306, and the robotic arm 114. [0113] The example processor 1102 is configured to store results from the calibration (e.g., calibration data), in, for example, the memory 1150 and/or the memory 1120 of FIG. 11 . The calibration results may be stored to calibration registers and/or lookup tables (“LUTs”) in the memories 1150 and/or 1120. The stored calibration data relates or maps optical, functional, and/or performance characteristics to attributes of the camera 106, the robotic arm 114, and/or the coupling plate 306 that are adjustable, measurable, and/or verifiable by an operator or by the processor 1102. For example, a working distance actuator motor encoder position for the main objective assembly 702 (of FIG. 5 ) is mapped in a LUT to a working distance. In another example a zoom lens axial position along a linear encoder for the zoom lens assembly 716 is mapped in a LUT to the magnification level. For each of these examples, the example processor 1102 is configured to determine the proper level of an encoder characteristic, adjust, and verify that the characteristic provides the specified or desired working distance and/or magnification. In some embodiments, LUTs may be compound, where multiple performance characteristics are mapped to multiple platform 516 attributes for overall control of all relevant aspects of the camera 106, the robotic arm 114, and/or the coupling plate 306. [0114])
31. and there is an X-ray attenuation difference and an optical imaging difference between the calibration part (620) and the base plate (610). (Faraji: [0041] In some embodiments, the cart 120 may include a computer 124. In these embodiments, the computer 124 may control a robotic mechanical arm connected to the stereoscopic visualization camera 106. Additionally or alternatively, the computer 124 may process raw sensor data received from force/torque sensor 115 of the robotic arm 114. As will be described further, the computer 124 may perform calibration of the force/torque sensor across multiple dimensions in real-time to resolve any non-linearity in received sensor data, in order to provide accurate force/torque measurements in real-time. The computer 124 may also be used to store raw sensor data received over time into a file (stored to a memory). Such data may be used as training data for the force/torque sensor to learn correct calibration parameters. Further, the computer 124 may also send control signals to the stereoscopic visualization camera 106 to select settings and/or perform calibration. C. Robotic Arm and Stereoscopic Camera Calibration Embodiment [0112] As discussed above, the example stereoscopic visualization camera 106 is configured to provide high-resolution stereoscopic video images of a target surgical site at different magnifications. As part of the stereoscopic robotic platform 302, the stereoscopic visualization camera 106 operates in connection with the robotic arm 114 and/or the coupling plate 306 for precise and clear changes to image focus, working distance, magnification, etc. To accomplish the image acquisition flexibility, the stereoscopic robotic platform 302 is configured to operate one or more calibration, initialization, and/or reset routines. In some embodiments, the stereoscopic visualization camera 106, the robotic arm 114, the coupling plate 306, or more generally, the stereoscopic robotic platform 302 is calibrated during manufacture and/or after installation. Calibration of the camera 106 with the robotic arm 114 provides positioning information of the camera 106 relative to the robotic arm 114 and operator space. After power-up of the stereoscopic robotic platform 302, in some embodiments, the camera 106 and/or the robotic arm 114 is configured to perform further calibration/initialization to measure and verify a location and orientation of the camera 106 at that time.)
Consider Claim 3.
Faraji teaches:
3. (Currently Amended) The method of claim 1, when the X-ray imaging device (110, 400) is located at the first position and/or the second position, the first line and/or the second line overlaps with the target calibration point. (Faraji: [0064]-[0066], [0127] In other embodiments, the processor 1102 and/or the robotic arm controller 1106 may use alternative methods to calibrate the camera 106 to robot space. In this context, “calibration” is taken to mean “registration”, where the processor 1102 and/or the robotic arm controller 1106 are configured to calculate registration over a wide space in which the registration may vary. For example, a system can be used where a separate stereoscopic camera is used to observe and locate calibration targets on the cart 120 as well as similar calibration targets which are installed on the camera 106 and/or on a patient or surgical bed. The processor 1102 and/or the robotic arm controller 1106 are configured to model and track the camera 106, which is modeled and tracked as a surgical instrument with a view vector and working distance. The view vector and working distance define parameters for accurately visualizing a target surgical site. In these other embodiments, the other camera determines and reports location and orientation information for the coordinate frame of each such instrument in a reference frame, such as the stereoscopic camera 106. Then, using linear algebra, the poses and/or locations of instruments relative to each other are calculated by the processor 1102 and/or the robotic arm controller 1106, thereby resulting in a calibration of the camera 106 to the robot space.)
Consider Claim 4.
Faraji teaches:
4. (Original) The method of claim 3, wherein the first position is a position where the first line is perpendicular to a plane of the calibration target (510, 700, 810, 900), and the second position is a position where the second line is parallel to the plane of the calibration target (510, 700, 810, 900). (Faraji: [0064]-[0065] Joint R1 is connected to joint R2 via a link 430. The example link 430 includes a cylinder or other tubular structure configured to provide structural support for the downstream sections of the robotic arm 114. The link 430 is configured to provide a rotational secure connection with joint R2 to enable joint R2 to rotate while the link 430 is held in place by its connection to the joint R1. Joint R2 may include, for example, a shoulder joint configured to rotate around an axis 412. The example axis 412 is configured to be perpendicular (or substantially perpendicular) to axis 410. The axis 412 is configured to be within an x-y plane given the rotation of the joint R1 around the z-axis. [0066] Joint R2 is mechanically coupled to joint R3 via link 432. The link 432 is configured to have a greater length than the link 430 and is configured to provide structural support for downstream portions of the robotic arm 114. Joint R3 may include, for example, an elbow joint. Together with joint R2, joint R3 provides extensible positioning and/or orientating of the robotic arm 114. The joint R3 is configured to rotate around an axis 414, which is perpendicular or orthogonal to the axis 410 and parallel to the axis 412.)
Consider Claim 5.
Faraji teaches:
5. (Original) The method of claim 4, wherein determining, based on the first position and the second position, the spatial coordinates of the target calibration point (512) includes: determining, based on the first position, first two-dimensional (2D) coordinates of the target calibration point (512) within the plane of the calibration target (510, 700, 810, 900); determining, based on the second position, second 2D coordinates of the target calibration point (512) within a plane perpendicular to the plane of the calibration target (510, 700, 810, 900); and determining, based on the first 2D coordinates and the second 2D coordinates, the spatial coordinates. (Faraji: [0123] In some embodiments, the mechanical features of the camera 106, the coupling plate 306, and the robotic arm 114 exist such that, when mechanically connected together, the relationship between the camera 106, the coupling plate 306, and the robotic arm 114 is uniquely determined and known. In these embodiments, the processor 1102 and/or the robotic arm controller 1106 determine the position, direction, and/or orientation of the view vector from the known mechanical geometry of the camera 106, the coupling plate 306, and the robotic arm 114. [0124] In other embodiments where the mechanical features do not exist, the example processor 1102 and/or the robotic arm controller 1106 are configured to perform a routine to accurately determine a spatial relationship between the camera 106 and the robotic arm 114 in robot space. The processor 1102 and/or the robotic arm controller 1106 move the stereoscopic visualization camera 106 to a start position, which may include a stow position, a re-orientation position, or a surgical position. The stereoscopic visualization camera 106 then moves the camera from the start position to a position that approximately visualizes a calibration target located on the stationary base 404 of the robotic arm 114. The calibration target may be located, for example, at a convenient area of the cart 120 in a position within the motion sphere of the robotic arm 114. Some examples of the calibration target include, for example, small spheres or other uniquely recognizable objects that can be located relative to each other (in two-dimensional or stereoscopic images) in a unique, known orientation. The coordinates of the spheres are fixed and known with respect to the cart 120 and stationary base 404, and are hence known in robot space. The processor 1102 and/or the robotic arm controller 1106 are configured to store the coordinates to, for example, the memory 1120. [0125] During the calibration, the processor 1102 and/or the robotic arm controller 1106 receive view vector data 1207 regarding working distance, magnification, stereoscopic optical axis, and/or IPD. The stereoscopic visualization camera 106 is set to visualize the spheres at the calibration target simultaneously and determine their position through the use of parallax in the stereoscopic image. The processor 1102 and/or the robotic arm controller 1106 records the positions of the spheres in an initial coordinate system, for example, X, Y, and Z with respect to a fiducial in the camera 106 (i.e. “camera space). The X,Y,Z position may correspond to an origin location, and be defined in a file or LUT as being the origin or having other known coordinate values.)
Consider Claim 6.
Faraji teaches:
6. (Currently Amended) The method of claim 3, wherein the first position corresponds to a first angle, and the second position corresponds to a second angle; and determining, based on the first position and the second position, the spatial coordinates of the target calibration point (512) includes: determining first coordinates of the first line based on the first position; determining second coordinates of the second line based on the second position; and determining the spatial coordinates based on the first coordinates and the second coordinates. (Faraji: [0101] The memory 1120 may include one or more instructions that specify how joints R1 to R9 are moved based on a known position of the joints. The robotic arm controller 1106 is configured to execute the one or more instructions to determine how instructed camera movement is translated into joint movement. In an example, the robotic arm controller 1106 may receive messages from the processor 1102 indicative that the stereoscopic visualization camera 106 is to move downward along a z-axis and move sideward in an x-y plane. In other words, the processor 1102 transmits indicative of inputs received via the input devices 1110 a-1110 b regarding desired movement of the camera 106. The example robotic arm controller 1106 is configured to translate the vectors of movement in 3-dimensional coordinates into joint position movement information that achieves the desired position/orientation. The robotic arm controller 1106 may determine or take into account the current location of the links and joints of the robotic arm 114 and/or the coupling plate 306 (and/or a position/orientation of the camera 106) in conjunction with the desired movement to determine a movement delta vector. In addition, the robotic arm controller 1106 may perform one or more checks to ensure the desired movement does not cause the camera 106 to enter into or progress close to a restricted area, as specified by one or more three-dimensional boundaries that are defined in the same coordinate system as the arm 114 and coupling plate 306. Areas close to a boundary may specify a reduced scale factor that is applied by the robotic arm controller 1106 when movement signals are sent to the joints, which causes the joints to move slower as the robotic arm 114 approaches a boundary, and not move any further past a boundary. [0102] After the boundary checks are performed, the robotic arm controller 1106 uses the movement delta and the current position/orientation of each of joints R1 to R9 to determine an optimal or near optimal movement sequence for rotating one or more of the joints to cause the robotic arm 114 to move the camera 106 into the specified location. The robotic arm controller 1106 may use, for example, an optimization routine that determines a minimal amount of joint movement needed to satisfy the movement delta vector. After the amount of joint movement is determined, the example robotic arm controller 1106 is configured to send one or more messages (indicative of an amount of rotation and speed of rotation, taking into account any scale factors) to a motor controller 1124. The robotic arm controller 1106 may transmit a sequence of messages to cause the robotic arm 114 and/or coupling plate 306 to move in a defined or coordinated sequence. The sequence of messages may also cause a change in joint speed as, for example, the robotic arm 114 approaches a virtual or physical boundary.)
Consider Claim 8.
Faraji teaches:
8. (Currently Amended) The method of claim 3, wherein the obtaining the first position of the X-ray imaging device (110, 400) includes: acquiring a first image captured by the X-ray imaging device (110, 400) at a first candidate position, the first image including the target calibration point; determining a first magnification based on the first image; determining a first deviation between a first image position of the target calibration point (512) in the first image and an image center of the first image; determining, based on the first magnification and the first deviation, a second deviation between a spatial position of the target calibration point (512) and the first line; determining the first position based on the second deviation. (Faraji: C. Robotic Arm and Stereoscopic Camera Calibration Embodiment [0112] As discussed above, the example stereoscopic visualization camera 106 is configured to provide high-resolution stereoscopic video images of a target surgical site at different magnifications. As part of the stereoscopic robotic platform 302, the stereoscopic visualization camera 106 operates in connection with the robotic arm 114 and/or the coupling plate 306 for precise and clear changes to image focus, working distance, magnification, etc. To accomplish the image acquisition flexibility, the stereoscopic robotic platform 302 is configured to operate one or more calibration, initialization, and/or reset routines. In some embodiments, the stereoscopic visualization camera 106, the robotic arm 114, the coupling plate 306, or more generally, the stereoscopic robotic platform 302 is calibrated during manufacture and/or after installation. Calibration of the camera 106 with the robotic arm 114 provides positioning information of the camera 106 relative to the robotic arm 114 and operator space. After power-up of the stereoscopic robotic platform 302, in some embodiments, the camera 106 and/or the robotic arm 114 is configured to perform further calibration/initialization to measure and verify a location and orientation of the camera 106 at that time. [0113] In another example a zoom lens axial position along a linear encoder for the zoom lens assembly 716 is mapped in a LUT to the magnification level. For each of these examples, the example processor 1102 is configured to determine the proper level of an encoder characteristic, adjust, and verify that the characteristic provides the specified or desired working distance and/or magnification. In some embodiments, LUTs may be compound, where multiple performance characteristics are mapped to multiple platform 516 attributes for overall control of all relevant aspects of the camera 106, the robotic arm 114, and/or the coupling plate 306. [0114] The combination of a robotic arm 114 and the example stereoscopic visualization camera 106 provides highly accurate position, direction, and/or orientation information of the target view with respect to a frame of reference of the robotic arm 114. The following sections describe how the stereoscopic visualization camera 106 is calibrated to define a visual tip. After a visual tip is determined, the stereoscopic visualization camera 106 is registered to a frame of reference of the robotic arm 114 and/or the coupling plate 306. Accordingly, after calibration and registration, a stereoscopic view of a surgical site is unified with the integrated control of the stereoscopic visualization camera 106 combined with the position, direction, and orientation control of the robotic arm 114 and/or coupling plate 306. [0115] In some embodiments, the example processor 1102 of FIG. 11 is configured to integrate a registration of stereoscopic visualization camera 106, including its visual tip, precisely with a position, direction, and/or orientation calibration of the robotic arm 114 to define a unified position, direction, and/or orientation awareness of acquired stereoscopic images and all points therewithin, with respect to a prescribed coordinate frame. The example processor 1102 is configured to use intrinsic visual imaging data from the stereoscopic visualization camera 106 to coordinate or direct the physical positioning and/or orientating of the robotic arm 114 to provide visualization, as desired by an operator. In addition, such direction and coordination provided by the processor 1102 is provided to maintain preferred characteristics of the visualization such as focus, working distance, pre-defined positioning, orientating, etc.
[0116] In some embodiments, calibration of the stereoscopic visualization camera 106, the robotic arm 114, and/or the coupling plate 306 generally includes (i) determining and/or measuring inaccuracies of functional parameters of the stereoscopic robotic platform 302 that affect stereoscopic images; (ii) calibrating or adjusting, the stereoscopic robotic platform 302 to minimize the inaccuracies in the stereoscopic images at or below a desired level; (iii) verifying that the adjustments have been made within a desired level of calibration accuracy through simultaneous comparisons of the dual channels of the stereoscopic images to each other or calibration templates; and (iv) using the stereoscopic robotic platform 302 in the performance of its tasks, where a level of the accuracy of the calibration is detectable and maintained. [0117] In an alternative embodiment, the robotic arm 114 is provided with one or more fixed calibration fiducials that are used to precisely calibrate a physical relationship of the joint and links of the robotic arm 114 to one another as well as to calibrate a relationship of the visual tip of the camera 106 to the robotic arm 114 and/or an initial pose configuration. The robotic platform fixed calibration fiducials can be used to register or integrate the robotic arm 114 with an external environment, such as an operating room theater or with a patient or target space within an external environment. The fixed calibration fiducials can either include a dedicated attachment to an external portion of a body of the robotic arm 114 or a combination of known external features of the robotic arm 114, such as mounting points, joints, corners, or the like.)
Consider Claim 9.
Faraji teaches:
9. (Original) The method of claim 8, wherein determining the first magnification based on the first image includes: obtaining a size of at least part of the calibration target (510, 700, 810, 900) in the first image; obtaining an actual size of the at least part of the calibration target (510, 700, 810, 900); and determining the first magnification based on the size of the at least part of the calibration target (510, 700, 810, 900) in the first image and the actual size of the at least part of the calibration target (510, 700, 810, 900). (Faraji: C. Robotic Arm and Stereoscopic Camera Calibration Embodiment [0112] As discussed above, the example stereoscopic visualization camera 106 is configured to provide high-resolution stereoscopic video images of a target surgical site at different magnifications. As part of the stereoscopic robotic platform 302, the stereoscopic visualization camera 106 operates in connection with the robotic arm 114 and/or the coupling plate 306 for precise and clear changes to image focus, working distance, magnification, etc. To accomplish the image acquisition flexibility, the stereoscopic robotic platform 302 is configured to operate one or more calibration, initialization, and/or reset routines. In some embodiments, the stereoscopic visualization camera 106, the robotic arm 114, the coupling plate 306, or more generally, the stereoscopic robotic platform 302 is calibrated during manufacture and/or after installation. Calibration of the camera 106 with the robotic arm 114 provides positioning information of the camera 106 relative to the robotic arm 114 and operator space. After power-up of the stereoscopic robotic platform 302, in some embodiments, the camera 106 and/or the robotic arm 114 is configured to perform further calibration/initialization to measure and verify a location and orientation of the camera 106 at that time. [0113] In another example a zoom lens axial position along a linear encoder for the zoom lens assembly 716 is mapped in a LUT to the magnification level. For each of these examples, the example processor 1102 is configured to determine the proper level of an encoder characteristic, adjust, and verify that the characteristic provides the specified or desired working distance and/or magnification. In some embodiments, LUTs may be compound, where multiple performance characteristics are mapped to multiple platform 516 attributes for overall control of all relevant aspects of the camera 106, the robotic arm 114, and/or the coupling plate 306. [0114] The combination of a robotic arm 114 and the example stereoscopic visualization camera 106 provides highly accurate position, direction, and/or orientation information of the target view with respect to a frame of reference of the robotic arm 114. The following sections describe how the stereoscopic visualization camera 106 is calibrated to define a visual tip. After a visual tip is determined, the stereoscopic visualization camera 106 is registered to a frame of reference of the robotic arm 114 and/or the coupling plate 306. Accordingly, after calibration and registration, a stereoscopic view of a surgical site is unified with the integrated control of the stereoscopic visualization camera 106 combined with the position, direction, and orientation control of the robotic arm 114 and/or coupling plate 306. [0115] In some embodiments, the example processor 1102 of FIG. 11 is configured to integrate a registration of stereoscopic visualization camera 106, including its visual tip, precisely with a position, direction, and/or orientation calibration of the robotic arm 114 to define a unified position, direction, and/or orientation awareness of acquired stereoscopic images and all points therewithin, with respect to a prescribed coordinate frame. The example processor 1102 is configured to use intrinsic visual imaging data from the stereoscopic visualization camera 106 to coordinate or direct the physical positioning and/or orientating of the robotic arm 114 to provide visualization, as desired by an operator. In addition, such direction and coordination provided by the processor 1102 is provided to maintain preferred characteristics of the visualization such as focus, working distance, pre-defined positioning, orientating, etc. [0116] In some embodiments, calibration of the stereoscopic visualization camera 106, the robotic arm 114, and/or the coupling plate 306 generally includes (i) determining and/or measuring inaccuracies of functional parameters of the stereoscopic robotic platform 302 that affect stereoscopic images; (ii) calibrating or adjusting, the stereoscopic robotic platform 302 to minimize the inaccuracies in the stereoscopic images at or below a desired level; (iii) verifying that the adjustments have been made within a desired level of calibration accuracy through simultaneous comparisons of the dual channels of the stereoscopic images to each other or calibration templates; and (iv) using the stereoscopic robotic platform 302 in the performance of its tasks, where a level of the accuracy of the calibration is detectable and maintained. [0117] In an alternative embodiment, the robotic arm 114 is provided with one or more fixed calibration fiducials that are used to precisely calibrate a physical relationship of the joint and links of the robotic arm 114 to one another as well as to calibrate a relationship of the visual tip of the camera 106 to the robotic arm 114 and/or an initial pose configuration. The robotic platform fixed calibration fiducials can be used to register or integrate the robotic arm 114 with an external environment, such as an operating room theater or with a patient or target space within an external environment. The fixed calibration fiducials can either include a dedicated attachment to an external portion of a body of the robotic arm 114 or a combination of known external features of the robotic arm 114, such as mounting points, joints, corners, or the like.)
Consider Claim 10.
Faraji teaches:
10. (Original) The method of claim 8, wherein determining the first magnification based on the first image includes: acquiring a third image captured by the X-ray imaging device (110, 400) at an auxiliary position, the third image including the target calibration point (512); obtaining first coordinates related to the X-ray imaging device (110, 400) at the first candidate position; obtaining second coordinates related to related to the X-ray imaging device (110, 400) at the auxiliary position; obtaining a third coordinates of the target calibration point (512) in the first image; obtaining a fourth coordinates of the target calibration point (512) in the third image; and determining the first magnification based on the first coordinates, the second coordinates, the third coordinates, and the fourth coordinates. (Faraji: C. Robotic Arm and Stereoscopic Camera Calibration Embodiment [0112] As discussed above, the example stereoscopic visualization camera 106 is configured to provide high-resolution stereoscopic video images of a target surgical site at different magnifications. As part of the stereoscopic robotic platform 302, the stereoscopic visualization camera 106 operates in connection with the robotic arm 114 and/or the coupling plate 306 for precise and clear changes to image focus, working distance, magnification, etc. To accomplish the image acquisition flexibility, the stereoscopic robotic platform 302 is configured to operate one or more calibration, initialization, and/or reset routines. In some embodiments, the stereoscopic visualization camera 106, the robotic arm 114, the coupling plate 306, or more generally, the stereoscopic robotic platform 302 is calibrated during manufacture and/or after installation. Calibration of the camera 106 with the robotic arm 114 provides positioning information of the camera 106 relative to the robotic arm 114 and operator space. After power-up of the stereoscopic robotic platform 302, in some embodiments, the camera 106 and/or the robotic arm 114 is configured to perform further calibration/initialization to measure and verify a location and orientation of the camera 106 at that time. [0113] In another example a zoom lens axial position along a linear encoder for the zoom lens assembly 716 is mapped in a LUT to the magnification level. For each of these examples, the example processor 1102 is configured to determine the proper level of an encoder characteristic, adjust, and verify that the characteristic provides the specified or desired working distance and/or magnification. In some embodiments, LUTs may be compound, where multiple performance characteristics are mapped to multiple platform 516 attributes for overall control of all relevant aspects of the camera 106, the robotic arm 114, and/or the coupling plate 306. [0114] The combination of a robotic arm 114 and the example stereoscopic visualization camera 106 provides highly accurate position, direction, and/or orientation information of the target view with respect to a frame of reference of the robotic arm 114. The following sections describe how the stereoscopic visualization camera 106 is calibrated to define a visual tip. After a visual tip is determined, the stereoscopic visualization camera 106 is registered to a frame of reference of the robotic arm 114 and/or the coupling plate 306. Accordingly, after calibration and registration, a stereoscopic view of a surgical site is unified with the integrated control of the stereoscopic visualization camera 106 combined with the position, direction, and orientation control of the robotic arm 114 and/or coupling plate 306. [0115] In some embodiments, the example processor 1102 of FIG. 11 is configured to integrate a registration of stereoscopic visualization camera 106, including its visual tip, precisely with a position, direction, and/or orientation calibration of the robotic arm 114 to define a unified position, direction, and/or orientation awareness of acquired stereoscopic images and all points therewithin, with respect to a prescribed coordinate frame. The example processor 1102 is configured to use intrinsic visual imaging data from the stereoscopic visualization camera 106 to coordinate or direct the physical positioning and/or orientating of the robotic arm 114 to provide visualization, as desired by an operator. In addition, such direction and coordination provided by the processor 1102 is provided to maintain preferred characteristics of the visualization such as focus, working distance, pre-defined positioning, orientating, etc. [0116] In some embodiments, calibration of the stereoscopic visualization camera 106, the robotic arm 114, and/or the coupling plate 306 generally includes (i) determining and/or measuring inaccuracies of functional parameters of the stereoscopic robotic platform 302 that affect stereoscopic images; (ii) calibrating or adjusting, the stereoscopic robotic platform 302 to minimize the inaccuracies in the stereoscopic images at or below a desired level; (iii) verifying that the adjustments have been made within a desired level of calibration accuracy through simultaneous comparisons of the dual channels of the stereoscopic images to each other or calibration templates; and (iv) using the stereoscopic robotic platform 302 in the performance of its tasks, where a level of the accuracy of the calibration is detectable and maintained. [0117] In an alternative embodiment, the robotic arm 114 is provided with one or more fixed calibration fiducials that are used to precisely calibrate a physical relationship of the joint and links of the robotic arm 114 to one another as well as to calibrate a relationship of the visual tip of the camera 106 to the robotic arm 114 and/or an initial pose configuration. The robotic platform fixed calibration fiducials can be used to register or integrate the robotic arm 114 with an external environment, such as an operating room theater or with a patient or target space within an external environment. The fixed calibration fiducials can either include a dedicated attachment to an external portion of a body of the robotic arm 114 or a combination of known external features of the robotic arm 114, such as mounting points, joints, corners, or the like.)
Consider Claim 11.
Faraji teaches:
11. (Original) The method of claim 10, wherein a line connecting the radiation source (430, 520) and the detector (440, 530) at the first candidate position and a line connecting the radiation source (430, 520) and the detector (440, 530) at the auxiliary position are parallel. (Faraji: [0066] Joint R2 is mechanically coupled to joint R3 via link 432. The link 432 is configured to have a greater length than the link 430 and is configured to provide structural support for downstream portions of the robotic arm 114. Joint R3 may include, for example, an elbow joint. Together with joint R2, joint R3 provides extensible positioning and/or orientating of the robotic arm 114. The joint R3 is configured to rotate around an axis 414, which is perpendicular or orthogonal to the axis 410 and parallel to the axis 412. [0074] The joints R7 to R9 may be connected together via respective links. In addition, a link 718 is provided for connection to the connector 450 of the robotic arm 114. Joint R7 is configured to rotate around axis 710, while joint R8 is configured to rotate around axis 712, and joint R9 is configured to rotate around axis 714. The axes 710 and 714 are parallel with each other and orthogonal to the axis 712. Joints R7 and R9 may be configured to provide +/- 360° rotation. In other examples, joints R7 and R9 may provide +/- 90°, +/- 180° rotation or +/- 270° rotation around the respective axes 710 and 714. Joint R8 may provide +/- 90° rotation around the axis 712. In some examples, joint R8 may only be set at + 90°, 0°, and -90°.)
Consider Claim 12.
Faraji teaches:
12. (Original) The method of claim 11, wherein the line connecting the radiation source (430, 520) and the detector (440, 530) at the first candidate position and the line connecting the radiation source (430, 520) and the detector (440, 530) at the auxiliary position are at an angle with a line vertical to a plane of the calibration target (510, 700, 810, 900). (Faraji: C. Robotic Arm and Stereoscopic Camera Calibration Embodiment [0112] As discussed above, the example stereoscopic visualization camera 106 is configured to provide high-resolution stereoscopic video images of a target surgical site at different magnifications. As part of the stereoscopic robotic platform 302, the stereoscopic visualization camera 106 operates in connection with the robotic arm 114 and/or the coupling plate 306 for precise and clear changes to image focus, working distance, magnification, etc. To accomplish the image acquisition flexibility, the stereoscopic robotic platform 302 is configured to operate one or more calibration, initialization, and/or reset routines. In some embodiments, the stereoscopic visualization camera 106, the robotic arm 114, the coupling plate 306, or more generally, the stereoscopic robotic platform 302 is calibrated during manufacture and/or after installation. Calibration of the camera 106 with the robotic arm 114 provides positioning information of the camera 106 relative to the robotic arm 114 and operator space. After power-up of the stereoscopic robotic platform 302, in some embodiments, the camera 106 and/or the robotic arm 114 is configured to perform further calibration/initialization to measure and verify a location and orientation of the camera 106 at that time. [0113] In another example a zoom lens axial position along a linear encoder for the zoom lens assembly 716 is mapped in a LUT to the magnification level. For each of these examples, the example processor 1102 is configured to determine the proper level of an encoder characteristic, adjust, and verify that the characteristic provides the specified or desired working distance and/or magnification. In some embodiments, LUTs may be compound, where multiple performance characteristics are mapped to multiple platform 516 attributes for overall control of all relevant aspects of the camera 106, the robotic arm 114, and/or the coupling plate 306.)
Consider Claim 13.
Faraji teaches:
13. (Original) The method of claim 12, wherein the first magnification is determined based further on the angle. (Faraji: [0050] The example coupling plate 306 may include a sensor 115 configured to detect forces and/or torques imparted by an operator for moving the camera 106, as previously described in relation to FIGS. 1 and 2 (e.g., force/torque sensor 115 in FIGS. 1 and 2 ). The user may position and/or orient the camera 106 with assistance from the robotic arm 114. The sensor 115 detects a force vector or torque angle provided by the operator. The example platform 302 disclosed herein uses the sensed force/torque to determine which joints of the robotic arm 114 should be rotated (and how quickly the joints should be rotated) to provide assisted movement of the camera 106 that corresponds to the forces/torques provided by the operator. The sensor 115 may be located at an interface between the coupling plate 306 and the camera 106 for detecting the forces and/or torques imparted by an operator via the mechanical arms 304 [0131] Frame or joint R7 represents the pitch joint of the coupling plate 306, which, can change between 0° and 90°. Frame or joint R8 represents the yaw joint of the coupling plate 306, and can change between -90°, 0°, and 90°, depending on the yaw configuration. Joints R7 to R9 of the coupling plate may include a voltage source and a potentiometer. The connector 3450 and/or the coupler controller 1130 of the robotic arm 114 may include an I/O tool-tip connector that is configured to receive a voltage output from the potentiometer. The processor 1102 and/or the robotic arm controller 1106 are configured to receive the output voltage and correspondingly determine pitch and yaw angles of the coupling plate 306. The processor 1102 and/or the robotic arm controller 1106 combines the pitch and yaw information of the coupling plate with sensor output data from joints R1 to R6 of the robotic arm 114 to calculate position of the frames R1 to R10 to determine the three-dimensional position of the robotic arm 114, the coupling plate 306, and/or the camera 106.)
Consider Claim 14.
Faraji teaches:
14. (Currently Amended) The method of claim 10, wherein the first coordinates and the second coordinates are in a same plane parallel to a plane of the calibration target (510, 700, 810, 900). (Faraji: [0124] In other embodiments where the mechanical features do not exist, the example processor 1102 and/or the robotic arm controller 1106 are configured to perform a routine to accurately determine a spatial relationship between the camera 106 and the robotic arm 114 in robot space. The processor 1102 and/or the robotic arm controller 1106 move the stereoscopic visualization camera 106 to a start position, which may include a stow position, a re-orientation position, or a surgical position. The stereoscopic visualization camera 106 then moves the camera from the start position to a position that approximately visualizes a calibration target located on the stationary base 404 of the robotic arm 114. The calibration target may be located, for example, at a convenient area of the cart 120 in a position within the motion sphere of the robotic arm 114. Some examples of the calibration target include, for example, small spheres or other uniquely recognizable objects that can be located relative to each other (in two-dimensional or stereoscopic images) in a unique, known orientation. The coordinates of the spheres are fixed and known with respect to the cart 120 and stationary base 404, and are hence known in robot space. The processor 1102 and/or the robotic arm controller 1106 are configured to store the coordinates to, for example, the memory 1120. [0125] During the calibration, the processor 1102 and/or the robotic arm controller 1106 receive view vector data 1207 regarding working distance, magnification, stereoscopic optical axis, and/or IPD. The stereoscopic visualization camera 106 is set to visualize the spheres at the calibration target simultaneously and determine their position through the use of parallax in the stereoscopic image. The processor 1102 and/or the robotic arm controller 1106 records the positions of the spheres in an initial coordinate system, for example, X, Y, and Z with respect to a fiducial in the camera 106 (i.e. “camera space). The X,Y,Z position may correspond to an origin location, and be defined in a file or LUT as being the origin or having other known coordinate values. The processor 1102 and/or the robotic arm controller 1106 also use output data from joint sensors to determine position and orientation of the joints and links in the robotic arm 114. The processor 1102 and/or the robotic arm controller 1106 also receive position information to determine a position and orientation of the coupling plate 306. Together, the position and orientation of the robotic arm 114 and the coupling plate 306 enable the processor 1102 and/or the robotic arm controller 1106 to determine a pose of the camera 106. The processor 1102 and/or the robotic arm controller 1106 are configured to perform a coordinate transformation between the camera space and robot space based on the positions of the spheres of the calibration target as recorded by the camera, and as the positions of the robotic arm 114 and/or coupling plate 306. The processor 1102 and/or the robotic arm controller 1106 may store the coordinate transformation to the LUT 1203, a different LUT for the robotic arm 114, and/or one or more calibration registers.)
Consider Claim 15.
Faraji teaches:
15. (Currently Amended) The method of claim 10, wherein the first coordinates and/or the second coordinates are two-dimensional (2D) coordinates related to a plane of the calibration target (510, 700, 810, 900). (Faraji: [0126] In some embodiments, the camera 106 is moved to record images of multiple calibration targets located either on a cart 120, a ceiling, a wall, and/or within a surgical area. Each of the calibration targets may have a unique orientation that enables it physical X,Y, Z location to be identified. The processor 1102 and/or the robotic arm controller 1106 perform additional coordinate transformations for each of the calibration targets and store the transformations to one or more LUTs and/or registers. [0127] In other embodiments, the processor 1102 and/or the robotic arm controller 1106 may use alternative methods to calibrate the camera 106 to robot space. In this context, “calibration” is taken to mean “registration”, where the processor 1102 and/or the robotic arm controller 1106 are configured to calculate registration over a wide space in which the registration may vary. For example, a system can be used where a separate stereoscopic camera is used to observe and locate calibration targets on the cart 120 as well as similar calibration targets which are installed on the camera 106 and/or on a patient or surgical bed. The processor 1102 and/or the robotic arm controller 1106 are configured to model and track the camera 106, which is modeled and tracked as a surgical instrument with a view vector and working distance. The view vector and working distance define parameters for accurately visualizing a target surgical site. In these other embodiments, the other camera determines and reports location and orientation information for the coordinate frame of each such instrument in a reference frame, such as the stereoscopic camera 106. Then, using linear algebra, the poses and/or locations of instruments relative to each other are calculated by the processor 1102 and/or the robotic arm controller 1106, thereby resulting in a calibration of the camera 106 to the robot space.)
Consider Claim 16.
Faraji teaches:
16. (Currently Amended) The method of claim 3, wherein the obtaining the second position of the X-ray imaging device (110, 400) includes: acquiring a second image captured by the X-ray imaging device (110, 400) at a second candidate position, the second image including the target calibration point; determining a second magnification based on the second image; determining a third deviation between a second image position of the target calibration point (512) in the second image and an image center of the second image; determining, based on the second magnification and the third deviation, a fourth deviation between the spatial position of the target calibration point (512) and the second line; determining the second position based on the fourth deviation. (Faraji: C. Robotic Arm and Stereoscopic Camera Calibration Embodiment [0112] As discussed above, the example stereoscopic visualization camera 106 is configured to provide high-resolution stereoscopic video images of a target surgical site at different magnifications. As part of the stereoscopic robotic platform 302, the stereoscopic visualization camera 106 operates in connection with the robotic arm 114 and/or the coupling plate 306 for precise and clear changes to image focus, working distance, magnification, etc. To accomplish the image acquisition flexibility, the stereoscopic robotic platform 302 is configured to operate one or more calibration, initialization, and/or reset routines. In some embodiments, the stereoscopic visualization camera 106, the robotic arm 114, the coupling plate 306, or more generally, the stereoscopic robotic platform 302 is calibrated during manufacture and/or after installation. Calibration of the camera 106 with the robotic arm 114 provides positioning information of the camera 106 relative to the robotic arm 114 and operator space. After power-up of the stereoscopic robotic platform 302, in some embodiments, the camera 106 and/or the robotic arm 114 is configured to perform further calibration/initialization to measure and verify a location and orientation of the camera 106 at that time. [0113] In another example a zoom lens axial position along a linear encoder for the zoom lens assembly 716 is mapped in a LUT to the magnification level. For each of these examples, the example processor 1102 is configured to determine the proper level of an encoder characteristic, adjust, and verify that the characteristic provides the specified or desired working distance and/or magnification. In some embodiments, LUTs may be compound, where multiple performance characteristics are mapped to multiple platform 516 attributes for overall control of all relevant aspects of the camera 106, the robotic arm 114, and/or the coupling plate 306. [0114] The combination of a robotic arm 114 and the example stereoscopic visualization camera 106 provides highly accurate position, direction, and/or orientation information of the target view with respect to a frame of reference of the robotic arm 114. The following sections describe how the stereoscopic visualization camera 106 is calibrated to define a visual tip. After a visual tip is determined, the stereoscopic visualization camera 106 is registered to a frame of reference of the robotic arm 114 and/or the coupling plate 306. Accordingly, after calibration and registration, a stereoscopic view of a surgical site is unified with the integrated control of the stereoscopic visualization camera 106 combined with the position, direction, and orientation control of the robotic arm 114 and/or coupling plate 306. [0115] In some embodiments, the example processor 1102 of FIG. 11 is configured to integrate a registration of stereoscopic visualization camera 106, including its visual tip, precisely with a position, direction, and/or orientation calibration of the robotic arm 114 to define a unified position, direction, and/or orientation awareness of acquired stereoscopic images and all points therewithin, with respect to a prescribed coordinate frame. The example processor 1102 is configured to use intrinsic visual imaging data from the stereoscopic visualization camera 106 to coordinate or direct the physical positioning and/or orientating of the robotic arm 114 to provide visualization, as desired by an operator. In addition, such direction and coordination provided by the processor 1102 is provided to maintain preferred characteristics of the visualization such as focus, working distance, pre-defined positioning, orientating, etc.
[0116] In some embodiments, calibration of the stereoscopic visualization camera 106, the robotic arm 114, and/or the coupling plate 306 generally includes (i) determining and/or measuring inaccuracies of functional parameters of the stereoscopic robotic platform 302 that affect stereoscopic images; (ii) calibrating or adjusting, the stereoscopic robotic platform 302 to minimize the inaccuracies in the stereoscopic images at or below a desired level; (iii) verifying that the adjustments have been made within a desired level of calibration accuracy through simultaneous comparisons of the dual channels of the stereoscopic images to each other or calibration templates; and (iv) using the stereoscopic robotic platform 302 in the performance of its tasks, where a level of the accuracy of the calibration is detectable and maintained. [0117] In an alternative embodiment, the robotic arm 114 is provided with one or more fixed calibration fiducials that are used to precisely calibrate a physical relationship of the joint and links of the robotic arm 114 to one another as well as to calibrate a relationship of the visual tip of the camera 106 to the robotic arm 114 and/or an initial pose configuration. The robotic platform fixed calibration fiducials can be used to register or integrate the robotic arm 114 with an external environment, such as an operating room theater or with a patient or target space within an external environment. The fixed calibration fiducials can either include a dedicated attachment to an external portion of a body of the robotic arm 114 or a combination of known external features of the robotic arm 114, such as mounting points, joints, corners, or the like.)
Consider Claim 21.
Faraji teaches:
21. (Currently Amended) The method of claim 1, wherein the calibration target (510, 700, 810, 900) includes at least one calibration unit (600, 711, 712, 721, 910, 920, 930, 940, 950, 960), each of the at least one calibration unit (600, 711, 712, 721, 910, 920, 930, 940, 950, 960) (Faraji: [0042] In some aspects, the computer 124 may process video (or stereoscopic video) signals (e.g., an image or frame stream) from the stereoscopic visualization camera 106 for display on the display monitors 122A and 122B. [0043] FIG. 2 shows a diagram of the microsurgical environment 100 with the patient 102 in a sitting position for a posterior-approach skull base neurosurgery. In the illustrated embodiment, the stereoscopic visualization camera 106 is placed into a horizontal position to face the back of the head of the patient 102. The mechanical robotic arm 114 includes joints that enable the stereoscopic visualization camera 106 to be positioned as shown. In addition, the cart 120 includes a monitor 122C, which may be aligned with the surgeon’s natural view direction.)
21. includes a calibration part (620) and a base plate (610), a geometric center (630) of the calibration part (620) is set as the calibration point, the calibration part (620) is disposed on the base plate (610), (Faraji: [0122] After the example stereoscopic visualization camera 106 is connected to the robotic arm 114, the example processor 1102 and/or the robotic arm controller 1106 are configured to calibrate the camera and its view vector into a coordinate system originated around the stationary base 404 of the robotic arm 114 (block 1206). The coordinate system is referred to herein as “robot space” or “robotic space”. During this calibration step, known movements to the robotic arm 114 are used by the processor 1102 and/or the robotic arm controller 1106 to determine an orientation and a location of a view vector and object plane of the camera 106 during visualization of a target surgical site. [0123] In some embodiments, the mechanical features of the camera 106, the coupling plate 306, and the robotic arm 114 exist such that, when mechanically connected together, the relationship between the camera 106, the coupling plate 306, and the robotic arm 114 is uniquely determined and known. In these embodiments, the processor 1102 and/or the robotic arm controller 1106 determine the position, direction, and/or orientation of the view vector from the known mechanical geometry of the camera 106, the coupling plate 306, and the robotic arm 114. [0113] The example processor 1102 is configured to store results from the calibration (e.g., calibration data), in, for example, the memory 1150 and/or the memory 1120 of FIG. 11 . The calibration results may be stored to calibration registers and/or lookup tables (“LUTs”) in the memories 1150 and/or 1120. The stored calibration data relates or maps optical, functional, and/or performance characteristics to attributes of the camera 106, the robotic arm 114, and/or the coupling plate 306 that are adjustable, measurable, and/or verifiable by an operator or by the processor 1102. For example, a working distance actuator motor encoder position for the main objective assembly 702 (of FIG. 5 ) is mapped in a LUT to a working distance. In another example a zoom lens axial position along a linear encoder for the zoom lens assembly 716 is mapped in a LUT to the magnification level. For each of these examples, the example processor 1102 is configured to determine the proper level of an encoder characteristic, adjust, and verify that the characteristic provides the specified or desired working distance and/or magnification. In some embodiments, LUTs may be compound, where multiple performance characteristics are mapped to multiple platform 516 attributes for overall control of all relevant aspects of the camera 106, the robotic arm 114, and/or the coupling plate 306. [0114])
21. and there is an X-ray attenuation difference and/or an optical imaging difference between the calibration part (620) and the base plate (610).. (Faraji: [0041] In some embodiments, the cart 120 may include a computer 124. In these embodiments, the computer 124 may control a robotic mechanical arm connected to the stereoscopic visualization camera 106. Additionally or alternatively, the computer 124 may process raw sensor data received from force/torque sensor 115 of the robotic arm 114. As will be described further, the computer 124 may perform calibration of the force/torque sensor across multiple dimensions in real-time to resolve any non-linearity in received sensor data, in order to provide accurate force/torque measurements in real-time. The computer 124 may also be used to store raw sensor data received over time into a file (stored to a memory). Such data may be used as training data for the force/torque sensor to learn correct calibration parameters. Further, the computer 124 may also send control signals to the stereoscopic visualization camera 106 to select settings and/or perform calibration. C. Robotic Arm and Stereoscopic Camera Calibration Embodiment [0112] As discussed above, the example stereoscopic visualization camera 106 is configured to provide high-resolution stereoscopic video images of a target surgical site at different magnifications. As part of the stereoscopic robotic platform 302, the stereoscopic visualization camera 106 operates in connection with the robotic arm 114 and/or the coupling plate 306 for precise and clear changes to image focus, working distance, magnification, etc. To accomplish the image acquisition flexibility, the stereoscopic robotic platform 302 is configured to operate one or more calibration, initialization, and/or reset routines. In some embodiments, the stereoscopic visualization camera 106, the robotic arm 114, the coupling plate 306, or more generally, the stereoscopic robotic platform 302 is calibrated during manufacture and/or after installation. Calibration of the camera 106 with the robotic arm 114 provides positioning information of the camera 106 relative to the robotic arm 114 and operator space. After power-up of the stereoscopic robotic platform 302, in some embodiments, the camera 106 and/or the robotic arm 114 is configured to perform further calibration/initialization to measure and verify a location and orientation of the camera 106 at that time.)
Consider Claim 27.
Faraji teaches:
27. (Currently Amended) The method of claim 21, wherein the calibration target (510, 700, 810, 900) includes a support component, the support component includes a support plane, and the at least one calibration unit (600, 711, 712, 721, 910, 920, 930, 940, 950, 960) is disposed on the support plane. (Faraji: [0064] Joint R1 includes a base joint that is mechanically coupled to a flange 402, which is secured to a stationary structure 404. The flange 402 may include any type of mechanical connector. The stationary structure 404 may include, for example, the cart 120 of FIG. 3 , a wall, a ceiling, a table, etc. The joint R1 is configured to rotate around a first axis 410, which may include the z-axis. [0065] Joint R1 is connected to joint R2 via a link 430. The example link 430 includes a cylinder or other tubular structure configured to provide structural support for the downstream sections of the robotic arm 114. The link 430 is configured to provide a rotational secure connection with joint R2 to enable joint R2 to rotate while the link 430 is held in place by its connection to the joint R1. Joint R2 may include, for example, a shoulder joint configured to rotate around an axis 412. The example axis 412 is configured to be perpendicular (or substantially perpendicular) to axis 410. The axis 412 is configured to be within an x-y plane given the rotation of the joint R1 around the z-axis.)
Consider Claim 28.
Faraji teaches:
28. (Original) The method of claim 27, wherein the support component is foldable. (Faraji: [0035] FIGS. 1 and 2 show diagrams of a surgical environment including one or more robotic arms having a force/torque sensor, according to example embodiments of the present disclosure. The surgical environment may include a patient 102 that is subjected to or intended to be subjected to a surgical procedure, surgical tools 104, surgical camera (e.g., a stereoscopic visualization camera 106), robotic controls (e.g., a foot pedals 108), at least one robotic arm 114, a surgeon 112, and a surgeon assistant 118. In at least one embodiment, the surgical camera may comprise a stereoscopic visualization camera 106 used within a microsurgical environment 100, according to example embodiments of the present disclosure. The small footprint and maneuverability of the stereoscopic visualization camera (especially when used in conjunction with a multiple-degree of freedom arm) enables flexible positioning with respect to a patient 102. A portion of the patient 102 in view of the stereoscopic visualization camera 106 includes a target site 110. A surgeon 112 can position the stereoscopic visualization camera 106 in virtually any orientation while leaving more than sufficient surgical space above the patient 102 (lying in the supine position). The stereoscopic visualization camera 106 accordingly is minimally intrusive (or not intrusive) to enable the surgeon 112 to perform a life-altering microsurgical procedure without distraction or hindrance. [0064] Joint R1 includes a base joint that is mechanically coupled to a flange 402, which is secured to a stationary structure 404. The flange 402 may include any type of mechanical connector. The stationary structure 404 may include, for example, the cart 120 of FIG. 3 , a wall, a ceiling, a table, etc. The joint R1 is configured to rotate around a first axis 410, which may include the z-axis.)
Consider Claim 29.
Faraji teaches:
29. (Currently Amended) The method of claim 27, wherein the calibration target (510, 700, 810, 900) further includes a fixing component configured to fix the calibration target to a scanning bed of the X-ray imaging device, and one end of the fixing component is connected to the support component. (Faraji: [0035] FIGS. 1 and 2 show diagrams of a surgical environment including one or more robotic arms having a force/torque sensor, according to example embodiments of the present disclosure. The surgical environment may include a patient 102 that is subjected to or intended to be subjected to a surgical procedure, surgical tools 104, surgical camera (e.g., a stereoscopic visualization camera 106), robotic controls (e.g., a foot pedals 108), at least one robotic arm 114, a surgeon 112, and a surgeon assistant 118. In at least one embodiment, the surgical camera may comprise a stereoscopic visualization camera 106 used within a microsurgical environment 100, according to example embodiments of the present disclosure. The small footprint and maneuverability of the stereoscopic visualization camera (especially when used in conjunction with a multiple-degree of freedom arm) enables flexible positioning with respect to a patient 102. A portion of the patient 102 in view of the stereoscopic visualization camera 106 includes a target site 110. A surgeon 112 can position the stereoscopic visualization camera 106 in virtually any orientation while leaving more than sufficient surgical space above the patient 102 (lying in the supine position). The stereoscopic visualization camera 106 accordingly is minimally intrusive (or not intrusive) to enable the surgeon 112 to perform a life-altering microsurgical procedure without distraction or hindrance. [0036] In FIG. 1 , the stereoscopic visualization camera 106 is connected to a mechanical arm 114 via mounting bracket 116. The robotic arm 114 may include one or more rotational or extendable joints with electromechanical brakes to facilitate easy repositioning of the stereoscopic visualization camera 106. Furthermore, at least one region of the robotic arm may be equipped with or affixed to a force/torque sensor 115. As will be discussed further herein, the force/torque sensor may comprise a 6-DOF device capable of measuring a plurality of forces and torques applied to and/or perceived at the location of the robotic arm. [0064] Joint R1 includes a base joint that is mechanically coupled to a flange 402, which is secured to a stationary structure 404. The flange 402 may include any type of mechanical connector. The stationary structure 404 may include, for example, the cart 120 of FIG. 3 , a wall, a ceiling, a table, etc. The joint R1 is configured to rotate around a first axis 410, which may include the z-axis.)
CANCELLED CLAIMS
2. (Cancelled)
7. (Cancelled)
17-20. (Cancelled)
22-26. (Cancelled)
30. (Cancelled)
32-40. (Cancelled)
42-43. (Cancelled)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claims 1 and 41 are further rejected under 35 U.S.C. 103 as being unpatentable over Faraji et al. (US PGPub US 20230110248 filed September 23, 2022 with priority to September 24, 2021), in view of Hever et al. (US PGPub US US20210174117A1), hereby referred to as “Hever”.
Consider Claims 1 and 41.
Faraji teaches:
1. (Currently Amended) A camera calibration method for X-ray imaging implemented on a computing device having one or more processors and one or more storage devices, the method comprising: / 41. (Currently Amended) A system, comprising: at least one storage medium including a set of instructions; at least one processor in communication with the at least one storage medium, wherein when executing the set of instructions, the at least one processor is directed to cause the system to perform operations including: (Faraji: abstract, New and innovative systems and methods for calibrating and correcting sensors associated with a collaborative robot are disclosed. An example system comprises: at least one robotic arm; a sensor affixed to a location on the robotic arm, wherein the sensor measures force and torque across six degrees of freedom (6DOF); a processor; and memory. The system may receive, from the sensor, sensor input in real-time that indicate a measured force or torque. The system may generate, in real-time, sensor corrections that correspond to offset, linear, and non-linear deviations of the measured force in each sensor axis. The sensor corrections may correspond to offset, linear, and non-linear cross-coupling of the measured force between two or more sensor axes. The sensor corrections may be determined by applying offset, linear, and non nonlinear corrections to each degree of freedom (DOF) from every other DOF. [0034] Section I. . Exemplary Surgical Environment for the Force/Torque Sensor [0035] FIGS. 1 and 2 show diagrams of a surgical environment including one or more robotic arms having a force/torque sensor, according to example embodiments of the present disclosure. The surgical environment may include a patient 102 that is subjected to or intended to be subjected to a surgical procedure, surgical tools 104, surgical camera (e.g., a stereoscopic visualization camera 106), robotic controls (e.g., a foot pedals 108), at least one robotic arm 114, a surgeon 112, and a surgeon assistant 118. In at least one embodiment, the surgical camera may comprise a stereoscopic visualization camera 106 used within a microsurgical environment 100, according to example embodiments of the present disclosure. The small footprint and maneuverability of the stereoscopic visualization camera (especially when used in conjunction with a multiple-degree of freedom arm) enables flexible positioning with respect to a patient 102. A portion of the patient 102 in view of the stereoscopic visualization camera 106 includes a target site 110. A surgeon 112 can position the stereoscopic visualization camera 106 in virtually any orientation while leaving more than sufficient surgical space above the patient 102 (lying in the supine position). The stereoscopic visualization camera 106 accordingly is minimally intrusive (or not intrusive) to enable the surgeon 112 to perform a life-altering microsurgical procedure without distraction or hindrance.)
1. acquiring at least one image taken by a camera to be calibrated (160), wherein the at least one image includes a calibration target (510, 700, 810, 900), and the calibration target (510, 700, 810, 900) includes at least one calibration point; / 41. acquiring at least one image taken by a camera to be calibrated, wherein the at least one image includes a calibration target (510, 700, 810, 900), and the calibration target (510, 700, 810, 900) includes at least one calibration point; (Faraji: [0042] In some aspects, the computer 124 may process video (or stereoscopic video) signals (e.g., an image or frame stream) from the stereoscopic visualization camera 106 for display on the display monitors 122A and 122B. [0043] FIG. 2 shows a diagram of the microsurgical environment 100 with the patient 102 in a sitting position for a posterior-approach skull base neurosurgery. In the illustrated embodiment, the stereoscopic visualization camera 106 is placed into a horizontal position to face the back of the head of the patient 102. The mechanical robotic arm 114 includes joints that enable the stereoscopic visualization camera 106 to be positioned as shown. In addition, the cart 120 includes a monitor 122C, which may be aligned with the surgeon’s natural view direction.)
1. selecting any calibration point of the at least one calibration point as a target calibration point; / 41. selecting any calibration point of the at least one calibration point as a target calibration point; (Faraji: [0041] In some embodiments, the cart 120 may include a computer 124. In these embodiments, the computer 124 may control a robotic mechanical arm connected to the stereoscopic visualization camera 106. Additionally or alternatively, the computer 124 may process raw sensor data received from force/torque sensor 115 of the robotic arm 114. As will be described further, the computer 124 may perform calibration of the force/torque sensor across multiple dimensions in real-time to resolve any non-linearity in received sensor data, in order to provide accurate force/torque measurements in real-time. The computer 124 may also be used to store raw sensor data received over time into a file (stored to a memory). Such data may be used as training data for the force/torque sensor to learn correct calibration parameters. Further, the computer 124 may also send control signals to the stereoscopic visualization camera 106 to select settings and/or perform calibration. C. Robotic Arm and Stereoscopic Camera Calibration Embodiment [0112] As discussed above, the example stereoscopic visualization camera 106 is configured to provide high-resolution stereoscopic video images of a target surgical site at different magnifications. As part of the stereoscopic robotic platform 302, the stereoscopic visualization camera 106 operates in connection with the robotic arm 114 and/or the coupling plate 306 for precise and clear changes to image focus, working distance, magnification, etc. To accomplish the image acquisition flexibility, the stereoscopic robotic platform 302 is configured to operate one or more calibration, initialization, and/or reset routines. In some embodiments, the stereoscopic visualization camera 106, the robotic arm 114, the coupling plate 306, or more generally, the stereoscopic robotic platform 302 is calibrated during manufacture and/or after installation. Calibration of the camera 106 with the robotic arm 114 provides positioning information of the camera 106 relative to the robotic arm 114 and operator space. After power-up of the stereoscopic robotic platform 302, in some embodiments, the camera 106 and/or the robotic arm 114 is configured to perform further calibration/initialization to measure and verify a location and orientation of the camera 106 at that time.)
1. determining, based on the at least one image, image coordinates of the target calibration point; / 41. determining, based on the at least one image, image coordinates of the target calibration point; (Faraji: [0113] The example processor 1102 is configured to store results from the calibration (e.g., calibration data), in, for example, the memory 1150 and/or the memory 1120 of FIG. 11 . The calibration results may be stored to calibration registers and/or lookup tables (“LUTs”) in the memories 1150 and/or 1120. The stored calibration data relates or maps optical, functional, and/or performance characteristics to attributes of the camera 106, the robotic arm 114, and/or the coupling plate 306 that are adjustable, measurable, and/or verifiable by an operator or by the processor 1102. For example, a working distance actuator motor encoder position for the main objective assembly 702 (of FIG. 5 ) is mapped in a LUT to a working distance. In another example a zoom lens axial position along a linear encoder for the zoom lens assembly 716 is mapped in a LUT to the magnification level. For each of these examples, the example processor 1102 is configured to determine the proper level of an encoder characteristic, adjust, and verify that the characteristic provides the specified or desired working distance and/or magnification. In some embodiments, LUTs may be compound, where multiple performance characteristics are mapped to multiple platform 516 attributes for overall control of all relevant aspects of the camera 106, the robotic arm 114, and/or the coupling plate 306. [0114] )
1. obtaining a first position and a second position of an X-ray imaging device (110, 400) , wherein when the X-ray imaging device (110, 400) is located in the first position and the second position, / 41. obtaining a first position and a second position of an X-ray imaging device (110, 400), wherein when the X-ray imaging device (110, 400) is located in the first position and the second position, (Faraji: [0115] In some embodiments, the example processor 1102 of FIG. 11 is configured to integrate a registration of stereoscopic visualization camera 106, including its visual tip, precisely with a position, direction, and/or orientation calibration of the robotic arm 114 to define a unified position, direction, and/or orientation awareness of acquired stereoscopic images and all points therewithin, with respect to a prescribed coordinate frame. The example processor 1102 is configured to use intrinsic visual imaging data from the stereoscopic visualization camera 106 to coordinate or direct the physical positioning and/or orientating of the robotic arm 114 to provide visualization, as desired by an operator. In addition, such direction and coordination provided by the processor 1102 is provided to maintain preferred characteristics of the visualization such as focus, working distance, pre-defined positioning, orientating, etc. [0055] As will be described further, systems and methods are disclosed for calibrating the force/torque sensor 115 across multiple dimensions in real-time to resolve any non-linearity in received sensor data. The real-time calibration will assist in providing accurate force/torque measurements in real-time. [0056] In the illustrated embodiments, a first end of the robotic arm 114 may be mounted to the cart 120 while a second, opposite end of the robotic arm is mechanically connected to stereoscopic visualization camera 106 (e.g., a robot end effector). FIG. 3 shows the robotic arm 114 holding the stereoscopic visualization camera 106 in an extended position, such as positioning the stereoscopic visualization camera 106 above a surgical site while keeping the rest of the platform 302 out of the way of a surgeon. The cart 120 can be configured to securely hold the stereoscopic robotic platform 302 and can be weighted and balanced to prevent tipping under prescribed operating positions. A. Robotic Arm Embodiment [0057] FIG. 4 illustrates an embodiment of the example robotic arm 114, according to an example embodiment of the present disclosure. In some embodiments, the robotic arm 114 is similar to or comprises model UR5 from Universal Robots S/A. The exterior surfaces of the robotic arm 114 comprise aluminum and plastic materials, which are compatible for use in an operating room and easily cleaned. [0058] While the robotic arm 114 is described herein as being electromechanical, in other examples, the robotic arm 114 may be mechanical, hydraulic, or pneumatic. In some embodiments, the robotic arm 114 may have mixed actuation mechanisms, for example, using a vacuum chuck with a control valve to hold and manipulate the camera 106. Further, while the robotic arm 114 is described below as including a certain number of joints and links, it should be appreciated that the robotic arm 114 may include any number of joints, any lengths of links, and/or comprise any types of joints, or sensors.)
1. the target calibration point (512) is within an imaging field of view of the X-ray imaging device (110, 400) , / 41. the target calibration point (512) is within an imaging field of view of the X-ray imaging device (110, 400), (Faraji: [0059] As described herein, the robotic arm 114 is situated and the joints are oriented to provide an unrestricted view of an operating field while providing a 3D stereoscopic display for an operator for any surgical procedure for a patient. Movement of the robotic arm 114 in noncritical motions is provided to be fast enough for an operator to be convenient yet safe. Movement of the robotic arm 114 is controlled during surgery to be meticulous and accurate. In addition, movement of the robotic arm is controlled to be smooth and predictable through the entire range of motion required for a surgical procedure. As described herein, movement of the robotic arm 114 is controllable by remote control or via manual manipulation of the arm itself. In some embodiments, the robotic arm 114 is configured to be positionable with minimal force (e.g., via an assisted guidance feature) with just the use of, for example, a single auricular finger. [0060] In some embodiments, the robotic arm 114 may include mechanically or electronically locking brakes on the joints. The brakes may be engaged once the aim or “pose”, which is generally the location and direction, of the camera 106 after it is set by an operator. The robotic arm 114 may include a locking or unlocking switch or other input device to prevent undesired manual or accidental motion. When locked, the example robotic arm provides sufficient stability that enables the stereoscopic visualization camera 106 to provide a stable, clear image. The robotic arm 114 may additionally or alternatively include one or more dampening devices to absorb or attenuate vibrations following movement of the stereoscopic visualization camera 106 to a new pose. The dampening devices may include, for example, fluid-filled linear or rotational dampeners, rubber-based vibration isolation mounting dampeners, and/or tuned mass-spring dampeners. Alternatively, or in addition, the arm 114 may include electromechanical dampening, for example, through the use of a proportional integral derivative (“PID”) servo system.)
1. and a first line and a second line are not parallel, the first line connecting a radiation source (430, 520) and a detector (440, 530) of the X-ray imaging device (110, 400) at the first position, the second line connecting the radiation source (430, 520) and the detector (440, 530) of the X-ray imaging device (110, 400) at the second position; / 41. and a first line and a second line are not parallel, the first line connecting a radiation source and a detector (440, 530) of the X-ray imaging device (110, 400) at the first position, the second line connecting the radiation source (430, 520) and the detector (440, 530) of the X-ray imaging device (110, 400) at the second position; (Faraji: [0062] In the illustrated embodiment of FIG. 4 , the robotic arm 114 includes six joints, labeled R1, R2, R3, R4, R5, and R6. In other embodiments, the robotic arm 114 may include fewer or additional joints. Additionally, in some embodiments, at least some of the joints R1 to R6 have rotational motion capabilities of +/-360°. The rotational motion may be provided by an electromechanical subsystem that includes, for each joint, an electric motor configured to drive a mechanical rotational joint through one or more anti-backlash joint gearboxes. Each of the joints R1 to R6 may include one or more rotational sensors to detect joint position. Further, each joint may include a slip clutch and/or an electromechanical brake.
[0063] Each of the joints R1 to R6 may have an overall repeatability of motion (with the camera 106 attached) of approximately +/-⅒ of a millimeter (“mm”). The joints may be have variable rotational speeds that can be controlled between 0.5° to 180° per second. Together, this translates to camera movement between 1 mm per second to 1 meter per second. In some embodiments, the stereoscopic robotic platform 302 may have speed governors for one or more of the joints R1 to R6 that are in place during surgical procedures. Each of the joints R1 to R6 may be electrically connected to a power source and/or command line in a controller of the robotic arm 114. Wires for power and command signals may be routed internally within the joints and links. Further, one or more of the joints may include dampeners, such as o-rings for connection to links. The dampeners may, for example, reduce or absorb vibrations in the robotic arm 114, vibrations from the cart 120, and/or vibrations imparted via the stereoscopic visualization camera 106.)
1. determining, based on the first position and the second position, spatial coordinates of the target calibration point; / 41. (v) determining, based on the first position and the second position, spatial coordinates of the target calibration point; and (Faraji: 0064] Joint R1 includes a base joint that is mechanically coupled to a flange 402, which is secured to a stationary structure 404. The flange 402 may include any type of mechanical connector. The stationary structure 404 may include, for example, the cart 120 of FIG. 3 , a wall, a ceiling, a table, etc. The joint R1 is configured to rotate around a first axis 410, which may include the z-axis. [0065] Joint R1 is connected to joint R2 via a link 430. The example link 430 includes a cylinder or other tubular structure configured to provide structural support for the downstream sections of the robotic arm 114. The link 430 is configured to provide a rotational secure connection with joint R2 to enable joint R2 to rotate while the link 430 is held in place by its connection to the joint R1. Joint R2 may include, for example, a shoulder joint configured to rotate around an axis 412. The example axis 412 is configured to be perpendicular (or substantially perpendicular) to axis 410. The axis 412 is configured to be within an x-y plane given the rotation of the joint R1 around the z-axis. [0066] Joint R2 is mechanically coupled to joint R3 via link 432. The link 432 is configured to have a greater length than the link 430 and is configured to provide structural support for downstream portions of the robotic arm 114. Joint R3 may include, for example, an elbow joint. Together with joint R2, joint R3 provides extensible positioning and/or orientating of the robotic arm 114. The joint R3 is configured to rotate around an axis 414, which is perpendicular or orthogonal to the axis 410 and parallel to the axis 412. [0067] Joint R3 is connected to joint R4 via link 434, which provides structural support for downstream portions of the robotic arm 114. The example joint R4 may be, for example, a first wrist joint configured to provide rotation around axis 416, which may be orthogonal to the axes 412 and 414. Joint R4 is mechanically connected to joint R5 via link 436. Joint R5 may be a second wrist joint configured to provide rotation around an axis 418, which is orthogonal to axis 416. Joint R5 is mechanically connected to joint R6 via link 438. Joint R6 may be a third wrist joint configured to rotate around axis 420, which is orthogonal to the axis 418. Together, the wrist joints R4 to R6 provide precise flexibility in positioning the stereoscopic visualization camera 106 described herein. [0124])
1. and performing calibration on the camera to be calibrated (160) based on the image coordinates and the spatial coordinates./ 41. (vi) performing calibration on the camera to be calibrated (160) based on the image coordinates and the spatial coordinates. (Faraji: [0068] The example robotic arm 114 includes a connector 450. The example connector 3450 is connected to joint R6 via link 440. In some embodiments, the example link 440 may include a sleeve that enables joint R6 to rotate the connector 450. As discussed herein, the connector 3450 may be configured to mechanically couple to the coupling plate 306 or the stereoscopic visualization camera 106 directly when a coupling plate is not used. The connector 450 may include one or more screws to secure the robotic arm 114 to the coupling plate 304 and/or the stereoscopic visualization camera 106. [0071] FIG. 6 shows an embodiment where the robotic arm 114 is mounted to a stationary structure 404 that is a ceiling plate. The robotic arm may be suspended from the ceiling of an operating room via the flange 402 to reduce floor space clutter. The robotic arm 114, including the joints, can be positioned above and traversed from the area where surgical activity is being performed, out of the way of the surgeon and operating room staff, yet still providing functional positioning and/or orientating of the camera 106 while providing a clear view of the display monitors (e.g., display monitors 122A-E as previously described in relation to FIGS. 1-3 ). [0072] FIG. 7 shows an embodiment of the coupling plate 306. In the illustrated example, a first end 702 of the coupling plate 306 is connected to the connector 450 of the robotic arm 114. A second end 704 of the coupling plate 306 is connected to the stereoscopic visualization camera 106. The example coupling plate 306 is configured to provide additional degrees of freedom for moving the stereoscopic visualization camera 106. The coupling plate 306 also extends the maximum reach of the robotic arm 114. The coupling plate 306 may have a length between 10 cm to 100 cm. [0102] After the boundary checks are performed, the robotic arm controller 1106 uses the movement delta and the current position/orientation of each of joints R1 to R9 to determine an optimal or near optimal movement sequence for rotating one or more of the joints to cause the robotic arm 114 to move the camera 106 into the specified location. The robotic arm controller 1106 may use, for example, an optimization routine that determines a minimal amount of joint movement needed to satisfy the movement delta vector. After the amount of joint movement is determined, the example robotic arm controller 1106 is configured to send one or more messages (indicative of an amount of rotation and speed of rotation, taking into account any scale factors) to a motor controller 1124. The robotic arm controller 1106 may transmit a sequence of messages to cause the robotic arm 114 and/or coupling plate 306 to move in a defined or coordinated sequence. The sequence of messages may also cause a change in joint speed as, for example, the robotic arm 114 approaches a virtual or physical boundary.)
Even if Faraji does not specifically teach: “1. determining, based on the first position and the second position, spatial coordinates of the target calibration point; / 41. (v) determining, based on the first position and the second position, spatial coordinates of the target calibration point;”
Hever teaches:
1. (Currently Amended) A camera calibration method for imaging implemented on a computing device having one or more processors and one or more storage devices, the method comprising: / 41. (Currently Amended) A system, comprising: at least one storage medium including a set of instructions; at least one processor in communication with the at least one storage medium, wherein when executing the set of instructions, the at least one processor is directed to cause the system to perform operations including: (Hever: abstract, A system configured for verification of imaged target detection performs the following: (a) receive information indicative of at least two images of object(s). This information comprises candidate target detection region(s), indicative of possible detection of a target associated with the object(s). Images have at least partial overlap. The candidate region(s) appears at least partially in the overlap area. The images are associated with different relative positions of capturing imaging device(s) and of an imaged portion of the object(s). (b) process the information to determine whether the candidate region(s) meets a detection repetition criterion, the criterion indicative of repetition of candidate region(s) in locations of the images that are associated with a same location on a data representation of the object(s). (c) if the criterion is met, classify the candidate region(s) as verified target detection region(s). This facilitates output of an indication of the verified region(s). [0009] The system according to this aspect of the presently disclosed subject matter can include on or features (i) to (xliv) in any desired combination or permutation [0136], [0227]-[0230] In the non-limiting example of FIGS. 1, 7 and 8, the imaging devices 140 etc. and the illumination devices 130 etc., as well as monitoring camera 136, are shown as separate from imaging system 705 and as operatively coupled to it. In other examples, these devices are part of the imaging system 705, e.g. as part of image capture system 843. Similarly, in some examples, monitoring camera 136 can be part of triggering system 820 and/or brightness compensation system 830, within imaging system 705.)
1. acquiring at least one image taken by a camera to be calibrated (160), wherein the at least one image includes a calibration target (510, 700, 810, 900), and the calibration target (510, 700, 810, 900) includes at least one calibration point; / 41. acquiring at least one image taken by a camera to be calibrated, wherein the at least one image includes a calibration target (510, 700, 810, 900), and the calibration target (510, 700, 810, 900) includes at least one calibration point; (Haver: [0137] The example system 100 illustrated in FIG. 1 is a computer-based vehicle inspection system for automatically inspecting a vehicle for damage or other anomalies. In some examples, system 100 is also referred to herein as image acquisition system 100. System 100 comprises a computerized system 102 for object inspection, and target detection and verification. One non-limiting example of system 102 is for vehicle damage inspection, detection and verification. Examples of the computerized system are disclosed further herein with reference to FIGS. 7, 8 and 12. Example system 100 includes a set of image acquisition devices 140, 144, 142, 146, 148 (also referred to herein as imaging devices). In some examples, these devices are also referred to herein as capturing imaging devices, as distinguished from monitoring imaging devices 136 (disclosed below). System 100 can be configured to obtain, from the set of imaging devices such as 140, a plurality of sets of images capturing/acquiring a plurality of segments or portions of a vehicle, e.g. images of the vehicle surface. The set of imaging devices such as 140 can be operatively connected to system 102 and the captured images can be transmitted to system 102 via wired or wireless communication 104. [0138] The imaging acquisition devices used herein can refer to any kind of imaging devices or general-purpose devices equipped with image acquisition functionalities that can be used to capture vehicle images at a certain resolution and frequency, such as a digital camera with image and/or video recording functionalities. The set of imaging devices 140, 144, 142, 146, 148 can comprise multiple camera devices located (mounted or otherwise situated) on at least one side of a vehicle 105 (e.g. on at least one side of an inspection passage/lane through which the vehicle 105 passes), and they may be configured to capture a plurality of segments or portions of a vehicle.)
1. selecting any calibration point of the at least one calibration point as a target calibration point; / 41. selecting any calibration point of the at least one calibration point as a target calibration point; (Haver: [0009] i. transform the coordinates of a first candidate detection region, associated with a first image frame captured by a first imaging device, where the coordinates of the first candidate detection region are in the coordinate system of a first imaging device, using a known calibration of the two devices and the geometric map, to the coordinate system of a second imaging device, ii. compare the transformed coordinates of the first candidate detection region to the coordinates of a second candidate detection region associated with a second image frame captured by the second imaging device. iii. determine whether coordinates of a second candidate detection region match the transformed coordinates of the first candidate detection region, within a defined tolerance. [0138] The imaging acquisition devices used herein can refer to any kind of imaging devices or general-purpose devices equipped with image acquisition functionalities that can be used to capture vehicle images at a certain resolution and frequency, such as a digital camera with image and/or video recording functionalities. The set of imaging devices 140, 144, 142, 146, 148 can comprise multiple camera devices located (mounted or otherwise situated) on at least one side of a vehicle 105 (e.g. on at least one side of an inspection passage/lane through which the vehicle 105 passes), and they may be configured to capture a plurality of segments or portions of a vehicle. In some examples, some or all of the cameras are configured to capture still images. In other examples, some or all of the cameras are configured to capture image frames of a video recording. In other examples, some or all of the cameras are configured to capture both still and video images. In some embodiments, there are camera devices located on both sides of the vehicle, such that images of both sides of the vehicle can be simultaneously acquired and processed. Note also that the presently disclosed subject matter discloses the non-limiting example of two-dimensional (2-D) image frames, such as those shown in FIGS. 3, 4 and 5. However, the disclosure applies as well to one-dimensional image captures, such as those captured by a line scanner 140, 144.)
1. determining, based on the at least one image, image coordinates of the target calibration point; / 41. determining, based on the at least one image, image coordinates of the target calibration point; (Haver: [0009] i. transform the coordinates of a first candidate detection region, associated with a first image frame captured by a first imaging device, where the coordinates of the first candidate detection region are in the coordinate system of a first imaging device, using a known calibration of the two devices and the geometric map, to the coordinate system of a second imaging device, ii. compare the transformed coordinates of the first candidate detection region to the coordinates of a second candidate detection region associated with a second image frame captured by the second imaging device. iii. determine whether coordinates of a second candidate detection region match the transformed coordinates of the first candidate detection region, within a defined tolerance.)
1. obtaining a first position and a second position of an imaging device, wherein when the imaging device (110, 400) is located in the first position and the second position, / 41. obtaining a first position and a second position of an imaging device (110, 400), wherein when the imaging device (110, 400) is located in the first position and the second position, (Haver: [0009] i. transform the coordinates of a first candidate detection region, associated with a first image frame captured by a first imaging device, where the coordinates of the first candidate detection region are in the coordinate system of a first imaging device, using a known calibration of the two devices and the geometric map, to the coordinate system of a second imaging device, ii. compare the transformed coordinates of the first candidate detection region to the coordinates of a second candidate detection region associated with a second image frame captured by the second imaging device. iii. determine whether coordinates of a second candidate detection region match the transformed coordinates of the first candidate detection region, within a defined tolerance.)
1. the target calibration point (512) is within an imaging field of view of the imaging device (110, 400) , / 41. the target calibration point (512) is within an imaging field of view of the imaging device (110, 400), (Haver: [0009] i. transform the coordinates of a first candidate detection region, associated with a first image frame captured by a first imaging device, where the coordinates of the first candidate detection region are in the coordinate system of a first imaging device, using a known calibration of the two devices and the geometric map, to the coordinate system of a second imaging device, ii. compare the transformed coordinates of the first candidate detection region to the coordinates of a second candidate detection region associated with a second image frame captured by the second imaging device. iii. determine whether coordinates of a second candidate detection region match the transformed coordinates of the first candidate detection region, within a defined tolerance. (xvi) in the steps (b) and (c), the temporal detection repetition criterion constitutes the at least one detection repetition criterion, the method further comprising: (d) setting the at least one verified target detection region to constitute the at least one candidate target detection region; and (e) repeating said steps (b) and (c), wherein the spatial detection repetition criterion constitutes the at least one detection repetition criterion, thereby generating a doubly-verified target detection region.
(xvii) the information indicative of the at least two images comprises the at least two images.
(xviii) the information indicative of at the least two images comprises an indication that at least one image of the at least two images is not associated with any candidate target detection region of the at least one candidate target detection region.
(xix) the method further comprising performing, after said step (c), the following:
(f) in response to classifying the at least one candidate target detection region as at least one verified target detection region, outputting the indication of the at least one verified target detection region.)
1. determining, based on the first position and the second position, spatial coordinates of the target calibration point; / 41. (v) determining, based on the first position and the second position, spatial coordinates of the target calibration point; and (Haver: [0272] In some examples, block 1034 utilizes motion maps 712 to determine the positions of the portion of the vehicle(s) that is associated with the image frames being processed. The motion maps may be used as well, in some examples, when determining overlap areas 610, 620 (further disclosed with reference to FIG. 6), using for example overlap determination module 742. [0273] The following is one non-limiting example of a method for determining 1034 the position 310, 410 within each of the other image frames 338, 330, 458, 450 of the images being used in the verification, where the candidate damage region 360, 463, 515, 550 (associated with a first image frame 334, 454, 562 of the images being used in the verification) should appear, while utilizing a motion map 712, within the process of block 1034, carried out for example by region position determination module 744: i. determine the positions of the portion of the data representation of the vehicle 105 that is associated with the images 334, 338, 330, 454, 458, 450 being compared, and determine the different relative positions 230, 234, 238, 250, 254, 258 of the capturing imaging devices associated with the image frames and the imaged portion of the object associated with the image frames. This determination, in some examples, is based, at least, on information indicative of the capturing imaging device 140 associated with the each image of two or more images 334, 338, 330, 454, 458, 450 (e.g. the camera IDs associated with each image), on the positions and the orientations of the capturing imaging device(s) 140, and on the motion map(s) 712; ii. determine a position (e.g. 310, 410) of the candidate damage region 360, 463, 515, 550 with respect to a position of that portion of the data representation of the vehicle which is associated with the first image frame 334, 454, which includes the candidate damage region. This position of the candidate damage region 360, 463 is also referred to herein as the first relative position 310, 410;
iii. determine one or more expected relative positions (e.g. 310, 410) of one or more expected corresponding candidate damage regions 327, 468, 460 on other image frames 338, 330, 458, 450 (not the first image 334, 454). This determination may be based at least on the positions of that portion of the data representation of the vehicle 105 that is associated with the images 334, 338, 330, 454, 458, 450 being processed, on the different relative positions 230, 234, 238, 250, 254, 258 of the capturing imaging device(s) associated with the images and the imaged portion of the object associated with the images, and on the first relative position (e.g. 310, 410) of the candidate damage region 360, 463, 515, 550. In some examples this may make use of the motion map(s) 712; and iv. determining whether the expected corresponding candidate damage region(s) appears on some or all of the other images at the expected relative position(s) (e.g. 310, 410). For example, at 468, 460 the expected corresponding candidate damage region appears, while at 327 the expected corresponding candidate damage region does not appear.)
1. and performing calibration on the camera to be calibrated (160) based on the image coordinates and the spatial coordinates./ 41. (vi) performing calibration on the camera to be calibrated (160) based on the image coordinates and the spatial coordinates. (Haver: [0263] Note that in some examples, the use of geometric maps requires at least two imaging devices that are calibrated together, and that there is a known transformation from the coordinates of e.g. camera 142 to the coordinates of camera 140. [0282] In response to determination that all candidate damage regions 515, 550 have been process and evaluated, the process may continue to block 1050. In 1050, a first temporally-verified damage region 463 is selected, e.g. by processor 740, to constitute a candidate damage region 463 for further verification. [0283] In block 1052, it is determined whether, in image frames 474, 572, 574, 576, captured at the same time by other cameras 142, 146, corresponding regions 535, 540 show the same damage appearance 425 as appears in temporally-verified candidate damage region 474, 515, that is whether there is repetition of the particular candidate damage region 463, 515 in locations of these images 474, 572, 574, 576 that are associated with the same location on a data representation of the vehicle 105 as does the candidate damage region 463, 515. Such a determination 1052 may consider spatially-varying image frames, taken by other cameras 142, 146. This block may include use of the overlap determination module 742, to determine overlap areas 610, 620, within which repetitions of the candidate damage region 463, 515 will be sought. Overlap areas are further disclosed with reference to FIG. 6.) This block may also use region position determination module 744, to determine the position (e.g. 410) within each of the other image frames 474, 572, 574, 576 where the candidate damage region 463, 515 should appear. The block may also use damage region verification module 746, to perform statistics on the number of spatial repetitions found for the first temporally-verified candidate damage region 515. Examples of such region matches, choice of other image frames to process, and spatial detection repetition criteria, are disclosed with reference to FIG. 5. Example spatial detection repetition criteria are disclosed also with reference to images 334, 364, images 454, 474, and images 562, 572, 574, 576, in FIGS. 3-5. i. In some examples, the positions of the portion of the vehicle associated with the processed image frames, are determined utilizing geometric map(s) 714, 718. For example, assume that cameras A and B (e.g. 140, 142) capture at the same moment image frames Fa (a first image frame) and Fb (a second image frame), respectively. Assume that in each of these two frames there is a single target detection in the overlap area, denoted Da and Db respectively. In some examples, the method may proceed as follows: transform the coordinates of the first candidate detection region Da, which are in the coordinate system first imaging device A, to the coordinate system of a second imaging device, camera B, using the known calibration and the geometric map model. ii. compare the transformed coordinates of first candidate detection region Da to the coordinates of the second candidate detection region Db on the object of interest. [0286] In block 1054, it is determined whether the relevant spatial detection repetition criteria are met. This may, in some examples, be carried out by damage region verification module 746. A non-limiting example criterion is that the candidate damage region 463, 515 also appears in the relevant location (e.g. 410) in image frames 474, 572, 574, 576 of two nearby-positioned cameras 142, 146, that were captured at the same time 206 that the image 454, 562 containing the candidate region 463, 515 was captured. In the example above, the system may determine whether the two sets of coordinates Da (transformed coordinates of the first candidate detection region) and Db (coordinates of the second candidate detection region) match, within a defined tolerance.)
It would have been obvious before the effective filing date of the claimed invention was made to one of ordinary skill in the art to modify the method and system of Faraji for stereoscopic visualization camera with Hever’s overall imaging device calibration algorithm for imaged target detection. The determination of obviousness is predicated upon the following findings: Both are in the same field of image analysis and processing and lend themselves for combination. One skilled in the art would have been motivated to modify Faraji in order to improve the accuracy of the target detection by more accurately ensuring the calibration of the imaging devices. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Faraji, while the teaching of Hever continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of improving the overall accuracy of the target detection with an improved camera calibration algorithm. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Conclusion
The prior art made of record in form PTO-892 and not relied upon is considered pertinent to applicant's disclosure.
PNG
media_image1.png
251
892
media_image1.png
Greyscale
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAHMINA ANSARI whose telephone number is 571-270-3379. The examiner can normally be reached on IFP Flex - Monday through Friday 9 to 5.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’NEAL MISTRY can be reached on 313-446-4912. The fax phone numbers for the organization where this application or proceeding is assigned are 571-273-8300 for regular communications and 571-273-8300 for After Final communications. TC 2600’s customer service number is 571-272-2600.
Any inquiry of a general nature or relating to the status of this application or proceeding should be directed to the receptionist whose telephone number is 571-272-2600.
2674
/Tahmina Ansari/
September 29, 2026
/TAHMINA N ANSARI/Primary Examiner, Art Unit 2674
