DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant claims the benefit of US Provisional Application No. 63/119,549, filed 11/30/2020. Claims 1-10 and 32-41 have been afforded the benefit of this filing date.
Response to Arguments
Applicant’s arguments, see pg. 6-8, filed 05/27/2026, with respect to the rejection(s) of claim(s) 1 and 32 under 35 USC 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Seeber (US 20180228343 A1) in view of Itkowitz et al. (US 20180064499 A1, hereinafter "Itkowitz"), using a different interpretation of Itkowitz than was previously relied upon.
Regarding argument I (pg. 7), applicant argues that Seeber does not teach an icon that “appears affixed to the surface of the indicator body”. However, Seeber is not relied upon to teach this limitation; it is taught by Itkowitz (see “Claim Rejections - 35 USC § 103” section).
Regarding argument II (pg. 7-8), applicant argues that Itkowitz does not teach that “a position of the icon remains unchanged relative to the indicator body as the position of the instrument changes relative to the field of view”. However, Itkowitz does teach this limitation, as the text on the badge’s central portion 220 is now considered to correspond to the claimed “icon” as opposed to the text on the orbiting portion 222 (see “Claim Rejections - 35 USC § 103” section for more detail).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-7, 9-10, 32-38, and 40-41 is/are rejected under 35 U.S.C. 103 as being unpatentable over Seeber (US 20180228343 A1) in view of Itkowitz et al. (US 20180064499 A1, hereinafter "Itkowitz").
Regarding claim 1, Seeber teaches: A medical system ([0003] “The device is used in particular in a system for robot-assisted surgery, in particular for a telerobot-assisted procedure.”) comprising:
a display system ([0054] “FIG. 4 shows an image 1200 of the field of view captured by the endoscope 312, which image is displayed on the display unit 44.”); and
a control system, wherein the control system includes a processing unit including one or more processors ([0002] “At least one control unit processes the image data and outputs images corresponding to the image data on at least one display unit.”), and wherein the processing unit is configured to:
display, on the display system, an image of a field of view of a surgical environment, wherein the image is generated by an imaging component ([0054] “FIG. 4 shows an image 1200 of the field of view captured by the endoscope 312, which image is displayed on the display unit 44. In the displayed image 1200, the visible tissue structures 1210 in the surgical area 30 as well as further pieces of information are displayed.”);
generate a three-dimensional synthetic indicator for a position of an instrument outside of the field of view of the surgical environment (fig. 4, indicator 1250 points toward a position 1240 on a surgical instrument; [0054] “The end effector 310 is located outside the field of view of the endoscope 312 and is thus not illustrated in the image 1200 displayed by the display unit 44. A point of rotation 1240 at the front end of the bendable instrument shaft 309 is used as a reference point of the end effector 310 in the previous embodiment. In other embodiments also other points of the end effector can be used as reference points. Between the point of intersection 303b and the reference point 1240 the distance vector in the three-dimensional space, preferably in the three-dimensional device coordinate system X, Y, Z of the manipulator 12 is determined by the control unit 40. The value of the vector is optionally inserted into the displayed image 1200 as distance information 1260. The illustrated arrow 1250 extends in a section of the vector. The vector is defined by the line segment between the point of intersection 303b and the reference point 1240.”
[0057] teaches that the indicator can be three-dimensional: “If the endoscope 312 is a stereo endoscope, the images 1200, 1300 and 1400 may also be captured and output as three-dimensional images. The arrows 1250 and 1450 can then be inserted with a corresponding orientation along the course of the determined vector into the three-dimensional image 1200, 1300, 1400 such that also the illustration of the arrows 1250 and 1450 is made in a three-dimensional manner and thus the surgeon obtains a real three-dimensional direction information.”), wherein the three-dimensional synthetic indicator includes an indicator body and an icon (fig. 4, arrow body 1250 can be considered the “indicator body”, and numerical readout 1260 can be considered the “icon”); and
display the three-dimensional synthetic indicator with the image of the field of view of the surgical environment (displayed image 1200 includes tissue 1210 in the surgical area as well as indicator 1250; [0054] provides more detail),
Seeber does not explicitly teach: wherein the indicator body has a surface and the icon appears affixed to the surface of the indicator body; or
wherein an orientation of the icon changes relative to the indicator body as the position of the instrument changes relative to the field of view, and wherein a position of the icon remains unchanged relative to the indicator body as the position of the instrument changes relative to the field of view.
Itkowitz teaches an indicator “badge” for a surgical instrument including a central portion and an orbiting portion (fig. 3D-3F show badge 218 with central portion 220 and orbiting portion 222). The combined shape of the full badge, comprising both the central portion 220 and orbiting portion 222, corresponds to the claimed “indicator body”. The central portion of the badge contains text information, which corresponds to the claimed “icon”.
PNG
media_image1.png
388
912
media_image1.png
Greyscale
Itkowitz teaches: wherein the indicator body has a surface and the icon appears affixed to the surface of the indicator body (the text is overlaid on the surface of the badge).
PNG
media_image2.png
386
995
media_image2.png
Greyscale
The above image shows a simplified version of the combined shape of the central and orbiting portions of the badge comprising the claimed “indicator body”. Though Itkowitz describes the functionality of the orbiting portion as a translational rather than rotational motion, the effect on the combined shape from a user’s perspective is that the badge appears to rotate so that the part of the shape comprising the orbiting portion faces away from the position of an instrument’s end effector – the direction of the orbiting portion serving a similar purpose as the head of Seeber’s arrow. However, the text “icon” on the central portion remains upright regardless of the rotation of the badge, meaning that an orientation of the icon changes relative to the indicator body as the position of the instrument changes relative to the field of view. Furthermore, the text “icon” always remains in the center of the badge’s central portion as the badge rotates to match the instrument’s position, meaning that a position of the icon remains unchanged relative to the indicator body as the position of the instrument changes relative to the field of view.
Seeber and Itkowitz are both analogous to the claimed invention because they are in the same field of graphical displays to aid in robotic surgery by indicating the location of surgical instruments. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Seeber with the teachings of Itkowitz to automatically rotate the informative icon to always remain upright, rather than following the direction of the indicator. Since the indicators of Seeber and Itkowitz are both graphical displays implemented in a similar manner and a similar context, one of ordinary skill in the art would have found it obvious to substitute aspects of the indicator of Seeber with aspects the indicator of Itkowitz to obtain predictable results. An additional motivation may have been to improve the user experience by making the icon easier for the surgeon to quickly read, regardless of the orientation of the associated indicator.
Regarding claim 2, the combination of Seeber in view of Itkowitz teaches: The medical system of claim 1 wherein the icon on the indicator body indicates a controlling manipulator arm (Itkowitz [0042] “The central portion 214 and orbiting portion 216 may contain different pieces of association information. For example, as shown in FIG. 2, the central portion 214 includes a number indicating the arm 54b (e.g. Arm “3”) to which the medical instrument 26b is attached.”).
Seeber and Itkowitz are both analogous to the claimed invention because they are in the same field of graphical displays to aid in robotic surgery by indicating the location of surgical instruments. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Seeber with the teachings of Itkowitz to include information in the icon indicating which arm the associated indicator points toward. The motivation would have been to improve the user experience by providing the surgeon with pertinent information in a convenient, easily visible location.
Regarding claim 3, the combination of Seeber in view of Itkowitz teaches: The medical system of claim 2, wherein an orientation of the icon changes relative to the indicator body as the field of view changes (Itkowitz fig. 3D-3F show the rotation of the shape of the badge 218, comprising both the central portion 220 and orbiting portion 222, based on the orientation of the surgical tool and the user’s control device, as described in [0046], and the number on the central portion remaining in the upright orientation to display the text upright; similar to what was discussed in claim 1, if the indicator of Seeber is combined with the icon of Itkowitz, as the field of view changes, the badge will rotate to correspond with the instrument’s position while the icon will remain upright).
The rationale and motivation to combine the invention of Seeber with the teachings of Itkowitz would have been similar to the motivation described in claim 1.
Regarding claim 4, the combination of Seeber in view of Itkowitz teaches: The medical system of claim 1, wherein the three-dimensional synthetic indicator includes a three-dimensional directional indicator portion oriented toward the position of the instrument (Seeber fig. 4, synthetic indicator 1250 points toward reference point 1240 located on the instrument;
[0054] “A point of rotation 1240 at the front end of the bendable instrument shaft 309 is used as a reference point of the end effector 310 in the previous embodiment. In other embodiments also other points of the end effector can be used as reference points. Between the point of intersection 303b and the reference point 1240 the distance vector in the three-dimensional space, preferably in the three-dimensional device coordinate system X, Y, Z of the manipulator 12 is determined by the control unit 40. The value of the vector is optionally inserted into the displayed image 1200 as distance information 1260. The illustrated arrow 1250 extends in a section of the vector. The vector is defined by the line segment between the point of intersection 303b and the reference point 1240.”;
[0057] teaches that the displayed arrow 1250 may be three-dimensional: “If the endoscope 312 is a stereo endoscope, the images 1200, 1300 and 1400 may also be captured and output as three-dimensional images. The arrows 1250 and 1450 can then be inserted with a corresponding orientation along the course of the determined vector into the three-dimensional image 1200, 1300, 1400 such that also the illustration of the arrows 1250 and 1450 is made in a three-dimensional manner and thus the surgeon obtains a real three-dimensional direction information.”).
Regarding claim 5, the combination of Seeber in view of Itkowitz teaches: The medical system of claim 4 wherein generating the three-dimensional synthetic indicator includes determining a ray between a centerline of the imaging component and a point on the instrument (Seeber fig. 4, indicator 1250 is positioned along a line between center point 303b and reference point 1240 on the instrument;
[0054] explains further: “The crossing dash-dotted lines in the center of the displayed image 1200 show the point of intersection 303b of the optical axis 316 of the endoscope 312 and the object plane 304b. The end effector 310 is located outside the field of view of the endoscope 312 and is thus not illustrated in the image 1200 displayed by the display unit 44. A point of rotation 1240 at the front end of the bendable instrument shaft 309 is used as a reference point of the end effector 310 in the previous embodiment. In other embodiments also other points of the end effector can be used as reference points. Between the point of intersection 303b and the reference point 1240 the distance vector in the three-dimensional space, preferably in the three-dimensional device coordinate system X, Y, Z of the manipulator 12 is determined by the control unit 40. The value of the vector is optionally inserted into the displayed image 1200 as distance information 1260. The illustrated arrow 1250 extends in a section of the vector.”).
Regarding claim 6, the combination of Seeber in view of Itkowitz teaches: The medical system of claim 1, wherein the three-dimensional synthetic indicator includes a height dimension, a width dimension, and a depth dimension (Seeber [0057] teaches a 3D indicator: “If the endoscope 312 is a stereo endoscope, the images 1200, 1300 and 1400 may also be captured and output as three-dimensional images. The arrows 1250 and 1450 can then be inserted with a corresponding orientation along the course of the determined vector into the three-dimensional image 1200, 1300, 1400 such that also the illustration of the arrows 1250 and 1450 is made in a three-dimensional manner and thus the surgeon obtains a real three-dimensional direction information.”; 3D models must necessarily be measurable in three spatial dimensions).
Regarding claim 7, the combination of Seeber in view of Itkowitz teaches: The medical system of claim 1 wherein the three-dimensional synthetic indicator is pivotable relative to the position of the instrument as the field of view changes (Seeber [0053] teaches that the endoscope may be pivoted to change the field of view: “When the endoscope 312 is pivoted by the manipulator arm 16b such that the optical axis 316 is shifted or pivoted in the direction of the arrow P1…”; [0054] teaches that the direction of the synthetic indicator 1250 is dependent only on the location of the reference point 1240 located on the instrument, meaning that if the field of view changes, the synthetic indicator will also pivot to remain pointing towards the instrument).
Regarding claim 9, the combination of Seeber in view of Itkowitz teaches: The medical system of claim 1 wherein generating the three-dimensional synthetic indicator includes determining a depth range for structures in the field of view (Seeber [0020] “Further, it is advantageous when the distance of the point to the object plane is preset in the control unit as a parameter such that it lies within the depth of field. In an imaging optical system of an image capturing unit, the object plane is a plane running perpendicularly to the optical axis and in particular containing the object point. Only the points of the object plane are sharply imaged on the image sensor of the image capturing unit. The depth of field, often also referred to as field depth, is a measure for the extension of the sharp area in the object space of an imaging optical system. The depth of field describes the size of the distance range within which an object is imaged with sufficient sharpness.”; where generating a synthetic indicator depends on imaging the corresponding instrument).
Regarding claim 10, the combination of Seeber in view of Itkowitz teaches: The medical system of claim 1 wherein the three-dimensional synthetic indicator morphs to indicate a distance between the three-dimensional synthetic indicator and the instrument (Seeber [0054] “Alternatively or additionally, the length of the displayed arrow 1250 can be dependent on the value of the vector. I.e. when the end effector 310 has a longer distance to the field of view of the endoscope 312 and thus a longer distance to the point of intersection 303b, a longer arrow 1250 is displayed and when the distance is shorter a shorter arrow 1250 is displayed in the displayed image 1200.”).
Regarding claim 32, it is rejected with the same references, rationale, and motivation to combine as claim 1 because its limitations substantially correspond to the limitations of claim 1, with the additional limitation of a machine-readable media storing instructions (Itkowitz [0066] “The program or code segments can be stored in a processor readable storage medium…”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Seeber with the teachings of Itkowitz to store necessary code in standard computer-readable storage media so that the invention is compatible with standard computational hardware.
Regarding claims 33-38 and 40-41, their limitations substantially correspond to the limitations of claims 2-7 and 9-10 respectively; therefore, they are rejected with the same references, rationale, and motivation to combine as their corresponding claims.
Claim(s) 8 and 39 is/are rejected under 35 U.S.C. 103 as being unpatentable over Seeber (US 20180228343 A1) in view of Itkowitz (US 20180064499 A1) as applied to claims 1 and 32 above, and further in view of Robinson et al. (GB 2560390 A, hereinafter "Robinson").
Regarding claim 8, the combination of Seeber in view of Itkowitz teaches: the medical system of claim 1, but does not explicitly teach: wherein the three-dimensional synthetic indicator indicates instrument status or control mode.
Robinson teaches: wherein the three-dimensional synthetic indicator indicates instrument status or control mode (fig. 10 icons 1002, 1004, 1006, and 1008 are synthetic indicators which may indicate instrument status; pg. 33 lines 10-14 “Referring to figure 10, the icons 1002, 1004, 1006, 1008 and the imaging device icon 1012 comprise a cluster of three circular sub-icons. Each of the sub-icons suitably provides a status associated with the respective arm. For example, each sub-icon suitably provides a status of a different aspect of the instrument or arm which that icon represents.”; fig. 11 further shows that the aforementioned icons are each associated with a particular instrument or manipulator arm).
Robinson and the combination of Seeber in view of Itkowitz are both analogous to the claimed invention because they are in the same field of graphical displays to aid in robotic surgery. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Seeber in view of Itkowitz with the teachings of Robinson to display information about the status of an associated instrument on the synthetic indicator. The motivation would have been to improve the user experience by providing the surgeon with pertinent information in a convenient, easily visible location.
Regarding claim 39, its limitations substantially correspond to the limitations of claim 8; therefore, it is rejected with the same references, rationale, and motivation to combine as claim 8.
References Cited
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Mustufa et al. (US 20190231459 A1) teaches an indicator for a medical instrument in a graphical interface (fig. 3 and 4 element 114), which also corresponds to the limitations which Itkowitz is relied upon to teach:
wherein the indicator body has a surface and the icon appears affixed to the surface of the indicator body (fig. 3 and 4, icon 118 appears affixed to pointer 116, which corresponds to the “indicator body”); and
wherein an orientation of the icon changes relative to the indicator body as the position of the instrument changes relative to the field of view ([0044] “Responsive to movement of the control device 36 associated with the instrument 26b, the selector icon 114 is animated to pivot about the center of the body 118 to move the pointer 116 in a dial-like fashion.”; it is suggested that the pointer moves while the icon remains stationary), and wherein a position of the icon remains unchanged relative to the indicator body as the position of the instrument changes relative to the field of view (the icon remains in the center of the indicator as the pointer rotates).
Larkin et al. (US 20080004603 A1) teaches a three-dimensional synthetic indicator for a position of an instrument outside the field of view of a surgical environment. The indicator may change in size depending on the distance to the instrument, and it has an icon representing a controlling manipulator arm.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BENJAMIN STATZ whose telephone number is (571)272-6654. The examiner can normally be reached Mon-Fri 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at (571)272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BENJAMIN TOM STATZ/Examiner, Art Unit 2611
/TAMMY GODDARD/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights