DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 1 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
With regards to claim 1, in line 11, it is unclear as to whether the “PoV real images” is referring to the same “PoV real images” set forth in lines 6-7 of claim 1, or referring to different images. For examination purposes, Examiner assumes the former. Claim 12 is similarly rejected.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hannaford et al. (US Pub No. 2011/0238079) in view of Nam et al. (US Pub No. 2015/0272428), Panescu et al. (US Pub No. 2017/0172662), Van Dinther (US Pub No. 2013/0046523), Zamorano et al. (US Pub No. 2003/0179308) and de Almeida Barreto (US Pub No. 2018/0071032)
With regards to claim 1, Hannaford et al. disclose a non-transitory computer-readable medium, a system and a method of guiding a robotic endoscope in a system including the robotic endoscope, the method comprising:
identifying a region of interest (52) of an affected body part of a subject patient (paragraph [0068], referring to the remove surgical site (52) point; paragraph [0014], referring to the preselected margins to dissect during the surgery; Figure 14);
creating rules for the region of interest for generating alerts during a surgical procedure (paragraph [0075], referring to the “ “augmented reality” blending graphical images with real-world views and real robot slaves e.g. go/no-go zones”, wherein the go/no-go zones correspond to rules for the regions of interest and serves as visual alert during a surgical procedure; paragraph [0111], referring to the 3D auditory input devices being used to convey information, such as collisions between surgical tools, contact between the tool tip and some types of tissue, etc., via sound cues; Figure 14);
endoscope image referencing by displaying a point of view (PoV) image using POV real images of the subject patient captured by the robotic endoscope (100) along with PoV augmented reality (AR) anatomical images of the region of interest based on a PoV of the robotic endoscope (paragraphs [0013]- [0014], referring to the display device further “depicted augmented reality for the operator comprising augmented information shown on the display and superimposed over the image of the remote surgical site” and “The display device can further display further augmented information either to a side of or superimposed over the image of the remote surgical site…The image of the remote surgical site and the augmented information can comprise blending graphical images with real-world views of the remote surgical site, and can be provided by at least one of an endoscopic camera”, wherein such superimposition/blending of the augmented information with the endoscopic real-world images/views would require that the AR images are ultimately based on the PoV of the robotic endoscope; paragraphs [0068], [0104], Figures 14-15);
wherein the endoscope image referencing includes:
inserting the robotic endoscope into the subject patient, guiding the robotic endoscope using an interface (i.e. local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, wherein the local surgical cockpit comprises a local 3D audio (i.e. a voice interface for ultimately guiding the robotic enodscope) configured to obtain remote 3D audio input from a remote operation site, etc.) for interacting with the system based on the displayed real images and the AR anatomical images (paragraph [0011], referring to the head-mounted display which extends to the local surgeon’s eyes from an articulated boom or other retention structure disposed in front of the local surgeon’s eyes, wherein the retention structure can be actuated by at least one hand control located on the cockpit “or by voice control..”; paragraph [0014], referring to the display device displaying augmented information either to a side or superimposed over the image of the remote surgical site, wherein the augmented information comprises a preplanned trajectory for a surgical tool, etc.; paragraphs [0036]-[0037], referring to the local surgical cockpit comprising a local surgical console comprising local 3D audio configured to obtain remote 3D audio input from a remote operation site, wherein the system can comprise remote 3D audio sensors connected to the local 3D audio such that the local 3D audio precisely transmits 3D audio signals from the remote 3D audio sensors and the 3D audio signal are correlated with tactile feedback to provide correlated response to haptic input devices at the local surgical cockpit; paragraphs [0007],[0068]-[0069], [0102]-[0104], [0108], referring to the visual information (i.e. images captured by endoscope along with augmented information) being provided to surgeons to perform procedures by teleoperating robotic arms which can contain the endoscope (100) in remote surgical instrument 88a coupled with positioning and/or zooming the camera (i.e. guiding the camera of the endoscope); paragraphs [ 0075]-[0078], [0081]-[0082] referring to haptic feedback, robot control commands, etc., wherein the cockpit (2) can include at least two functional interfaces for the surgeon’s hands (master robotic arm) and the feet (foot pedal array), wherein the interfaces can include two robotic arms (20) and eight foot pedals (28), [0113]-[0114]; paragraphs [0081], [0102]-[0105], [0107], referring to the cockpit system/surgical consoles which can be configured such that an operator can control a remote robotic arm; paragraphs [0109]-[0112], referring to the headset (90) comprising a 3D auditory input device, wherein the auditory interface/input devices (80) permits local surgical areas/cockpits to become consistent virtual listening areas and can be used to convey information, such as collisions between the surgical tools, contact between the tool tip and some types of tissue, stress levels applied to the tissue, etc., via either natural or synthetic sound cues, wherein “Audio may play an important role in perceiving the important information generated by the surgical site or the OR by the surgeon located in a remote site”; paragraph [0113], referring to the multiple interfaces, including the display (50) and 3D audio input devices; Figures 6-16),
wherein the interface is a command line interface, a graphical user interface, or a voice interface (i.e. audio interface devices (80)) for guiding the robotic endoscope (paragraphs [ 0075]-[0078], [0081]-[0082], wherein the display serves as a graphical user interface; paragraphs [0011], [0036]-[0037], [0077], [0109]-[0116], referring to the audio/voice cues which would be associated with a voice/audio interface).
However, Hannaford et al. do not specifically disclose that the created rules comprise of “distance-based” rules, wherein the distance-based rules further include displaying, during the endoscope image referencing, distances from the affected body part, as a function of speed of the endoscope.
Further, Hannaford et al. do not specifically disclose that the PoV image is a PoV composite image using the PoV real images and the robotic endoscope includes a plurality of cameras provided around a periphery of a head of the robotic endoscope having an orientation and direction to provide a front PoV of the robotic endoscope such that all of the plurality of cameras capture PoV real images that are integrated together to produce the PoV composite image of the robotic endoscope.
Additionally, Hannaford et al. do not disclose that the endoscope image referencing includes updating the displayed PoV composite image and the AR anatomical images based on a change in a viewing angle of the robotic endoscope and guiding the robotic endoscope based on a position and a direction of a virtual version of the robotic endoscope relative to the updated displayed PoV composite image and the AR anatomical images.
Nam et al. disclose an endoscope comprising an end portion configured to take an image, wherein a lens apparatus (120) is configured to surround the exterior surface of the end portion (100) of the endoscope, and thus it may obtain three-dimensional or multi-view images in a short period of time, and change the curvature of the lens (121) by electric signals and obtain multi-focused images (Abstract; paragraphs [0039]-[0040], [0049], [0054]-[0056]; Figures 1-2). The lens apparatus (120) may comprise a lens (121) for taking an image, an image sensor (122) for extracting an image taken by the lens (121) and a lens supporter (123) for supporting the lens (121) (paragraph [0044]). Therefore, the lens apparatus is capable of obtaining three-dimensional, multi-view and multi-focused images in a short period of time, thereby providing an effect of reducing the time needed for scanning an external object in order to obtain images, thus reducing the pain to the examinee (paragraphs [0056]-[0057]). The image sensor (122) may extract the images as a high definition image, and thus the image sensor (122) may make 3D images of an external object by combining the extracted numerous images (paragraph [0051], note that the 3D images correspond to a “composite”/combined image as they are made by “combining”/integrating the images). As depicted in Figure 2, the endoscope includes a plurality of image sensors/cameras (122) provided around a periphery of a head of the endoscope having an orientation and direction to provide a front PoV of the endoscope (paragraph [0052], wherein, as depicted in Fig. 2, the image sensors/cameras (122) can image a POV in front of the distal end of the endoscope).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have the PoV image of Hannaford et al. be a PoV composite image using the PoV real images and the robotic endoscope of Hannaford et al. include a plurality of cameras provided around a periphery of a head of the robotic endoscope having an orientation and direction to provide a front PoV of the robotic endoscope such that all of the plurality of cameras capture PoV real images that are integrated together to produce the PoV composite image of the robotic endoscope, as taught by Nam et al., in order to obtain three-dimensional, multi-view and multi-focused images in a short period of time, thereby providing an effect of reducing the time needed for scanning an external object in order to obtain images, thus reducing the pain to the examinee (paragraphs [0056]-[0057]).
However, the above combined references do not specifically disclose that the created rules comprise of “distance-based” rules, wherein the distance-based rules further include displaying, during the endoscope image referencing, distances from the affected body part, as a function of speed of the endoscope.
Additionally, the above combined references do not disclose that the endoscope image referencing includes updating the displayed PoV composite image and the AR anatomical images based on a change in a viewing angle of the robotic endoscope and guiding the robotic endoscope based on a position and a direction of a virtual version of the robotic endoscope relative to the updated displayed PoV composite image and the AR anatomical images.
Panescu et al. disclose a system that includes an endoscope disposed to image a field of view and a processor that produces a quantitative 3D model of a scene and identifies target instruments and structures (Abstract; paragraph [0003]). The system determines a “no fly zone” adjacent to at least one target within a Q3D model, wherein the determination is made as to whether the “no fly zone” is violated based at least in part upon whether a closest distance between the first target and the second target is less than a threshold distance (paragraphs [0013], [0159]). An output signal (i.e. alert) is provided in response to a determination that the closest distance between the first and second targets is less than the threshold distance, wherein said targets may be an instrument, a structure or an anatomic organ (paragraphs [0013], [0124], [0159], note that the determination that the closest distance between the targets is less than a threshold distance corresponds to creating “distance-based” rules for the region of interest for generating alerts; Figure 20). The distance between the instrument and target can be displayed, wherein both the distance can be displayed and the proximity alert set (paragraphs [0060], [0077], [0079], [0125]; Figures 4 and 20). The invention prevents any instruments from violating a no-fly zone and getting too close and perforating a delicate anatomical structure (paragraph [0159]).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have the created rules of the above combined references comprise of “distance-based” rules, wherein the distance-based rules further include displaying, during the endoscope image referencing, distances from the affected body part, as taught by Panescu et al., in order to prevent any instruments from violating a no-fly zone and getting too close and perforating a delicate anatomical structure (paragraph [0159]).
However, the above combined references do not specifically disclose that the distances are displayed “as a function of speed of the endoscope”.
Additionally, the above combined references do not disclose that the endoscope image referencing includes updating the displayed PoV composite image and the AR anatomical images based on a change in a viewing angle of the robotic endoscope and guiding the robotic endoscope based on a position and a direction of a virtual version of the robotic endoscope relative to the updated displayed PoV composite image and the AR anatomical images.
Van Dinther discloses an endoscope simulator, wherein the system includes means for determining parameters associated with movement of the endoscope in a simulated passageway or cavity (Abstract; paragraphs [0001], [0055]). The parameters may include a speed of movement, a speed immediately prior to a collision, etc., which are used to measure the competency of operation by a user, wherein the parameters may be displayed (paragraphs [0055]-[0056], [0141]; note that a speed is defined by distance/time, and therefore distances are displayed as a function of speed (i.e. distance/time).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have the distances of the above combined references be displayed/represented as a function of speed of the endoscope, as taught by Van Dinther, in order to provide an effective measure of the competency of operation by a user and/or further to provide an alternative parameter representative of collision (paragraph [0055]).
However, the above combined references do not disclose that the endoscope image referencing includes updating the displayed PoV composite image and the AR anatomical images based on a change in a viewing angle of the robotic endoscope and guiding the robotic endoscope based on a position and a direction of a virtual version of the robotic endoscope relative to the updated displayed PoV composite image and the AR anatomical images.
Zamorano et al. discloses an augmented reality system (100) that employs a robotic positioning device (125) to position a video camera (120) in a desired position relative to an object of interest (110), the video camera (120) being positioned at an end-effector (126) of the robotic positioning device (paragraphs [0012]-[0014]). In order to generate the augmented reality image (191), the augmented reality processor (180) employs the tracking system (150), which obtains tracking data with respect to the location of the video camera (120), and the object registration module (160) in order to ensure that the data image (193, AR image) that is merged with the video image (192, video data obtained from the video camera (120)) corresponds both in time and in space to the video image (192), thereby eliminating the need for a user to separately view both a video image obtained from a video camera and displayed on a display device and a separate image having additional information (paragraphs [0018]-[0020], note that such merging of data over “time” would require updating the displayed images over time; Figures 1-2). Further, data is acquired at different positions/orientations and the tracking system provides the current position and orientation (viewing angle) to be used to generate data image (193) that corresponds to the displayed video image, and thus the displayed anatomical image would be updated for every current position/orientation corresponding to the different positions and orientation/viewing angle of the camera (paragraphs [0031]-[0034], [0037], note that the updated images would therefore be based on a change in a viewing angel/orientation of the video camera; paragraphs [0001], [0052], referring to data being obtained “during the performance of the surgical procedure” and the merging of the real-time video data with data, which would require updating the displayed anatomical image to provide the “real-time” display).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have the endoscope image referencing of the above combined references include updating the displayed PoV composite image and the AR anatomical images based on a change in a viewing angle of the robotic endoscope, as taught by Zamorano et al., in order to have the AR anatomical image correspond both in time and in space to the displayed real image, thereby eliminating the need for a user to separately view both a real image displayed on a display device and a separate image having additional information (paragraphs [0018]-[0020]).
However, the above combined references do not disclose that guiding the robotic endoscope is based on a position and a direction of a virtual version of the robotic endoscope relative to the updated displayed PoV composite image and the AR anatomical images.
de Almeida Barreto discloses computer-assisted procedures of surgery and diagnosis, wherein small visual markers are attached to instruments and anatomy of interest (Abstract; paragraph [0011]). A surgical plan can be overlaid into images in real-time (Augmented Reality) for the purpose of guiding the surgeon in positioning and/or orienting an instrument (paragraph [0020], [0067]; Figure 1E). The clinical execution might require multiple different instruments, wherein each of these instruments is assumed to have a Tool Marker (TM) attached that defines a local system of coordinates where the instrument’s relative parts, such as a tip, symmetry axis, or even a complete CAD model (i.e. “virtual version” of the instrument) are represented (paragraphs [0067]-[0068]; Figure 1E). The instruments can be located in the world coordinate system and their poses can be related with the 3D information stored in the WM reference frame, and ultimately provide real-time assistance to the surgeon (paragraph [0068]; Figure 1E). The surgeon is thus assisted by performing continuous processing of the video for estimating in real-time the 3D pose of instruments with respect to patient anatomy and/or surgical plan represented in WM coordinates (paragraphs [0069], [0158]-[0159]; Figures 1E, 6). The assistance can be in the form of overlaying guidance information in video using Augmented Reality (AR), using computer graphics to animate the motion of instruments in a Virtual Reality (VR) environment showing the patient’s anatomy and/or surgical plan, etc., which provides the surgeon the ability to visualize the expected outcome of the chosen instrument location and thus errors can be avoided (paragraphs [0041], [0069], [0158]-[0159]; Figures 1E, 6, note that a virtual version of the instrument, including movement/motion of the instrument, can be overlaid on the patient anatomy real-time images (i.e. continuously updated images), thus providing guidance of the instrument based on movement of a virtual version (i.e. CAD/computer graphics animation of instrument) of the instrument relative to the displayed anatomical image/patient anatomy).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have the guiding of the robotic endoscope of the above combined references be based on a position and a direction of a virtual version of the robotic endoscope relative to the updated displayed PoV composite image and the AR anatomical images, as taught by de Almeida Barreto, in order to provide real-time assistance to the surgeon and provide the surgeon the ability to visualize the expected outcome of the chosen instrument location and thus errors can be avoided (paragraphs [0041], [0069], [0158]-[0159]).
Conclusion
This is a continuation of applicant's earlier Application No.18/174,287. All claims are identical to, patentably indistinct from, or have unity of invention with the invention claimed in the earlier application (that is, restriction (including lack of unity) would not be proper) and could have been finally rejected on the grounds and art of record in the next Office action if they had been entered in the earlier application. Accordingly, THIS ACTION IS MADE FINAL even though it is a first action in this case. See MPEP § 706.07(b). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATHERINE L FERNANDEZ whose telephone number is (571)272-1957. The examiner can normally be reached Monday-Friday 9:00 AM - 5:30 PM (ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pascal Bui-Pho can be reached at (571) 272-2714. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATHERINE L FERNANDEZ/Primary Examiner, Art Unit 3798
Prosecution Insights