Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This Office Action is in response to the amendments dated March 31, 2026.
Claims 1-2, 4-12, and 14-26 are pending.
Claim Objections
Claim 22 is objected to because of the following informalities: line 2 includes the feature “a second pose of the front-end portion”. Examiner believes that this is a typographical error, and should read “the second pose of the front-end portion” by referring to “a second pose” in the front-end portion introduced in lines 16-17 of Claim 1. Appropriate correction by Applicant is required.
Claim Rejections - 35 USC § 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 26 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. More specifically, Claim 26 determines a distance between a target and an image sensor based on how many pixels (“pixel displacement”) are used to capture the image. Line 4 of Claim 26 claims “the pixel displacement representing a disparity of the second body part on a display”. It is unclear if the disparity is of the second body part itself, or a disparity between the first image of the second body part and the second image of the second body part.
Appropriate correction by Applicant is required.
For purposes of examination, Examiner interprets Claim 26 as claiming that the distance between the image sensor and the second body part is determined by a disparity between the first image of the second body part and the second image of the second body part.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1 and 11 are provisionally rejected on the ground of obviousness-type nonstatutory double patenting as being unpatentable over respective Claims 1 and 11 of copending Application No. 18/585,495 in view of Ono (US PGPUB 2010/0079659 – “Ono ‘659”).
Claims 1 and 11 of the present patent application include all features found in Claims 1 and 11 of copending Application No. 18/585,495 (see chart below), except for the feature of calculating of the relative pose information using a distance between the identified at least one second body part and the front-end portion, the distance being determined based on changes in image information between a first image and a second image of the one or more images.
Ono ‘659 teaches the relative pose information representing a change in pose of the front-end portion from a first pose to a second pose, and the calculating of the relative pose information using a distance between the identified at least one second body part and the front-end portion, the distance being determined based on changes in image information between a first image and a second image of the one or more images (Ono ‘659 FIG. 1, position identifying section; Ono ‘659 FIG. 4A, first imaging position 301-1 and second imaging position 301-2 relative to the subject position 300 shown in Ono ‘659 FIG. 4B; Ono ‘659 paragraph [0063], “if the object captured at the position 301-1 in the first image is captured at the position 301-2in the second image, the position identifying section 150 can identify the distance to the subject based on the difference between the position 301-1 and the position 301-2”).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Ono ‘659’s pose/position/distance method with the method disclosed by Gormley. A person having ordinary skill in the art would be motivated to combine these prior art elements according to known methods to yield the predictable result of a method that can measure distances between an optical sensor and a target of interest without using additional sensors, such as time-of-flight sensors, magnetic sensors, gyroscopic sensors, etc. for determining the pose of the optical sensor in situ.
Dependent Claims 2, 4-10, 12, and 14-26 are rejected under obviousness-type provisional nonstatutory double patenting rejection based on their dependence on their respective independent/base Claims 1 and 11.
Current Application 18/585,449
Co-pending Application 18/585,495
Claim 1
Claim 1.
A method of controlling an endoscopic device, the method comprising:
A method of controlling an endoscopic device, the method comprising:
acquiring one or more images of an upper gastrointestinal tract from an image sensor;
acquiring environment information with respect to a front-end portion of the endoscopic device, wherein the environment information includes structural information with respect to a body shape that characterizes a three-dimensional shape of a lumen of the lower gastrointestinal tract
detecting at least one first body part from the one or more images, based on a pre-trained model;
detecting at least one first body part from the image, based on a pre-trained model;
calculating relative position information between the detected at least one first body part and a front-end portion of the endoscopic device;
calculating relative position information between the at least one first body part and the front-end portion of the endoscopic device;
generating, based on the relative position information, a first control signal for steering the front-end portion to correspond to the at least one first body part;
generating, based on the environment information and the relative position information, a first control signal for steering the front-end portion to correspond to the at least one first body part, and wherein generating the first control signal comprises determining a steering direction and a steering magnitude that conforms to the three- dimensional shape of the lumen; and
transmitting the first control signal to a driver;
transmitting the first control signal to a driver;
identifying at least one second body part from the one or more images based on the pre-trained model;
identifying at least one second body part from the image, based on the pre-trained model;
calculating relative pose information between the identified at least one second body part and the front-end portion,
calculating relative pose information between the identified at least one second body part and the front-end portion,
the relative pose information representing a change in pose of the front-end portion from a first pose to a second pose,
the relative pose information representing a change in pose of the front-end portion from a first pose to a second pose;
the calculating of the relative pose information using a distance between the identified at least one second body part and the front-end portion, the distance being determined based on changes in image information between a first image and a second image of the one or more images;
generating, based on the relative pose information, a second control signal with respect to photographing at least one photographing point corresponding to the identified at least one second body part; and
generating, based on the relative pose information, a second control signal, distinct from the first control signal, for positioning the front-end portion at a pose suitable for photographing the identified at least one second body part; and
transmitting the second control signal to the driver.
transmitting the second control signal to the driver.
Claim 11.
Claim 11.
An endoscopic device comprising:
An endoscopic device comprising:
a front-end portion including an image sensor;
a front-end portion comprising an image sensor to acquire an image of a lower gastrointestinal tract;
a driver configured to control a rotation angle of the front-end portion; and
a driver configured to control a rotation angle of the front-end portion; and
a controller configured to:
a controller configured to:
acquire the image of the lower gastrointestinal tract;
acquire environment information with respect to the front-end portion, wherein the environment information includes structural information with respect to a body shape that characterizes a three-dimensional shape of a lumen of the lower gastrointestinal tract;
detect, based on a pre-trained model, at least one first body part from one or more images captured by the image sensor;
detect, based on a pre-trained model, at least one first body part from the image;
calculate relative position information between the detected at least one first body part and the front-end portion;
calculate relative position information between the at least one first body part and the front-end portion of the endoscopic device;
generate, based on the relative position information, a first control signal for steering the front-end portion to correspond to the at least one first body part; and
generate, based on the environment information and the relative position information, a first control signal to steer the front-end portion to correspond to the at least one first body part, wherein the first control signal specifies a steering direction and a steering magnitude that conform to the three-dimensional shape of the lumen;
transmit the first control signal to the driver;
transmit the first control signal to the driver;
identify, based on the pre-trained model, at least one second body part from the one or more images;
identify at least one second body part from the image, based on the pre- trained model;
calculate relative pose information between the identified at least one second body part and the front-end portion,
calculate relative pose information between the identified at least one second body part and the front-end portion;
the relative pose information representing a change in pose of the front-end portion from a first pose to a second pose,
the relative pose information representing a change in pose of the front-end portion from a first pose to a second pose;
the calculating of the relative pose information using a distance between the identified at least on second body part and the front-end portion, the distance being determined based on changes in image information between a first image and a second image of the one or more images;
generate, based on the relative pose information, a second control signal with respect to photographing at least one photographing point corresponding to the identified at least one second body part; and
generate, based on the relative pose information, a second control signal, distinct from the first control signal, for positioning the front-end portion at a pose suitable for photographing the identified at least one second body part; and
transmit the second control signal to the driver.
transmit the second control signal to the driver.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The present rejection(s) reference specific passages from cited prior art. However, Applicant is advised that the rejections are based on the entirety of each cited prior art. That is, each cited prior art reference “must be considered in its entirety”. Therefore, Applicant is advised to review all portions of the cited prior art if traversing a rejection based on the cited prior art.
Claims 1-2, 6-8, 11-12, 16-18, and 21-24 are rejected under 35 U.S.C. 103 as being unpatentable over Gormley et al. (US PGPUB 2021/0059607 – “Gormley”) in view of Ono (US PGPUB 2010/0079659 – “Ono ‘659”).
Regarding Claim 1, Gormley discloses:
A method of controlling an endoscopic device (Gormley FIG. 1A, catheter tube 100), the method comprising:
acquiring an image of an upper gastrointestinal tract (Gormley FIG. 3, showing catheter tube 300 within gastrointestinal tract enteral system 346) from an image sensor (Gormley FIG. 1A, imaging device 108 at distal end of catheter tube 100);
detecting at least one first body part from the image, based on a pre-trained model (Gormley paragraph [0054], “at least one artificial intelligence model may include a detection and tracking model that processes the captured image data in near-real time, a deep-learning detector configured to identify orifices and structures within the enteral cavity or respiratory tract, the deep-learning detector including at least one convolutional-neural-network-based detection algorithm that is trained to learn unified hierarchical representations, that identifies the orifices and structures based on the captured image data, and that calculates the navigation data based on the captured image data and the target destination”);
calculating relative position information between the detected at least one first body part and a front-end portion of the endoscopic device; generating, based on the relative position information, a first control signal for steering the front-end portion to correspond to the at least one first body part (Gormley paragraph [0094], “The imaging device 108 may capture image data (“captured image data”) corresponding to structures (e.g., of the organ being traversed by the catheter tube 100) surrounding the distal end of the catheter tube 100. For example, LiDAR, time of flight imaging, visual image sensing (e.g., which may involve the capture of still images and/or video), or other applicable imaging techniques may be applied to capture the image data. Topographic image data that may be included in the captured image data may provide information related to the shape, volume, consistency, and location of the organ, or the portion of the organ, through which the distal end of the catheter tube 100 is traversing. The captured image data may be transmitted to and used by one or more artificial intelligence (AI) models executed by the remote device that is in wireless electronic communication with the transceiver 106, providing feedback to the AI model(s) regarding the location and position of the catheter tube 100 in the subject's body (e.g., in an organ thereof). Thus, the image data generated by the imaging device 108 may be used to guide the placement of the catheter tube 100 and to continuously monitor the location of the catheter tube 100 once it has reached the target location”);
transmitting the first control signal to a driver (Gormley FIG. 4A, robotic control and display center (RCDC) 400; Gormley paragraph [0144], ” the stylet of the robot extending from the RCDC would be placed through one end of the endotracheal tube to be inserted and brought out the other end. The stylet would then be placed either in the nostril of the patient (either the left or right nostril) or in the mouth of a patient (alongside a standard oral-pharyngeal tube)… Using images obtained from the visual and topographic cameras at the tip of the stylet, the computer's algorithm would begin to recognize structure in the nasopharynx or oropharynx (depending on the site of insertion) and given these images the robot would direct the stylet down the pharynx into the larynx…and through the use of the actuators and motors that control all of its degrees of freedom, steer the stylet anteriorly through the larynx and through the vocal chords into the trachea. This will all be done using computer vision as a guide, without input required from any clinician at the patient's side. The decisions guiding the direction of the stylet will all be automated through the computer algorithm and controlled through the mechanical system of the device.”); and
identifying at least one second body part from the one or more images based on the pre-trained model; calculating relative pose information between the identified at least one second body part and the front-end portion; generating, based on the relative pose information, a second control signal with respect to photographing of at least one photographing point corresponding to the identified at least one second body part; and transmitting the second control signal to the driver (Gormley paragraph [0093], “transceiver 106 may also wirelessly transmit imaging data (e.g., topographic image data, still image data, and/or video data) captured by the imaging device 108 to the remote device. The transceiver 106 may transmit this data to the remote device both during the device placement process (e.g., as the catheter tube 100 is automatically driven to a target location)”; Gormley paragraph [0094], “the image data generated by the imaging device 108 may be used to guide the placement of the catheter tube 100”; Gormley paragraph [0144], “Using images obtained from the visual and topographic cameras at the tip of the stylet, the computer's algorithm would begin to recognize structure in the nasopharynx or oropharynx (depending on the site of insertion) and given these images the robot would direct the stylet down the pharynx into the larynx. At this point the epiglottis will come into the sight of the robot, which will be recognized. The algorithm will recognize the juncture of the larynx anteriorly and the esophagus posteriorly and through the use of the actuators and motors that control all of its degrees of freedom”; Examiner interprets this continuous capturing of images of interim landmarks while traversing to the target during the automatic drive of the catheter tube disclosed by Gormley as disclosing the presently claimed steps of 1) identifying an interim second body part from the image based on the pre-trained model and 2) adjusting the movement of the endoscope by use of this interim second body part image.).
Gormley does not explicitly disclose:
the relative pose information representing a change in pose of the front-end portion from a first pose to a second pose, and the calculating of the relative pose information using a distance between the identified at least one second body part and the front-end portion, the distance being determined based on changes in image information between a first image and a second image of the one or more images.
Ono ‘659 teaches the relative pose information representing a change in pose of the front-end portion from a first pose to a second pose, and the calculating of the relative pose information using a distance between the identified at least one second body part and the front-end portion, the distance being determined based on changes in image information between a first image and a second image of the one or more images (Ono ‘659 FIG. 1, position identifying section; Ono ‘659 FIG. 4A, first imaging position 301-1 and second imaging position 301-2 relative to the subject position 300 shown in Ono ‘659 FIG. 4B; Ono ‘659 paragraph [0063], “if the object captured at the position 301-1 in the first image is captured at the position 301-2in the second image, the position identifying section 150 can identify the distance to the subject based on the difference between the position 301-1 and the position 301-2”).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Ono ‘659’s pose/position/distance method with the method disclosed by Gormley. A person having ordinary skill in the art would be motivated to combine these prior art elements according to known methods to yield the predictable result of a method that can measure distances between an optical sensor and a target of interest without using additional sensors, such as time-of-flight sensors, magnetic sensors, gyroscopic sensors, etc. for determining the pose of the optical sensor in situ.
Regarding Claim 2, Gormley in view of Ono ‘659 teaches the features of Claim 1, as described above.
Gormley further discloses wherein the acquiring of the image of the upper gastrointestinal tract comprises: acquiring a first image at a first point; and acquiring a second image at a second point, and the calculating of the relative position information comprises calculating a target rotation angle of the front-end portion of the endoscopic device based on i) an angle change of the front-end portion of the endoscopic device between the first point and the second point and ii) a change between the first image and the second image (Gormley paragraph [0093], “transceiver 106 may also wirelessly transmit imaging data (e.g., topographic image data, still image data, and/or video data) captured by the imaging device 108 to the remote device. The transceiver 106 may transmit this data to the remote device both during the device placement process (e.g., as the catheter tube 100 is automatically driven to a target location)”; Gormley paragraph [0094], “the image data generated by the imaging device 108 may be used to guide the placement of the catheter tube 100”; Gormley paragraph [0144], “Using images obtained from the visual and topographic cameras at the tip of the stylet, the computer's algorithm would begin to recognize structure in the nasopharynx or oropharynx (depending on the site of insertion) and given these images the robot would direct the stylet down the pharynx into the larynx. At this point the epiglottis will come into the sight of the robot, which will be recognized. The algorithm will recognize the juncture of the larynx anteriorly and the esophagus posteriorly and through the use of the actuators and motors that control all of its degrees of freedom”; Examiner interprets this continuous endoscopic angle change and changes to captured images during the automatic drive of the catheter tube disclosed by Gormley as disclosing the presently claimed steps of 1) acquiring multiple images while traversing through the patient, and 2) adjusting the movement of the endoscope by use of calculated rotation angle and passing landmarks shown in the images).
Regarding Claim 6, Gormley in view of Ono ‘659 teaches the features of Claim 1, as described above.
Gormley further discloses wherein the pre-trained model comprises a classification model and a detection model both trained by using, as a data set, an image labelled with respect to the at least one first body part and the at least one second body part with respect to the upper gastrointestinal tract (Gormley paragraph [0054], “at least one artificial intelligence model may include a detection and tracking model that processes the captured image data in near-real time, a deep-learning detector configured to identify orifices and structures within the enteral cavity or respiratory tract, the deep-learning detector including at least one convolutional-neural-network-based detection algorithm that is trained to learn unified hierarchical representations, that identifies the orifices and structures based on the captured image data, and that calculates the navigation data based on the captured image data and the target destination, and a median-flow filtering based visual tracking module configured to predict the motion vector of the articulated stylet using sparse optical flow.”; Gormley paragraph [0168], “The training part was implemented by feeding the annotated image to Keras implementation of YOLOv3. The version for Keras was 2.2.4 and this version runs TensorFlow 1.15 on the backend. The dataset was created with an annotation software, VoTT R (Microsoft, Redmond, Wash.).”; See also highlighted portions of in the appendix provided herein, which describe VOTT labeling images for classification and detection.).
Regarding Claim 7, Gormley in view of Ono ‘659 teaches the features of Claim 1, as described above.
Gormley further discloses wherein the generating of the second control signal comprises: identifying at least one photographing point corresponding to the identified at least one second body part; generating the second control signal based on the at least one photographing point and the relative position information; and photographing an image when a position of the front-end portion corresponds to the at least one photographing point (see paragraphs [0093] – [0094] and [0144] of Gormley, as cited in the rejection of Claim 1, that describe continuous/repetitive photography of body parts).
Regarding Claim 8, Gormley in view of Ono ‘659 teaches the features of Claim 1, as described above.
Gormley further discloses adjusting a tension of at least one wire connected to the driver and controlling a rotation angle of the front-end portion based on the first control signal (Gormley FIG. 1A, catheter tube 100; Gormley FIGs. 5A-5B; Gormley paragraph [0110], “Various mechanisms of control may be carried out by the robotic control engine 424 to control the movement of the articulated distal end of the articulated stylet. In some embodiments, the mechanism of control may include a pull-wire type configuration where multiple wires are attached at different points in the catheter tube around locations (e.g., articulation joints) where movement (e.g., bending/flexing) of the catheter tube is desired.”).
Regarding Claim 11, Gormley discloses:
An endoscopic device (Gormley FIG. 1A, catheter tube 100) comprising:
a front-end portion including an image sensor (Gormley FIG. 1A, imaging device 108 at distal end of catheter tube 100);
a driver configured to control a rotation angle of the front-end portion (Gormley FIG. 4A, robotic control and display center (RCDC) 400; Gormley FIG. 4B, robotic control engine 424; Gormley paragraph [0144], “the stylet of the robot extending from the RCDC would be placed through one end of the endotracheal tube to be inserted and brought out the other end. The stylet would then be placed either in the nostril of the patient (either the left or right nostril) or in the mouth of a patient (alongside a standard oral-pharyngeal tube)… Using images obtained from the visual and topographic cameras at the tip of the stylet, the computer's algorithm would begin to recognize structure in the nasopharynx or oropharynx (depending on the site of insertion) and given these images the robot would direct the stylet down the pharynx into the larynx…and through the use of the actuators and motors that control all of its degrees of freedom, steer the stylet anteriorly through the larynx and through the vocal chords into the trachea. This will all be done using computer vision as a guide, without input required from any clinician at the patient's side. The decisions guiding the direction of the stylet will all be automated through the computer algorithm and controlled through the mechanical system of the device.”); and
a controller (Gormley FIG. 4A, controller 406) configured to:
detect, based on a pre-trained model at least one first body part from one or more images captured by the image sensor (Gormley paragraph [0054], “at least one artificial intelligence model may include a detection and tracking model that processes the captured image data in near-real time, a deep-learning detector configured to identify orifices and structures within the enteral cavity or respiratory tract, the deep-learning detector including at least one convolutional-neural-network-based detection algorithm that is trained to learn unified hierarchical representations, that identifies the orifices and structures based on the captured image data, and that calculates the navigation data based on the captured image data and the target destination”);
calculate relative position information between the detected at least one first body part and the front-end portion; generate, based on the relative position information, a first control signal for steering the front-end portion to correspond to the at least one first body part (Gormley paragraph [0094], “The imaging device 108 may capture image data (“captured image data”) corresponding to structures (e.g., of the organ being traversed by the catheter tube 100) surrounding the distal end of the catheter tube 100. For example, LiDAR, time of flight imaging, visual image sensing (e.g., which may involve the capture of still images and/or video), or other applicable imaging techniques may be applied to capture the image data. Topographic image data that may be included in the captured image data may provide information related to the shape, volume, consistency, and location of the organ, or the portion of the organ, through which the distal end of the catheter tube 100 is traversing. The captured image data may be transmitted to and used by one or more artificial intelligence (AI) models executed by the remote device that is in wireless electronic communication with the transceiver 106, providing feedback to the AI model(s) regarding the location and position of the catheter tube 100 in the subject's body (e.g., in an organ thereof). Thus, the image data generated by the imaging device 108 may be used to guide the placement of the catheter tube 100 and to continuously monitor the location of the catheter tube 100 once it has reached the target location”);
transmit the first control signal to the driver (Gormley FIG. 4A, robotic control and display center (RCDC) 400; Gormley paragraph [0144], “the stylet of the robot extending from the RCDC would be placed through one end of the endotracheal tube to be inserted and brought out the other end. The stylet would then be placed either in the nostril of the patient (either the left or right nostril) or in the mouth of a patient (alongside a standard oral-pharyngeal tube)… Using images obtained from the visual and topographic cameras at the tip of the stylet, the computer's algorithm would begin to recognize structure in the nasopharynx or oropharynx (depending on the site of insertion) and given these images the robot would direct the stylet down the pharynx into the larynx…and through the use of the actuators and motors that control all of its degrees of freedom, steer the stylet anteriorly through the larynx and through the vocal chords into the trachea. This will all be done using computer vision as a guide, without input required from any clinician at the patient's side. The decisions guiding the direction of the stylet will all be automated through the computer algorithm and controlled through the mechanical system of the device.”);
identify, based on the pre-trained model, at least one second body part from the one or more images; calculate relative pose information between the identified at least one second body part and the front-end portion; generate, based on the relative pose information, a second control signal with respect to photographing at least one photographing point corresponding to the identified at least one second body part; and transmit the second control signal to the driver (Gormley paragraph [0093], “transceiver 106 may also wirelessly transmit imaging data (e.g., topographic image data, still image data, and/or video data) captured by the imaging device 108 to the remote device. The transceiver 106 may transmit this data to the remote device both during the device placement process (e.g., as the catheter tube 100 is automatically driven to a target location)”; Gormley paragraph [0094], “the image data generated by the imaging device 108 may be used to guide the placement of the catheter tube 100”; Gormley paragraph [0144], “Using images obtained from the visual and topographic cameras at the tip of the stylet, the computer's algorithm would begin to recognize structure in the nasopharynx or oropharynx (depending on the site of insertion) and given these images the robot would direct the stylet down the pharynx into the larynx. At this point the epiglottis will come into the sight of the robot, which will be recognized. The algorithm will recognize the juncture of the larynx anteriorly and the esophagus posteriorly and through the use of the actuators and motors that control all of its degrees of freedom”; Examiner interprets this continuous capturing of images of interim landmarks while traversing to the target during the automatic drive of the catheter tube disclosed by Gormley as disclosing the presently claimed steps of 1) identifying an interim second body part from the image based on the pre-trained model and 2) adjusting the movement of the endoscope by use of this interim second body part image.).
Gormley does not explicitly disclose the relative pose information representing a change in pose of the front-end portion from a first pose to a second pose, the calculating of the relative pose information using a distance between the identified at least on second body part and the front-end portion, the distance being determined based on changes in image information between a first image and a second image of the one or more images.
Ono ‘659 teaches the relative pose information representing a change in pose of the front-end portion from a first pose to a second pose, the calculating of the relative pose information using a distance between the identified at least on second body part and the front-end portion, the distance being determined based on changes in image information between a first image and a second image of the one or more images (Ono ‘659 FIG. 1, position identifying section; Ono ‘659 FIG. 4A, first imaging position 301-1 and second imaging position 301-2 relative to the subject position 300 shown in Ono ‘659 FIG. 4B; Ono ‘659 paragraph [0063], “if the object captured at the position 301-1 in the first image is captured at the position 301-2in the second image, the position identifying section 150 can identify the distance to the subject based on the difference between the position 301-1 and the position 301-2”).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Ono ‘659’s pose/position/distance method with the method disclosed by Gormley. A person having ordinary skill in the art would be motivated to combine these prior art elements according to known methods to yield the predictable result of a method that can measure distances between an optical sensor and a target of interest without using additional sensors, such as time-of-flight sensors, magnetic sensors, gyroscopic sensors, etc. for determining the pose of the optical sensor in situ.
Regarding Claim 12, Gormley in view of Ono ‘659 teaches the features of Claim 11, as described above.
Gormley further discloses wherein the controller is further configured to acquire the first image at a first point and a second image at the second point at a second pint and wherein, to calculate the relative position information, the controller is configured to calculate a target rotation angle of the front-end portion of the endoscopic device, based on i) an angle change of the front-end portion between the first point and the second point and ii) a change between the first image and the second image, a target rotation angle of the front-end portion (Gormley paragraph [0093], “transceiver 106 may also wirelessly transmit imaging data (e.g., topographic image data, still image data, and/or video data) captured by the imaging device 108 to the remote device. The transceiver 106 may transmit this data to the remote device both during the device placement process (e.g., as the catheter tube 100 is automatically driven to a target location)”; Gormley paragraph [0094], “the image data generated by the imaging device 108 may be used to guide the placement of the catheter tube 100”; Gormley paragraph [0144], “Using images obtained from the visual and topographic cameras at the tip of the stylet, the computer's algorithm would begin to recognize structure in the nasopharynx or oropharynx (depending on the site of insertion) and given these images the robot would direct the stylet down the pharynx into the larynx. At this point the epiglottis will come into the sight of the robot, which will be recognized. The algorithm will recognize the juncture of the larynx anteriorly and the esophagus posteriorly and through the use of the actuators and motors that control all of its degrees of freedom”; Examiner interprets this continuous endoscopic angle change and changes to captured images during the automatic drive of the catheter tube disclosed by Gormley as disclosing the presently claimed steps of 1) acquiring multiple images while traversing through the patient, and 2) adjusting the movement of the endoscope by use of calculated rotation angle and passing landmarks shown in the images).
Regarding Claim 16, Gormley in view of Ono ‘659 teaches the features of Claim 11 , as described above.
Gormley further discloses wherein the pre-trained model comprises a classification model and a detection model both trained by using, as a data set, an image labelled with respect to the at least one first body part and the at least one second body part with respect to the upper gastrointestinal tract (Gormley paragraph [0054], “at least one artificial intelligence model may include a detection and tracking model that processes the captured image data in near-real time, a deep-learning detector configured to identify orifices and structures within the enteral cavity or respiratory tract, the deep-learning detector including at least one convolutional-neural-network-based detection algorithm that is trained to learn unified hierarchical representations, that identifies the orifices and structures based on the captured image data, and that calculates the navigation data based on the captured image data and the target destination, and a median-flow filtering based visual tracking module configured to predict the motion vector of the articulated stylet using sparse optical flow.”; Gormley paragraph [0168], “The training part was implemented by feeding the annotated image to Keras implementation of YOLOv3. The version for Keras was 2.2.4 and this version runs TensorFlow 1.15 on the backend. The dataset was created with an annotation software, VoTT R (Microsoft, Redmond, Wash.).”; See also highlighted portions of in the appendix provided herein, which describe VOTT labeling images for classification and detection.).
Regarding Claim 17, Gormley in view of Ono ‘659 teaches the features of Claim 11, as described above.
Gormley further discloses wherein the controller is further configured to: identify at least one photographing point corresponding to the identified at least one second body part; generate the second control signal, based on the at least one photographing point and the relative position information; and photograph an image when a position of the front-end portion corresponds to the at least one photographing point (see paragraphs [0093] – [0094] and [0144] of Gormley, as cited in the rejection of Claim 11, that describe continuous/repetitive photography of body parts).
Regarding Claim 18, Gormley in view of Ono ‘659 teaches the features of Claim 11, as described above.
Gormley further discloses further comprising a curved portion connected to the front-end portion (Gormley FIG. 1A, curved portion of distal end of catheter tube 100), wherein the controller is further configured to control the driver based on the first control signal and adjust a tension of at least one wire connected to the driver so as to adjust, based on a curve of the curved portion, the rotation angle of the front-end portion (Gormley FIG. 1A, catheter tube 100; Gormley FIGs. 5A-5B; Gormley paragraph [0110], “Various mechanisms of control may be carried out by the robotic control engine 424 to control the movement of the articulated distal end of the articulated stylet. In some embodiments, the mechanism of control may include a pull-wire type configuration where multiple wires are attached at different points in the catheter tube around locations (e.g., articulation joints) where movement (e.g., bending/flexing) of the catheter tube is desired.”).
Regarding Claim 21, Gormley in view of Ono ‘659 teaches the features of Claim 1, as described above.
Gormley discloses wherein the first control signal is generated for steering the front-end portion to correspond to the at least one first body part (Gormley FIG. 4A, robotic control and display center (RCDC) 400; Gormley paragraph [0144], ” the stylet of the robot extending from the RCDC would be placed through one end of the endotracheal tube to be inserted and brought out the other end. The stylet would then be placed either in the nostril of the patient (either the left or right nostril) or in the mouth of a patient (alongside a standard oral-pharyngeal tube)… Using images obtained from the visual and topographic cameras at the tip of the stylet, the computer's algorithm would begin to recognize structure in the nasopharynx or oropharynx (depending on the site of insertion) and given these images the robot would direct the stylet down the pharynx into the larynx…and through the use of the actuators and motors that control all of its degrees of freedom, steer the stylet anteriorly through the larynx and through the vocal chords into the trachea. This will all be done using computer vision as a guide, without input required from any clinician at the patient's side. The decisions guiding the direction of the stylet will all be automated through the computer algorithm and controlled through the mechanical system of the device.”), and wherein the second control signal is generated for photographing the at least one photographing point corresponding to the at least one second body part (Gormley paragraph [0093], “transceiver 106 may also wirelessly transmit imaging data (e.g., topographic image data, still image data, and/or video data) captured by the imaging device 108 to the remote device. The transceiver 106 may transmit this data to the remote device both during the device placement process (e.g., as the catheter tube 100 is automatically driven to a target location)”; Gormley paragraph [0094], “the image data generated by the imaging device 108 may be used to guide the placement of the catheter tube 100”; Gormley paragraph [0144], “Using images obtained from the visual and topographic cameras at the tip of the stylet, the computer's algorithm would begin to recognize structure in the nasopharynx or oropharynx (depending on the site of insertion) and given these images the robot would direct the stylet down the pharynx into the larynx. At this point the epiglottis will come into the sight of the robot, which will be recognized. The algorithm will recognize the juncture of the larynx anteriorly and the esophagus posteriorly and through the use of the actuators and motors that control all of its degrees of freedom”; Examiner interprets this continuous capturing of images of interim landmarks while traversing to the target during the automatic drive of the catheter tube disclosed by Gormley as disclosing the presently claimed steps of 1) identifying an interim second body part from the image based on the pre-trained model and 2) adjusting the movement of the endoscope by use of this interim second body part image.).
Regarding Claim 22, Gormley in view of Ono ‘659 teaches the features of Claim 1, as described above.
Ona ‘659 teaches wherein the at least one photographing point is associated with a second pose of the front-end portion (Ono ‘659 FIG. 1, position identifying section; Ono ‘659 FIG. 4A, first imaging position 301-1 and second imaging position 301-2 relative to the subject position 300 shown in Ono ‘659 FIG. 4B; Ono ‘659 paragraph [0063], “if the object captured at the position 301-1 in the first image is captured at the position 301-2in the second image, the position identifying section 150 can identify the distance to the subject based on the difference between the position 301-1 and the position 301-2”), and wherein the calculating of the relative pose information comprises calculating a change from the first pose to the second pose corresponding to the at least one photographing point (Ono ‘659 FIG. 1, position identifying section; Ono ‘659 FIG. 4A, first imaging position 301-1 and second imaging position 301-2 relative to the subject position 300 shown in Ono ‘659 FIG. 4B; Ono ‘659 paragraph [0063], “if the object captured at the position 301-1 in the first image is captured at the position 301-2in the second image, the position identifying section 150 can identify the distance to the subject based on the difference between the position 301-1 and the position 301-2”).
Regarding Claim 23, Gormley in view of Ono ‘659 teaches the features of Claim 1, as described above.
Gormley further discloses wherein the second control signal is generated after the front-end portion is steered, by the first control signal, toward the at least one first body part, and wherein the second control signal is generated for photographing the at least one photographing point (Gormley paragraph [0093], “transceiver 106 may also wirelessly transmit imaging data (e.g., topographic image data, still image data, and/or video data) captured by the imaging device 108 to the remote device. The transceiver 106 may transmit this data to the remote device both during the device placement process (e.g., as the catheter tube 100 is automatically driven to a target location)”; Gormley paragraph [0094], “the image data generated by the imaging device 108 may be used to guide the placement of the catheter tube 100”; Gormley paragraph [0144], “Using images obtained from the visual and topographic cameras at the tip of the stylet, the computer's algorithm would begin to recognize structure in the nasopharynx or oropharynx (depending on the site of insertion) and given these images the robot would direct the stylet down the pharynx into the larynx. At this point the epiglottis will come into the sight of the robot, which will be recognized. The algorithm will recognize the juncture of the larynx anteriorly and the esophagus posteriorly and through the use of the actuators and motors that control all of its degrees of freedom”; Examiner interprets this continuous capturing of images of interim landmarks while traversing to the target during the automatic drive of the catheter tube disclosed by Gormley as disclosing the presently claimed steps of 1) identifying an interim second body part from the image based on the pre-trained model and 2) adjusting the movement of the endoscope by use of this interim second body part image.).
Regarding Claim 24, Gormley in view of Ono ‘659 teaches the features of Claim 1, as described above.
Gormley further discloses wherein the changes between the first image and the second image comprise a change in image disparity, and wherein the relative pose information is calculated based on the change in image disparity and an angle change of the front-end portion (Gormley paragraph [0093], “transceiver 106 may also wirelessly transmit imaging data (e.g., topographic image data, still image data, and/or video data) captured by the imaging device 108 to the remote device. The transceiver 106 may transmit this data to the remote device both during the device placement process (e.g., as the catheter tube 100 is automatically driven to a target location)”; Gormley paragraph [0094], “the image data generated by the imaging device 108 may be used to guide the placement of the catheter tube 100”; Gormley paragraph [0144], “Using images obtained from the visual and topographic cameras at the tip of the stylet, the computer's algorithm would begin to recognize structure in the nasopharynx or oropharynx (depending on the site of insertion) and given these images the robot would direct the stylet down the pharynx into the larynx. At this point the epiglottis will come into the sight of the robot, which will be recognized. The algorithm will recognize the juncture of the larynx anteriorly and the esophagus posteriorly and through the use of the actuators and motors that control all of its degrees of freedom”; Examiner interprets this continuous endoscopic angle change and changes to captured images during the automatic drive of the catheter tube disclosed by Gormley as disclosing the presently claimed steps of 1) acquiring multiple images while traversing through the patient, and 2) adjusting the movement of the endoscope by use of calculated rotation angle and passing landmarks shown in the images).
Claims 4 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Gormley et al. (US PGPUB 2021/0059607 – “Gormley”) in view of Ono (US PGPUB 2010/0079659 – “Ono ‘659”) and Ikai et al. (US PGPUB 2014/0155709 – “Ikai”).
Regarding Claim 4, Gormley in view of Ono ‘659 teaches the features of Claim 1, as described above.
Gormley in view of Ono ‘659 does not explicitly teach wherein the calculating of the relative pose information comprises: identifying, based on the acquired image and brightness information with respect to an illuminator, a brightness difference on the image; estimating, based on the identified brightness difference, a distance between the front-end portion and the at least one second body part; and calculating the relative pose information based on the estimated distance.
Ikai teaches wherein the calculating of the relative pose information comprises: identifying, based on the acquired image and brightness information with respect to an illuminator, a brightness difference on the image; estimating, based on the identified brightness difference, a distance between the front-end portion and the at least one second body part; and calculating the relative pose information based on the estimated distance (Ikai FIG. 35, image processor 25 and control unit 22 in guidance device 30 for capsule endoscope 300; Ikai paragraph [0312], “the distance between the capsule endoscope 300 and the image-capturing target can be determined by the brightness of the in-vivo image in which the image-capturing target appears. More specifically, the farther the image-capturing target is away from the capsule endoscope 300, the darker the in-vivo image becomes. The closer the image-capturing target is to the capsule endoscope 300, the brighter the in-vivo image becomes. Accordingly, the control unit 22 causes the image processor 25 to calculate the brightness from the pixel value of a pixel constituting each in-vivo image based on the image data received from the capsule endoscope 300. The command information generation unit 31 generates, as command information, distance information corresponding to the brightness”).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the use of Ikai’s light-based range finder with the method taught by Gormley in view of Ono ‘659. A person having ordinary skill in the art would be motivated to combine these prior art elements according to known methods to yield the predictable result of a method that is able to determine the distance between the endoscope and the target, in order to determine how much farther the endoscope needs to travel, and to avoid a collision between the endoscope and the target.
Regarding Claim 14, Gormley in view of Ono ‘659 teaches the features of Claim 11, as described above.
Gormley in view of Ono ‘659 does not explicitly teach wherein the controller is further configured to: identify, based on the acquired image and brightness information with respect to an illuminator, a brightness difference on the image; estimate, based on the identified brightness difference, a distance between the front-end portion and the at least one second body part; and calculate the relative pose information based on the estimated distance.
Ikai teaches wherein the controller is further configured to: identify, based on the acquired image and brightness information with respect to an illuminator, a brightness difference on the image; estimate, based on the identified brightness difference, a distance between the front-end portion and the at least one second body part; and calculate the relative pose information based on the estimated distance (Ikai FIG. 35, image processor 25 and control unit 22 in guidance device 30 for capsule endoscope 300; Ikai paragraph [0312], “the distance between the capsule endoscope 300 and the image-capturing target can be determined by the brightness of the in-vivo image in which the image-capturing target appears. More specifically, the farther the image-capturing target is away from the capsule endoscope 300, the darker the in-vivo image becomes. The closer the image-capturing target is to the capsule endoscope 300, the brighter the in-vivo image becomes. Accordingly, the control unit 22 causes the image processor 25 to calculate the brightness from the pixel value of a pixel constituting each in-vivo image based on the image data received from the capsule endoscope 300. The command information generation unit 31 generates, as command information, distance information corresponding to the brightness”).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the use of Ikai’s light-based range finder with the method taught by Gormley in view of Ono ‘659. A person having ordinary skill in the art would be motivated to combine these prior art elements according to known methods to yield the predictable result of a method that is able to determine the distance between the endoscope and the target, in order to determine how much farther the endoscope needs to travel, and to avoid a collision between the endoscope and the target.
Claims 5 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Gormley et al. (US PGPUB 2021/0059607 – “Gormley”) in view of Ono (US PGPUB 2010/0079659 – “Ono ‘659”) and Inoue (US PGPUB 2016/0302653 – “Inoue”).
Regarding Claim 5, Gormley in view of Ono ‘659 teaches the features of Claim 1, as described above.
Gormley in view of Ono ‘659 does not explicitly teach wherein the acquiring of the image of the upper gastrointestinal tract comprises: acquiring a first image at a first point; and acquiring a second image at a second point, and the calculating of the relative pose information comprises: estimating a distance between the front-end portion and the at least one second body part based on i) an angle change of the front-end portion between the first point and the second point and ii) a change between the first image and the second image; and calculating the relative pose information.
Inoue teaches wherein the acquiring of the image of the upper gastrointestinal tract comprises: acquiring a first image at a first point; and acquiring a second image at a second point (Inoue FIG. 1, imaging unit 28 at distal end of endoscope 2; Inoue FIGs. 11A and 11B, showing first and second orientation of imaging unit 28 while capturing image of subject of interest shown in Inoue FIG. 12), and the calculating of the relative pose information comprises: estimating a distance between the front-end portion and the at least one second body part based on i) an angle change of the front-end portion between the first point and the second point and ii) a change between the first image and the second image; and calculating the relative pose information (Inoue FIG. 10, step S107; Inoue paragraph [0078], “In S107, the distance to the subject of interest is computed on the basis of the first image and the second image taken in S102 and S105.”).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to utilize Inoue’s parallax distance calculation with the method taught by Gormley in view of Ono ‘659. A person having ordinary skill in the art would be motivated to combine these prior art elements according to known methods to yield the predictable result of a method that is able to determine the distance between the endoscope and the target, in order to determine how much farther the endoscope needs to travel, and to avoid a collision between the endoscope and the target.
Regarding Claim 15, Gormley in view of Ono ‘659 teaches the features of Claim 11, as described above.
Gormley in view of Ono ‘659 does not explicitly teach wherein the controller is further configured to: acquire a first image at a first point and a second image at a second point; and based on i) an angle change of the front-end portion between the first point and the second point and ii) a change between the first image and the second image, estimate a distance between the front-end portion and the at least one second body part and calculate the relative pose information.
Inoue teaches wherein the controller is further configured to: acquire a first image at a first point and a second image at a second point (Inoue FIG. 1, imaging unit 28 at distal end of endoscope 2; Inoue FIGs. 11A and 11B, showing first and second orientation of imaging unit 28 while capturing image of subject of interest shown in Inoue FIG. 12); and based on i) an angle change of the front-end portion between the first point and the second point and ii) a change between the first image and the second image, estimate a distance between the front-end portion and the at least one second body part and calculate the relative pose information (Inoue FIG. 10, step S107; Inoue paragraph [0078], “In S107, the distance to the subject of interest is computed on the basis of the first image and the second image taken in S102 and S105.”).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to utilize Inoue’s parallax distance calculation with the method taught by Gormley in view of Ono ‘659. A person having ordinary skill in the art would be motivated to combine these prior art elements according to known methods to yield the predictable result of a method that is able to determine the distance between the endoscope and the target, in order to determine how much farther the endoscope needs to travel, and to avoid a collision between the endoscope and the target.
Claims 9-10 are rejected under 35 U.S.C. 103 as being unpatentable over Gormley et al. (US PGPUB 2021/0059607 – “Gormley”) in view of Ono (US PGPUB 2010/0079659 – “Ono ‘659”) and Rabindran et al. (US PGPUB 2019/0143506 – “Rabindran”).
Regarding Claim 9, Gormley in view of Ono ‘659 teaches the features of Claim 7, as described above.
Gormley in view of Ono ‘659 does not explicitly teach generating torque feedback and transmitting the torque feedback to a manipulator via the driver.
Rabindran teaches generating torque feedback and transmitting the torque feedback to a manipulator via the driver (Rabindran FIG. 1A, control system 20, medical tool 14, manipulator assembly 12; Rabindran paragraph [0043], “control system 20 may include one or more servo controllers that receive force and/or torque feedback from the medical tool 14 or from the manipulator assembly 12. Responsive to the feedback, the servo controllers transmit signals to the operator input system 16. The servo controller(s) may also transmit signals that instruct the manipulator assembly 12 to move the medical tool(s) 14 and/or 15 which extends into an internal procedure site within the patient body via openings in the body.”).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the use of Rabindran’s torque feedback with the method taught by Gormley in view of Ono ‘659. A person having ordinary skill in the art would be motivated to combine these prior art elements according to known methods to yield the predictable result of a method capable of local/autonomous control of movement of an endoscope (see Rabindran paragraph [0043], “the controller and manipulator assembly are provided as part of an integrated system such as a teleoperational arm cart positioned proximate to the patient's body during the medical procedure”).
Regarding Claim 10, Gormley in view of Ono ‘659 and Rabindran teaches the features of Claim 9, as described above.
Rabindran further teaches wherein the torque feedback has a positive correlation with a target rotation angle for moving the front-end portion according to the first control signal with respect to the photographing of the image (Rabindran paragraph [0046], “manipulator assembly 12 shown provides for the manipulation of…a medical tool 28 including an imaging device (e.g., medical tool 15), such as a stereoscopic endoscope used for the capture of images of…the site of the procedure (also called “work site”)”; Rabindran paragraph [0075], “kinematic mapping unit 311 receives the master torque feedback command signals from the joint control unit 318, and generates the corresponding Cartesian force at the tip of the tool 250 relative to the camera frame of the endoscope using the slave kinematic configuration and the previously calculated slave fulcrum (e.g., remote center 256) position information”).
Claims 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Gormley et al. (US PGPUB 2021/0059607 – “Gormley”) in view of Ono (US PGPUB 2010/0079659 – “Ono ‘659”), Ono et al. (US PGPUB 2014/0203170 – “Ono ’170”), and Rabindran et al. (US PGPUB 2019/0143506 – “Rabindran”).
Regarding Claim 19, Gormley in view of Ono ‘659 teaches the features of Claim 17, as described above.
Gormley in view of Ono ‘659 does not explicitly teach a manipulator including a curved steering portion.
Ono ‘170 teaches a manipulator (Ono ‘170 FIG. 1, operation unit 22) including a curved steering portion (Ono ‘170 FIG. 1, curving knob 221 having a curved shape).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Ono ‘170’s curving knob with the endoscope device taught by Gormley in view of Ono ‘659. A person having ordinary skill in the art would be motivated to combine these prior art elements according to known methods to yield the predictable result of an endoscope having a proximal controller that is easy for a user to feel and control without looking, based on the distinct/rounded shape of the controller.
Gormley in view of Ono ‘650 and Ono ‘170 does not explicitly teach wherein the controller is further configured to generate torque feedback and transmit the torque feedback to the curved steering portion via the driver.
Rabindran teaches wherein the controller is further configured to generate torque feedback and transmit the torque feedback to the curved steering portion via the driver (Rabindran FIG. 1A, control system 20, medical tool 14, manipulator assembly 12; Rabindran paragraph [0043], “control system 20 may include one or more servo controllers that receive force and/or torque feedback from the medical tool 14 or from the manipulator assembly 12. Responsive to the feedback, the servo controllers transmit signals to the operator input system 16. The servo controller(s) may also transmit signals that instruct the manipulator assembly 12 to move the medical tool(s) 14 and/or 15 which extends into an internal procedure site within the patient body via openings in the body.”).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the use of Rabindran’s torque feedback with the endoscopic device taught by Gormley in view of Ono ‘659 and Ono ‘170. A person having ordinary skill in the art would be motivated to combine these prior art elements according to known methods to yield the predictable result of a method capable of local/autonomous control of movement of an endoscope (see Rabindran paragraph [0043], “the controller and manipulator assembly are provided as part of an integrated system such as a teleoperational arm cart positioned proximate to the patient's body during the medical procedure”).
Regarding Claim 20, Gormley in view of Ono ‘659, Ono ‘170, and Rabindran teaches the features of Claim 19, as described above.
Rabindran further teaches wherein the torque feedback has a positive correlation with a target rotation angle for moving the front-end portion according to the first control signal with respect to the photographing of the image (Rabindran paragraph [0046], “manipulator assembly 12 shown provides for the manipulation of…a medical tool 28 including an imaging device (e.g., medical tool 15), such as a stereoscopic endoscope used for the capture of images of…the site of the procedure (also called “work site”)”; Rabindran paragraph [0075], “kinematic mapping unit 311 receives the master torque feedback command signals from the joint control unit 318, and generates the corresponding Cartesian force at the tip of the tool 250 relative to the camera frame of the endoscope using the slave kinematic configuration and the previously calculated slave fulcrum (e.g., remote center 256) position information”).
Claim 25 is rejected under 35 U.S.C. 103 as being unpatentable over Gormley et al. (US PGPUB 2021/0059607 – “Gormley”) in view of Ono (US PGPUB 2010/0079659 – “Ono ‘659”) and Uyama et al. (US PGPUB 2021/0244260 – “Uyama”).
Regarding Claim 25, Gormley in view of Ono ‘659 teaches the features of Claim 1, as described above.
Gormley in view of Ono ‘659 does not explicitly teach wherein the calculating of the relative pose information comprises calculating the distance to be proportional to: (i) a rotational radius of the front-end portion, (ii) an angle change of the front-end portion between the first image and the second image, and (iii) a focal distance of an image sensor of the front-end portion.
Uyama teaches wherein the calculating of the relative pose information comprises calculating the distance to be proportional to: (i) a rotational radius of the front-end portion, (ii) an angle change of the front-end portion between the first image and the second image, and (iii) a focal distance of an image sensor of the front-end portion (Uyama FIG. 21, microscopic surgery system 5300; Uyama paragraph [0208], the drive of the arm unit 5309 may be controlled so as to perform a pivot operation. Here, the pivot operation is an operation of moving the microscope unit 5303 so that the optical axis of the microscope unit 5303 constantly faces a predetermined point in space (hereinafter, referred to as a pivot point). With the pivot operation, it is possible to observe an identical observation position from various directions, enabling observation of the affected part in more detail. In a case where focal length adjustment is disabled in the microscope unit 5303, it is preferable to perform the pivot operation with a fixed distance between the microscope unit 5303 and the pivot point. In this case, the distance between the microscope unit 5303 and the pivot point is only required to be adjusted to a fixed focal length of the microscope unit 5303. With this configuration, the microscope unit 5303 will move on a hemisphere (schematically illustrated in FIG. 21) having a radius corresponding to the focal length centered on the pivot point, leading to acquisition of a clear image even when the observation direction is changed. In contrast, in a case where focal length adjustment is enabled in the microscope unit 5303, it is allowable to perform the pivot operation with a variable distance between the microscope unit 5303 and the pivot point. In this case, for example, it is allowable to have a configuration in which the control device 5317 calculates the distance between the microscope unit 5303 and the pivot point based on the information regarding the rotation angle of individual joints detected by the encoder, and automatically adjusts the focal length of the microscope unit 5303 based on the calculation result.“; Examiner interprets Uyama’s method of pivoting and longitudinally moving the microscope unit towards the target as making the distance to the target proportional to: (i) a rotational radius of the front-end portion, (ii) an angle change of the front-end portion between the first image and the second image, and (iii) a focal distance of an image sensor of the front-end portion).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Barreto’s method for calculating a distance between the image sensor and the target with the method taught by Gormley in view of Ono ‘659. A person having ordinary skill in the art would be motivated to combine these
prior art elements according to known methods to yield the predictable result of a method that adjusts the rotation, angle, and focal distance of the image sensor relative to the target, in order to capture a clear and focused image of the target.
Claim 26 is rejected under 35 U.S.C. 103 as being unpatentable over Gormley et al. (US PGPUB 2021/0059607 – “Gormley”) in view of Ono (US PGPUB 2010/0079659 – “Ono ‘659”), Uyama et al. (US PGPUB 2021/0244260 – “Uyama”), and Harris et al. (US PGPUB 2009/0189972 – “Harris”).
Regarding Claim 26, Gormley in view of Ono ‘659 and Uyama teaches the features of Claim 25, as described above.
Gormley in view of Ono ‘659 and Uyama does not explicitly teach wherein the calculating of the relative pose information further comprises calculating the distance to be inversely related to a pixel displacement of the identified second body part between the first image and the second image, the pixel displacement representing a disparity of the second body part on a display.
Harris teaches wherein the calculating of the relative pose information further comprises calculating the distance to be inversely related to a pixel displacement of the identified second body part between the first image and the second image, the pixel displacement representing a disparity of the second body part on a display (Harris paragraph [0241], “In the event a laser distance sensor (or sonar or other distance sensing device) is mounted on camera 101, one possible calibration technique includes the steps of (1) placing a known measurement scale in the field of view of the camera and at a selected distance from the laser distance sensor, say 50 mm; (2) examining the display screen (typically 1280.times.720 pixels) on which the image of the measurement scale that is generated using signals from the camera is shown; (3) determining the number of display screen pixels in a selected reference unit of measurement on the measurement scale, say one mm, (4) successively moving the camera (and therefore the laser distance sensor) incrementally closer to (or farther from) the measurement scale (while retaining the scale in the field of view of the camera) and recording the number of pixels equivalent to the selected reference unit of measurement of one mm for each distance of the laser sensor from the measurement scale, i.e., for distances of 48 mm 46 mm, 44 mm, etc., (5) generating an algorithm that indicates the number of pixels in the display screen 23 (FIG. 31) in a mm for a particular distance of the sensor from the measurement scale or from another object; and (6) using the algorithm in controller 30 and data in memory 29 (FIG. 26) to calculate a distance (in pixels) of one mm on the display screen (typically 1280.times.720 pixels) when the laser distance sensor (or other sensor) is a particular distance from a target….For example, when the camera is closer to a target, a distance of one mm will take up a greater number of pixels on the display screen 23 (FIG. 31). When the camera is further from a target, a distance of one mm will require a lesser number of pixels on the display screen 23.”; Examiner interprets this passage for teaching the known optical property of a particular object changing size, and thus requiring a different number of pixels on an image sensor, as that object is moved towards a camera and vice versa).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to utilize Harris’ method of determining a distance to an object based on the size of that object’s image on an image sensor with the method taught by Gormley in view of Ono ‘659 and Uyama. A person having ordinary skill in the art would be motivated to combine these prior art elements according to known methods to yield the predictable result of a method that determines a distance between a camera and a target using only captured images and an algorithm thereon, in order to have a redundant distance measuring method to validate the method taught by Ono ‘659.
Response to Arguments
Applicant’s arguments, see page 10, filed March 31, 2026, with respect to the rejection of Claim 16 under 35 U.S.C. 112(b) have been fully considered and are persuasive, in view of the present amendments to Claim 16. The rejection of Claim 16 under 35 U.S.C. 112(b) has been withdrawn.
However, a new rejection under 35 U.S.C. 112(b) is now made against new Claim 26, as described above.
Applicant’s arguments, see page 12, filed March 31, 2026, with respect to the anticipatory double-patenting rejections of Claims 1 and 11 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new obviousness-type non-statutory double patenting rejection of Claims 1 and 11 is made in view of Ono (US PGPUB 2010/0079659 – “Ono ‘659”), as described above. Thus, Claims 1-2, 4-12, and 14-26 stand rejected based on obviousness-type non-statutory double patenting, as described above.
Applicant’s arguments, see pages 10-12, filed March 31, 2026, with respect to the rejection of Claims 1 and 11 under 35 U.S.C. 102(a)(1) have been fully considered and are persuasive. The rejection of Claims 1 and 11 under 35 U.S.C. 102(a)(1) has been withdrawn. However, upon further consideration, a new ground(s) of rejection of Claims 1 and 11 is made under 35 U.S.C. 103 in view of Ono (US PGPUB 2010/0079659 – “Ono ‘659”). As such, Claims 1-2, 4-12, and 14-26 stand rejected under 35 U.S.C. 103 as described above.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIM BOICE whose telephone number is (571)272-6565. The examiner can normally be reached Monday-Friday 9:00am - 5:00pm Eastern.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anhtuan Nguyen can be reached at (571)272-4963. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JIM BOICE
Examiner
Art Unit 3795
/JAMES EDWARD BOICE/Examiner, Art Unit 3795
/ANH TUAN T NGUYEN/Supervisory Patent Examiner, Art Unit 3795
5/4/26
Prosecution Insights