DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 9/2/2025 has been entered.
Response to Amendment
Amendments to claim 1 of 9/2/2025 acknowledged and entered.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 2, 4, 5, 7, 8, 22, 24, 28-30 is/are rejected under 35 U.S.C. 103 as being unpatentable over DeFonzo (US 20200085516 A1) in view of Proksch (US 20210196382 A1).
Regarding claim 1, DeFonzo teaches A surgical device, comprising:
a processor configured to:
receive image data of a first scene within a field of view of a first scope device (fig. 22, element 230, [0135], endoscopi8c instrument 230) at least partially disposed within a natural body lumen or organ;
receive image data of a second scene within a field of view of a second scope (fig. 22, element 235, [0135], rigid laparoscopic instrument 235) device disposed within a body cavity and at least partially disposed outside of the natural body lumen or organ; and
generate a merged image (fig. 36, [0182] merged image with two separate images overlaid) of the first scene and the second scene,
DeFonzo does not explicitly teach the device wherein the merged image indicates a location of energy to be applied by an energy applying surgical instrument to surface on a first side of a tissue wall of the natural body lumen or organ at a surgical site relative to an intended interaction location of a first instrument on surface on a second side of the tissue wall in a subsequent procedure step at the surgical site, wherein the second side of the tissue wall is opposite the first side of the tissue wall.
However, Proksch teaches the device wherein the merged image (fig. 4a-d, element 400-1, [0051], augmented image 400-1) indicates a location of energy to be applied by an energy applying surgical instrument to surface on a first side of a tissue wall of the natural body lumen or organ at a surgical site (fig. 5d, element 510, [0068], select portion 510 may be a Doppler image showing subsurface elements on a second side of the lumen) relative to an intended interaction location of a first instrument on surface on a second side of the tissue wall (fig. 3a, element 306, [0071], portion of anatomy 306) in a subsequent procedure step at the surgical site, wherein the second side of the tissue wall is opposite the first side of the tissue wall ([0076] This may allow a surgical team member to put the imaging probe in a stationary state to trigger an augmentation of the visual image that may help the surgical team member determine and/or perform next steps of a medical pro).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the image of DeFonzo to include a next step overlay indicating an interaction location as taught in Proksch in order to identify key landmarks during the surgical process (Proksch [0071]).
Regarding claim 2, DeFonzo in view of Proksch teaches The surgical device of claim 1,
Further, DeFonzo teaches the device wherein the processor is further configured to output the merged image to a display (fig. 36, element 1090, [0182], laparoscopic view 1090).
Regarding claim 4, DeFonzo in view of Proksch teaches The surgical device of claim 1,
Further, Proksch teaches the device wherein the processor is further configured to generate a representation of the intended interaction location of the first instrument in the merged image ([0076] This may allow a surgical team member to put the imaging probe in a stationary state to trigger an augmentation of the visual image that may help the surgical team member determine and/or perform next steps of a medical pro).
Regarding claim 5, DeFonzo in view of Proksch teaches The surgical device system of claim 1,
Further, DeFonzo teaches the device wherein the first scene does not include the second scope device, and wherein the second scene does not include the first scope device (fig. 36, first scope device positioned within the bronchus, second scope outside of it).
Regarding claim 7, DeFonzo in view of Proksch teaches The surgical device of claim 1,
Further, DeFonzo teaches the device wherein the processor is further configured to determine, based on at least one of the image data of the first scene or the image data of the second scene, at least one of a location or an orientation of the first instrument relative to the first scope device, wherein the merged image comprises a depiction of at least a portion of the first instrument (fig. 35,36, element 1060, second view 1060 illustrating the position of the endoscope tip).
Regarding claim 8, DeFonzo in view of Proksch teaches The surgical device of claim 1,
Further, DeFonzo teaches the device wherein the processor is further configured to calculate, based on at least one of the image data of the first scene or the image data of the second scene, an insertion depth of the energy applying surgical instrument within tissue of the natural body lumen or organ (fig. 35,36, element 1060, second view 1060 illustrating the position of the endoscope tip).
Regarding claim 24, DeFonzo in view of Proksch teaches the surgical device of claim 1,
Further, DeFonzo teaches the device wherein the merged image comprises a live view of the first scene (fig. 36).
Regarding claim 25, DeFonzo in view of Proksch teach The surgical device of claim 24,
Further, DeFonzo teaches the device wherein the depiction of the second scene being illustrated so as to indicate that the second scene is not in the live view of the first scene comprises the depiction of the second scene being illustrated as one or more of: a dashed-line illustration of objects in the second scene, a semi-transparent illustration of the objects in the second scene, or a colored illustration of the objects in the second scene (fig. 36, solid line around first endoscope view 1050 delineates between the colored illustrations of first and second views).
Regarding claim 28, DeFonzo in view of Proksch teaches The surgical device of claim 1,
Further, Proksch teaches the device wherein the energy is applied by an energy applying surgical instrument to surface on a first side of a tissue wall during a first procedure step, the subsequent procedure step is a second procedure step, the intended interaction location is a first intended interaction location, and the processor is further configured to:
determine that the first procedure step has been completed ([0347], as procedure progresses, control circuit recognizes progress and removes the first overlay and displays a second distance indicator);
determine that the first instrument is at the intended interaction location ([0346], identifies pylorus and displays the overlay of energy device when positioned properly); and
update the merged image of the first scene and the second scene, wherein the updated merged image indicates the first intended interaction location of the first instrument relative to a second intended interaction location of a second instrument in a third procedure step at the surgical site, wherein the third procedure step is subsequent to the second procedure step ([0347], as procedure progresses, control circuit recognizes progress and removes the first overlay and displays a second distance indicator).
Regarding claim 30, DeFonzo in view of Proksch teaches The surgical device of claim 1,
Further, DeFonzo teaches the device wherein the first scope device is an endoscope and the second scope device is a laparoscope ([0135] endoscopic instrument 230, laparoscopic instrument 235).
Claim(s) 29 is/are rejected under 35 U.S.C. 103 as being unpatentable over DeFonzo in view of Proksch as applied to claim 1 in further view of Mumaw (US 20210196382 A1).
Regarding claim 29, DeFonzo in view of Proksch teaches The surgical device of claim 1,
DeFonzo in view of Proksch does not explicitly teach the device wherein the merged image depicts a starting point of an incision performed by the energy applying surgical instrument ([0346], overlay of end effector).
However, Mumaw teaches the device wherein the merged image depicts a starting point of an incision performed by the energy applying surgical instrument ([0346], overlay of end effector).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the merged image of DeFonzo to include procedural data as taught in Mumaw in order to identify the starting point of the procedure (Mumaw [0346]).
Claim(s) 6, 9, 10, 23 is/are rejected under 35 U.S.C. 103 as being unpatentable over DeFonzo in view of Proksch as applied to claim 1 in further view of Lu (US 20220175269 A1).
Regarding claim 6, DeFonzo in view of Proksch teaches The surgical device of claim 1, DeFonzo in view of Proksch does not explicitly teach a device wherein
the processor is further configured to determine a second interaction location of the first instrument or a second instrument based on one or more remaining steps in a procedure plan.
However, Lu teaches a device wherein
the processor is further configured to determine a second interaction location of the first instrument or a second instrument based on one or more remaining steps in a procedure plan (fig. 23, element 3600, [0261] radiopaque markers indicate areas of interest to be treated by the procedure plan).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the merged image of DeFonzo to include procedural data as taught in Lu in order to capture the areas to be treated via x-ray guidance of the scope (Lu [0261]).
Regarding claim 9, DeFonzo in view of Proksch teaches The surgical device of claim 1,
DeFonzo in view of Proksch does not explicitly teach the energy applying surgical instrument further includes a force sensor configured to sense a force applied to tissue of the natural body lumen or organ by the energy applying surgical instrument.
However, Lu teaches the energy applying surgical instrument further includes a force sensor configured to sense a force applied to tissue of the natural body lumen or organ by the energy applying surgical instrument ([0162, 170] pressure sensor location is tracked relative to imaging markers 1030. Pressure being a measure of force).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the merged image of DeFonzo to track positional depth as a measure of force as taught in Lu in order to display features alongside their recorded depth (Lu [0162]).
Regarding claim 10, DeFonzo in view of Proksch and Lu teaches The surgical device of claim 9,
Further, Lu teaches a system wherein
the controller is further configured to determine an insertion depth of the energy applying surgical instrument based on the sensed applied force ([0162, 170] pressure sensor location is tracked relative to imaging markers 1030. Pressure being a measure of force).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the merged image of Sato to track positional depth as a measure of force as taught in Lu in order to display features alongside their recorded depth (Lu [0162]).
Regarding claim 23, DeFonzo in view of Proksch teaches The surgical device of claim 1,
Further, Proksch teaches the device wherein the processor is further configured to:
determine a first location of the first scope device based on a first view of an anatomical landmark in the first scene ([0076] This may allow a surgical team member to put the imaging probe in a stationary state to trigger an augmentation of the visual image that may help the surgical team member determine and/or perform next steps of a medical pro);
Further, DeFonzo teaches generate a virtual image of the second scope device on the first scene based on at least one of the first or second locations, wherein the merged image comprises the virtual image of the second scope device (fig. 36, virtual image of the anatomic site is generated).
DeFonzo in view of Proksch does not disclose the device configured to determine a second location of the second scope device based on a second view of the anatomical landmark in the first scene;
However, Lu teaches the device configured to determine a second location of the second scope device based on a second view of the anatomical landmark in the first scene (fig. 21, element p1/p2, [0248] dual x-rays are used to determine the position of points in space using coordinate form);
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the merged image of DeFonzo to track coordinate position as taught in Lu in order to assist in development of machine learning models (Lu [0249]).
Regarding claim 26, DeFonzo in view of Proksch teaches The surgical device of claim 24,
DeFonzo in view of Proksch do not explicitly teach the device wherein the processor is further configured to:
receive, from a healthcare provider, a request to switch the merged image from a viewpoint of the first scope device to a viewpoint of the second scope device; and
based on the request, update the merged image to comprise a live view of the second scene and a depiction of the first scene, wherein the depiction of the first scene is illustrated so as to indicate that the first scene is not in the live view of the second scene.
However, Shelton teaches the device wherein the processor is further configured to:
receive, from a healthcare provider, a request to switch the merged image from a viewpoint of the first scope device to a viewpoint of the second scope device; and
based on the request, update the merged image to comprise a live view of the second scene and a depiction of the first scene, wherein the depiction of the first scene is illustrated so as to indicate that the first scene is not in the live view of the second scene (fig. 30, 33 [0303], operator can configure image to be either a POV of one camera or a virtual image).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the scopes of DeFonzo to accept user inputs to control image displayed in order to better represent the operation site (Shelton [0303]).
The applied reference has a common author with the instant application. Based upon the earlier effectively filed date of the reference, it constitutes prior art under 35 U.S.C. 102(a)(2).
This rejection under 35 U.S.C. 103 might be overcome by: (1) a showing under 37 CFR 1.130(a) that the subject matter disclosed in the reference was obtained directly or indirectly from the inventor or a joint inventor of this application and is thus not prior art in accordance with 35 U.S.C.102(b)(2)(A); (2) a showing under 37 CFR 1.130(b) of a prior public disclosure under 35 U.S.C. 102(b)(2)(B); or (3) a statement pursuant to 35 U.S.C. 102(b)(2)(C) establishing that, not later than the effective filing date of the claimed invention, the subject matter disclosed and the claimed invention were either owned by the same person or subject to an obligation of assignment to the same person or subject to a joint research agreement. See generally MPEP § 717.02.
Claim 26 is/are rejected under 35 U.S.C. 103 as being obvious over DeFonzo in view of Proksch as applied to claim 1 in further view of Sato (US 20080051655 A1) and Shelton (US 20210196384 A1).
Regarding claim 21, DeFonzo in view of Proksch teaches The surgical device of claim 1,
Further, Proksch teaches the device wherein the merged image depicts one or more of the first distance, second distance, or third distance (fig. 4c, element 412, [0052], depth lines 412).
Defonzo in view of Proksch does not explicitly teach wherein the processor is further configured to:
determine a first distance between the first scope device and the second scope device;
configured to determine a second distance between the first scope device and a tumor at the surgical site; and
determine, based on the first distance and the second distance, a third distance between the second scope device and the tumor, wherein the merged image depicts one or more of the first distance, second distance, or third distance.
However, Sato teaches wherein the processor is further configured to:
determine a first distance between the first scope device and the second scope device (fig. 21b, [0205], transducers 152a/b are used to determine positioning by reception of ultrasound signals to determine distance);
However, Shelton teaches a device configured to determine a second distance between the first scope device and a tumor at the surgical site (fig. 3, element dw, dy, dx, [0150], triangulation of depth dw is determined using known quantities dy and dx, critical structure 101 can be a tumor [0118]); and
determine, based on the first distance and the second distance, a third distance between the second scope device and the tumor (fig. 3, element dw, dy, dx, [0150], triangulation of depth dw is determined using known quantities dy and dx).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the scopes of DeFonzo to track the location of an energy applying device relative to the scope as taught in Sato in order to prevent unintended contact or injury (Sato [0207]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the scopes of Sato to triangulate the tumor location and distance to it in order to better inform a clinician of the tissue makeup and tool position (Shelton [0122]).
Claim 22 is/are rejected under 35 U.S.C. 103 as being obvious over DeFonzo in view of Proksch as applied to claim 1 in further view of Sato (US 20080051655 A1).
Regarding claim 22, DeFonzo in view of Proksch teaches The surgical device of claim 1,
Further, Sato teaches the device wherein the processor is further configured to:
receive first tracking data from the first scope device, second tracking data from the second scope device, and third tracking data from an instrument in the field of view of the second scope device (fig. 25a, [0212], first scope 212a is disposed with a camera for collecting and transmitting visual information of the scene, which could be used to track at least the positions of insertion portions 212a, 212b, and needle 154, which is an instrument. Further, distance between scopes on opposite sides of the intestinal wall IW is demonstrated in fig. 21a-d, wherein scopes 12a and 12b determine distance via ultrasonic sensors [0201]);
determine a first distance between the first scope device and the second scope device based on the first and second tracking data ([0201] distance between two scopes on opposite sides of a lumen determined through ultrasound);
determine a second distance between the second scope device and the instrument based on the second and third tracking data (fig. 21d, element 154, [0207], distance from needle to endoscope is recorded to prevent unintended injury);
determine a third distance between the first scope device and the instrument based on the first and second distances (fig. 21d, element 154, [0207], distance from needle to endoscope is recorded to prevent unintended injury); and
generate a virtual image of the instrument on the first scene based on at least one of the first, second, or third distances, wherein the merged image comprises the virtual image of the instrument (fig. 21d, element 154, [0207], distance from needle to endoscope is recorded to prevent unintended injury).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the scopes of DeFonzo to track the location of an energy applying device relative to the scope as taught in Sato in order to prevent unintended contact or injury (Sato [0207]).
Claim 27 is/are rejected under 35 U.S.C. 103 as being obvious over DeFonzo in view of Proksch as applied to claim 1 in further view of Goshayes (US 20160015469 A1).
Regarding claim 27, DeFonzo in view of Proksch teaches The surgical device of claim 1,
DeFonzo in view of Proksch does not explicitly teach the device wherein the merged image comprises a virtual overlay of the second scene on a live video stream of the first scene, or a virtual overlay of the first scene on a live video stream of the second scene.
However, Goshayes teaches the device wherein the merged image comprises a virtual overlay of the second scene on a live video stream of the first scene, or a virtual overlay of the first scene on a live video stream of the second scene (fig. 8, element 148, [0057], graphic symbols 148 are an overlay onto image 146, which indicates the position of the endoscopic image 142 in the merged image).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the scopes of DeFonzo to represent the first scene in the second scene as taught in Goshayes in order to help a user identify the location and procedural step being accomplished (Goshayes [0057]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TIMOTHY TUAN LUU whose telephone number is (703)756-4592. The examiner can normally be reached Monday-Tuesday, Thursday-Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Carey can be reached on 5712707235. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TIMOTHY TUAN LUU/Examiner, Art Unit 3795
/MICHAEL J CAREY/Supervisory Patent Examiner, Art Unit 3795
Prosecution Insights