DETAILED ACTION
Claims 1-20 are pending in this application. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 8, 11-12 19 and 20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. Claims 8, 19 and 20 are drawn to functional descriptive material as it is directed towards a “program” used in the surgery support system of Claim 1, Claim 8 and Claim 2, respectively. Upon further consideration, it appears that the elements recited in Claim 1 include an embodiment that appears to be software only as exemplified in paragraphs [0029]-[0033]. In light of the fact that these particular claims are directed towards the functional descriptive material itself, the broadest reasonable interpretation of a claim drawn to a “program” typically covers forms of non-transitory tangible media as well as transitory propagating signals per se, making the recited claim language directed towards non-statutory subject matter such as a “signal”.
“A transitory, propagating signal … is not a “process, machine, manufacture, or composition of matter.” Those four categories define the explicit scope and reach of subject matter patentable under 35 U.S.C. § 101; thus, such a signal cannot be patentable subject matter.” (In re Nuijten, 84 USPQ2d 1495 (Fed. Cir. 2007)).
Likewise, claims 11-12 and 20 are dependent upon Claim 7 and fail to overcome the problem recited for claim 7. Because the full scope of the claim as properly read in light of the disclosure appears to encompass non-statutory subject matter (i.e., because the specification is silent to the exact embodiment of a computer readable medium, it is interpreted as including the ordinary and customary meaning of computer readable medium covering both non-transitory media and transitory propagating signals, etc.) the claim as a whole is non-statutory. In view of the USPTO's Interim Examination Instructions for Evaluating Subject Matter Eligibility under 35 U.S.C. 101 (the "Guidelines"), and the Official Gazette Notice (1351 OG 212, made available February 23, 2010), the examiner suggests amending the claim to include the limitation "non-transitory" in order to exclude any non-statutory subject matter. Any amendment to the claim should be commensurate with its corresponding disclosure.
35 U.S.C. § 112 Sixth Paragraph - Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are: “unit” in claims 1-20.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Poulsen et al. (US PGPub 20140093852, filed September 27, 2013), hereby referred to as “Poulsen”, in view of Grupp et al. (US PGPub 20230196595, filed on December 19, 2022 with provisional priority dating to December 20, 2021), hereby referred to as “Grupp”.
Consider Claims 1 and 7.
Poulsen teaches:
1. Claim 1. (Original) A surgery support system comprising:/ Claim 7. (Original) A surgery support device comprising: (Poulsen: abstract, A model for practicing laparoscopic surgical skills is provided. The model includes a base having a plurality of practice stations at the upper surface. The practice stations include a cover having a first closed position in which a cavity is concealed beneath the cover and a second open position in which the cover is moved to uncover the cavity. The covers are connected to the surface in a number of ways to provide a variety of haptic responses useful in fine-tuning laparoscopic surgical skills. The cover is configured as a door hinged with or without a bias, a flexible flap, a sliding cover, a lid, and a penetrable sheet. An object for removal is hidden inside the cavity underneath the cover for practicing hand-to-hand transfer of instruments, use of both hands, switching instruments and determining and visualizing tissue planes in a laparoscopic environment. [0009]-[0011], [0014] A surgical training device 10 that is configured to mimic the torso of a patient such as the abdominal region is shown in FIG. 1. The surgical training device 10 provides a body cavity 12 substantially obscured from the user and configured for receiving simulated or live tissue as well as model organs or a training model of the like described in this invention.)
1. a surgical assistance device configured to assist use of a treatment tool inserted into a body cavity and having image capturing units configured to capture images inside the body cavity;
1. an image processing unit configured to process images obtained from the image capturing units; / 7. an image processing unit configured to process images (Poulsen: [0014] A surgical training device 10 that is configured to mimic the torso of a patient such as the abdominal region is shown in FIG. 1. The surgical training device 10 provides a body cavity 12 substantially obscured from the user and configured for receiving simulated or live tissue as well as model organs or a training model of the like described in this invention. The body cavity 12 is accessed via a tissue simulation region 14 that is penetrated by the user employing devices and instruments to practice surgical techniques on the tissue or organ model found located in the body cavity 12. Although the body cavity 12 is shown to be accessible through a tissue simulation region, a hand-assisted access device or single-site port device may be alternatively employed to access the body cavity 12. [0015] Still referencing FIG. 1, the surgical training device 10 includes a top cover 16 connected to and spaced apart from a base 18 by at least one leg 20. FIG. 1 shows a plurality of legs 20. The surgical training device 10 is configured to mimic the torso of a patient such as the abdominal region. The top cover 16 is representative of the anterior surface of the patient and the space between the top cover 16 and the base 18 is representative of an interior of the patient or body cavity where organs reside. The surgical trainer 10 is a useful tool for teaching, practicing and demonstrating various surgical procedures and their related instruments in simulation of a patient undergoing a surgical procedure. Surgical instruments are inserted into the cavity 12 through the tissue simulation region 14 as well as through pre-established apertures 22 in the top cover 16 or from the side between the top cover 16 and the base 18.)
1. and a display unit configured to display image data image-processed by the image processing unit, wherein the image capturing units are configured such that a plurality of image capturing units are provided to the surgical assistance device, each of the image capturing units captures an image so that inside of the body cavity appears in the image, (Paulsent: [0016] A video display monitor 28 that is hinged to the top cover 16 is shown in a closed orientation in FIG. 1. The video monitor 62 is connectable to a variety of visual systems for delivering an image to the monitor. For example, a scope inserted through one of the pre-established apertures 22 or a webcam located in the cavity 12 and used to observe the simulated procedure can be connected to the video monitor 28 and/or a mobile computing device to provide an image to the user. Also, audio recording or delivery means may also be provided and integrated with the trainer 10 to provide audio and visual capabilities. Means for connecting a portable memory storage device such as a flash drive, smart phone, digital audio or video player, or other digital mobile device is also provided to record training procedures and/or play back pre-recorded videos on the monitor for demonstration purposes. Of course, connection means for providing an audio visual output to a larger screen other than the monitor is provided. In another variation, the top cover 10 does not include a video display but includes means for supporting a laptop computer, a mobile digital device or tablet such as an IPAD® and connecting it by wire or wirelessly to the trainer.)
1. and when the treatment tool is inserted in the body cavity, any of the image capturing units captures an image so that the inside of the body cavity including the treatment tool appears in the image, / 7. obtained from a plurality of image capturing units of a surgical assistance device configured to assist use of a treatment tool inserted into a body cavity, (Paulsen: [0015] Various tools and techniques may be used to penetrate the top cover 16 to perform mock procedures on simulated organs or training models placed between the top cover 16 and the base 18. The base 18 includes a model-receiving area 24 or tray for staging or holding a training model. The model-receiving area 24 of the base 18 includes frame-like elements for holding the model (not shown) in place to keep it from sliding around while being manipulated by surgical instruments. To help retain the model on the base 18, a clip attached to a retractable wire is provided at locations 26. The retractable wire is extended and then clipped to hold the model in position substantially beneath the tissue simulation region 14. Other means for retaining the model include a patch of hook-and-loop type fastening material (VELCRO®) affixed to the base 18 in the model receiving area 24 such that it is removably connectable to a complementary piece of hook-and-loop type fastening material (VELCRO®) affixed to the model. [0016] A video display monitor 28 that is hinged to the top cover 16 is shown in a closed orientation in FIG. 1. The video monitor 62 is connectable to a variety of visual systems for delivering an image to the monitor. For example, a scope inserted through one of the pre-established apertures 22 or a webcam located in the cavity 12 and used to observe the simulated procedure can be connected to the video monitor 28 and/or a mobile computing device to provide an image to the user.)
1. and perform an obstructing object removal process of removing at least a part of an image of the treatment tool. / 7. and performs an obstructing object removal process of removing at least a part of an image of the treatment tool. (Poulsen: [0022] The third practice station 34 c includes a cover 52 connected to the top surface of the base 32 with a connector 54 and the cover 52 may further include a knob 56 connected to the outer surface of the cover 52. The connector 54 is a fastener that permits the cover 44 to slide, swivel or pivot about the fastener along the upper surface 36 of the base 32 to uncover a cavity 58 underneath the cover 52 with an object 60 optionally located inside the cavity 58. The object 60 removed from the cavity 58 may then be placed over a peg 38 that corresponds to the shape of the object 60. The cavity 58 is formed in the base 32 and may be any shape or size and depth. A deeper cavity may increase the difficulty level of extracting the object 60. The cavity 58 may also be sinuous or have a portion hidden from view by a wall such that the user would have to reach around the obstruction with an instrument to see if an object 60 is hiding behind the wall inside the cavity 58. The cavity 58 opens to the upper surface 36 such that when the cover 52 is moved from a first position covering the cavity 58 to a second position uncovering the cavity 58, the opening to the cavity 58 as well as the object 60 disposed inside the cavity 58 is revealed to the user. In this exercise, the user uses a surgical instrument such as a laparoscopic grasper inserted through an aperture 22, simulated tissue penetration region 14 or side of the trainer 10 to grab the knob 56 or cover 52 to turn it or slide it from a first closed position to a second open position. If a cavity 58 is provided underneath the cover 52, the user can then reach inside the cavity 58 and grab the object 60 with the same grasper or another grasper held in an opposite hand of the user and remove the object 60 from the cavity 58 and place it to the side or remove it from the trainer 10. Alternatively, the object 60 may then be placed over a peg 38 at the first station 34 a such that object 60 is placed over the peg 38 having a shape 76 that corresponds with the shape associated with the object 60.)
Even if Paulsen does not teach:
1. and wherein the image processing unit is configured to synthesize images obtained from respective image capturing units to generate a synthesized image, / 7. wherein the surgery support device synthesizes the images obtained from respective image capturing units to generate a synthesized image,
Grupp teaches:
1. Claim 1. (Original) A surgery support system comprising:/ Claim 7. (Original) A surgery support device comprising: (Grupp: abstract, Medical imaging systems, methods, and devices are disclosed herein. In some embodiments, an imaging system includes (i) a camera array configured to capture intraoperative image data of a surgical scene in substantially real-time and (ii) a processing device communicatively coupled to the camera array. The processing device can be configured to synthesize a three-dimensional (3D) image corresponding to a virtual perspective of the scene based on the intraoperative image data from the cameras. The imaging system is further configured to receive and/or store initial image data, such as medical scan data corresponding to a portion of a patient in the scene. The processing device can register the initial image data to the intraoperative image data, and overlay the registered initial image data over the corresponding portion of the 3D image of the scene to present a mediated-reality view. [0016]-[0018], [0025] FIG. 1 is a schematic view of an imaging system 100 (“system 100”) in accordance with embodiments of the present technology. In some embodiments, the system 100 can be a synthetic augmented reality system, a virtual-reality imaging system, an augmented-reality imaging system, a mediated-reality imaging system, and/or a non-immersive computational imaging system. In the illustrated embodiment, the system 100 includes a processing device 102 that is communicatively coupled to one or more display devices 104, one or more input controllers 106, and a camera array 110. In other embodiments, the system 100 can comprise additional, fewer, or different components.)
1. a surgical assistance device configured to assist use of a treatment tool inserted into a body cavity and having image capturing units configured to capture images inside the body cavity;
1. an image processing unit configured to process images obtained from the image capturing units; / 7. an image processing unit configured to process images (Grupp: [0025] FIG. 1 is a schematic view of an imaging system 100 (“system 100”) in accordance with embodiments of the present technology. In some embodiments, the system 100 can be a synthetic augmented reality system, a virtual-reality imaging system, an augmented-reality imaging system, a mediated-reality imaging system, and/or a non-immersive computational imaging system. In the illustrated embodiment, the system 100 includes a processing device 102 that is communicatively coupled to one or more display devices 104, one or more input controllers 106, and a camera array 110. In other embodiments, the system 100 can comprise additional, fewer, or different components. [0026] In the illustrated embodiment, the camera array 110 includes a plurality of cameras 112 (identified individually as cameras 112 a-112 n; which can also be referred to as first cameras) that can each capture images of a scene 108 (e.g., first image data) from a different perspective.)
1. and a display unit configured to display image data image-processed by the image processing unit, wherein the image capturing units are configured such that a plurality of image capturing units are provided to the surgical assistance device, each of the image capturing units captures an image so that inside of the body cavity appears in the image, (Grupp: [0030] In other embodiments, the image processing device 103 can generate the virtual camera perspective based only on the images captured by the cameras 112—without utilizing depth information from the depth sensor 114. For example, the image processing device 103 can generate the virtual camera perspective by interpolating between the different images captured by one or more of the cameras 112. [0031] The image processing device 103 can synthesize the output image from images captured by a subset (e.g., two or more) of the cameras 112 in the camera array 110, and does not necessarily utilize images from all of the cameras 112 [0033]-[0034] In some embodiments, functions attributed to the processing device 102, the image processing device 103, the registration processing device 105, and/or the tracking processing device 107 can be practically implemented by two or more physical devices. For example, in some embodiments a synchronization controller (not shown) controls images displayed by the projector 116 and sends synchronization signals to the cameras 112 to ensure synchronization between the cameras 112 and the projector 116 to enable fast, multi-frame, multicamera structured light scans. Additionally, such a synchronization controller can operate as a parameter server that stores hardware specific configurations such as parameters of the structured light scan, camera settings, and camera calibration data specific to the camera configuration of the camera array 110. The synchronization controller can be implemented in a separate physical device from a display controller that controls the display device 104, or the devices can be integrated together.)
1. and when the treatment tool is inserted in the body cavity, any of the image capturing units captures an image so that the inside of the body cavity including the treatment tool appears in the image, / 7. obtained from a plurality of image capturing units of a surgical assistance device configured to assist use of a treatment tool inserted into a body cavity, (Grupp: [0026] In the illustrated embodiment, the camera array 110 includes a plurality of cameras 112 (identified individually as cameras 112 a-112 n; which can also be referred to as first cameras) that can each capture images of a scene 108 (e.g., first image data) from a different perspective. The scene 108 can include for example, a patient undergoing surgery (e.g., spinal surgery) and/or another medical procedure. In other embodiments, the scene 108 can be another type of scene. The camera array 110 can further include dedicated object tracking hardware 113 (e.g., including individually identified trackers 113 a-113 n) that captures positional data of one more objects, such as an instrument 101 (e.g., a surgical instrument or tool) having a tip 109, to track the movement and/or orientation of the objects through/in the scene 108. In some embodiments, the cameras 112 and the trackers 113 are positioned at fixed locations and orientations (e.g., poses) relative to one another. For example, the cameras 112 and the trackers 113 can be structurally secured by/to a mounting structure (e.g., a frame) at predefined fixed locations and orientations. In some embodiments, the cameras 112 are positioned such that neighboring cameras 112 share overlapping views of the scene 108. In general, the position of the cameras 112 can be selected to maximize clear and accurate capture of all or a selected portion of the scene 108. Likewise, the trackers 113 can be positioned such that neighboring trackers 113 share overlapping views of the scene 108. Therefore, all or a subset of the cameras 112 and the trackers 113 can have different extrinsic parameters, such as position and orientation. [0064])
1. and wherein the image processing unit is configured to synthesize images obtained from respective image capturing units to generate a synthesized image, / 7. wherein the surgery support device synthesizes the images obtained from respective image capturing units to generate a synthesized image, (Grupp: [0030] The image processing device 103 can (i) receive the first image data captured by the cameras 112 (e.g., light field images, light field image data, RGB images) and depth information from the depth sensor 114 (e.g., the second image data captured by the depth cameras 118), and (ii) process the image data and depth information to synthesize (e.g., generate, reconstruct, render) a three-dimensional (3D) output image of the scene 108 corresponding to a virtual camera perspective. The output image can correspond to an approximation of an image of the scene 108 that would be captured by a camera placed at an arbitrary position and orientation corresponding to the virtual camera perspective. In some embodiments, the image processing device 103 can further receive and/or store calibration data for the cameras 112 and/or the depth cameras 118 and synthesize the output image based on the image data, the depth information, and/or the calibration data. More specifically, the depth information and the calibration data can be used/combined with the images from the cameras 112 to synthesize the output image as a 3D (or stereoscopic 2D) rendering of the scene 108 as viewed from the virtual camera perspective. In some embodiments, the image processing device 103 can synthesize the output image using any of the methods disclosed in U.S. patent application Ser. No. 16/457,780, titled “SYNTHESIZING AN IMAGE FROM A VIRTUAL PERSPECTIVE USING PIXELS FROM A PHYSICAL IMAGER ARRAY WEIGHTED BASED ON DEPTH ERROR SENSITIVITY,” and filed Jun. 28, 2019, which is incorporated herein by reference in its entirety. In other embodiments, the image processing device 103 can generate the virtual camera perspective based only on the images captured by the cameras 112—without utilizing depth information from the depth sensor 114. For example, the image processing device 103 can generate the virtual camera perspective by interpolating between the different images captured by one or more of the cameras 112. [0031] The image processing device 103 can synthesize the output image from images captured by a subset (e.g., two or more) of the cameras 112 in the camera array 110, and does not necessarily utilize images from all of the cameras 112. For example, for a given virtual camera perspective, the processing device 102 can select a stereoscopic pair of images from two of the cameras 112. In some embodiments, such a stereoscopic pair can be selected to be positioned and oriented to most closely match the virtual camera perspective. In some embodiments, the image processing device 103 (and/or the depth sensor 114) estimates a depth for each surface point of the scene 108 relative to a common origin to generate a point cloud and/or a 3D mesh that represents the surface geometry of the scene 108. Such a representation of the surface geometry can be referred to as a surface reconstruction, a 3D reconstruction, a 3D volume reconstruction, a volume reconstruction, a 3D surface reconstruction, a depth map, a depth surface, and/or the like. In some embodiments, the depth cameras 118 of the depth sensor 114 detect the structured light projected onto the scene 108 by the projector 116 to estimate depth information of the scene 108. In some embodiments, the image processing device 103 estimates depth from multiview image data from the cameras 112 using techniques such as light field correspondence, stereo block matching, photometric symmetry, correspondence, defocus, block matching, texture-assisted block matching, structured light, and the like, with or without utilizing information collected by the depth sensor 114. In other embodiments, depth may be acquired by a specialized set of the cameras 112 performing the aforementioned methods in another wavelength. [0032])
1. and perform an obstructing object removal process of removing at least a part of an image of the treatment tool. / 7. and performs an obstructing object removal process of removing at least a part of an image of the treatment tool. (Grupp: [0084] At block 1094, the method 1090 can determine that the spine of the patient is accessible for a surgical procedure based on the virtual model. For example, the system 100 (e.g., the processing device 102) can detect that some or all of a target vertebra (e.g., labeled as “bone”) is visible to the cameras 112. In an open surgical procedure, the system 100 can detect that some or all of the target vertebra is visible to the cameras 112 in the camera array 110 positioned above the patient while, in a minimally invasive surgical procedure and/or a percutaneous surgical procedure, the system 100 can detect that some or all of the target vertebra is visible to the camera array 110 and/or a percutaneously inserted camera/camera array. In some embodiments, the system 100 can detect that the spine is accessible for the surgical procedure by detecting that a tracked instrument has been removed from the scene 108, replaced with another instrument, and/or inserted into the scene 108. For example, in an open surgical procedure, the system 100 can detect that an instrument for use in exposing the patient's spine has been removed from the scene 108. Similarly, in a minimally invasive surgical procedure, the system 100 can detect that a minimally invasive surgical instrument has been inserted into the scene 108 and/or into the patient. [0085] In some embodiments, determining that the spine of the patient is accessible for the spinal surgical procedure can include determining that the spine is sufficiently exposed by calculating an exposure metric and comparing the exposure metric to a threshold (e.g., similar to blocks 655 and blocks 877 of the methods 650 and 870, respectively, described in detail above). The exposure metric can include, for example, a percentage, value, or other characteristic representing an exposure level of the spine (e.g., as visible to the camera array). If the exposure metric is not met, the method 1090 can continue determining if the spine of the patient is accessible (block 1094) in a continuous manner. When the exposure metric is greater than the threshold, the method 1090 can proceed to block 1095.)
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify Paulsen’s surgical training model for laproscopic procedures with Grupp’s method and system for registering pre-operative image data with intra-operative data in a surgical arena. The determination of obviousness is predicated upon the following findings: both are directed towards the same field of endeavor of image processing and registration in surgical imaging and analysis. One skilled in the art would have been motivated to modify Paulsen’s laproscopic training model in order to register and overlay the data in order to facilitate end-user navigation and guidance during surgical imaging as described by Grupp. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Paulsen while the teaching of Paulsen continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of Grupp for ensuring an enhanced end-user experience and a more accurate representation of the surgical data. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Consider Claim 2.
The combination of Paulsen and Grupp teaches:
Claim 2. (Original) The surgery support system according to claim 1 wherein the treatment tool has a treatment part used for treatment inside the body cavity, and a non-treatment part continuous to the treatment part, and wherein the image processing unit generates an obstructing object removed image by combining at least a position indication image indicating a position of the non-treatment part based on the shape of the non-treatment part, and a hidden-part image that is an image hidden by the non-treatment part. (Grupp: [0052] In some embodiments, the position and/or shape of an object within the scene 108 may change over time. For example, the relative positions and orientations of the spine of a patient may change during a surgical procedure as the patient is operated on. Accordingly, the method 430 can include periodically or continuously reregistering the initial image data to the intraoperative image data (e.g., returning from block 436 to block 432) to account for intraoperative movement. [0053] Referring again to FIGS. 5A and 5B, in some instances registering the initial image data 542 to the intraoperative image data 540 based on only two points 543 a-b can lead to mis-/ill-registrations in which the points 543 a-b are matched correctly but the corresponding vertebrae 541 (e.g., the second vertebra 541 b) are not. That is, selecting only two of the points 543 a-b can leave the registration problem under constrained. FIG. 5C, for example, illustrates an ill-registration of the initial image data 542 to the intraoperative image data 540 in which the points 543 a-b generally match one another correctly but the second vertebra 541 b is not accurately registered. Such an ill-registration can cause the display of the initial image data 542 (e.g., as described with reference to block 436 of the method 430 of FIG. 4 ) to have an implausible pose relative to the intraoperative image data 540 and the physical scene 108. For example, the initial image data 542 of the second vertebra 541 b may appear contorted—with the points 543 a-b on the second vertebra 541 b identified by the surgeon (e.g., using a tracked instrument) having small surface distances between the displayed intraoperative and initial image data 540, 542, at the expense of other regions, such as the spinous process or vertebra body, being grossly misaligned. Although such registration failures are more frequent when using a smaller number of identified regions (e.g., two screw entry points) and may be mitigated by having the surgeon identify more points (e.g., three or more points) using a tracked instrument, requiring the surgeon to identify a large number of points-per-vertebra potentially lengthens the registration procedure and distracts from the surgical workflow. [0072] At decision block 877, the method 870 can include comparing a convergence metric to a threshold tolerance. The convergence metric can provide an indication of how much the labeling has converged toward the estimated poses after an iterative process. If the convergence metric is less than a threshold tolerance (indicating that the labeling has sufficiently converged), the method 870 can continue to block 878 and register the initial image data to the 3D surface reconstruction based at least in part on the labels and a set of rules, as described in detail above with reference to block 765 of the method 760. If the convergence metric is greater than the threshold tolerance (indicating that the labeling has not sufficiently converged), the method 760 can return to block 874 to again estimate the pose of the vertebrae and relabel the regions of the 3D surface reconstruction accordingly. [0073] In this manner, the method 870 can iteratively refine the labeling and vertebrae poses until they sufficiently converge. More specifically, improving the accuracy of the labeling improves the estimated poses of the vertebrae because the poses are based on regions of the 3D surface reconstruction labeled as “bone.” Likewise, the estimated poses introduce additional information from the initial data that can improve the accuracy of the labeling. In some aspects of the present technology, this iterative process can improve the registration accuracy by improving the accuracy of the labels. In some embodiments, the iterative process described in blocks 875-878 of the method 870 can comprise an expectation-maximization (EM) framework and/or can resemble a multiple-body coherent point drift framework. [0084]-[0085] )
Consider Claim 3.
The combination of Paulsen and Grupp teaches:
Claim 3. (Original) The surgery support system according to claim 2, wherein the image processing unit generates the position indication image by performing machine learning on the non-treatment part from training images of the treatment tool. (Grupp: [0053] Referring again to FIGS. 5A and 5B, in some instances registering the initial image data 542 to the intraoperative image data 540 based on only two points 543 a-b can lead to mis-/ill-registrations in which the points 543 a-b are matched correctly but the corresponding vertebrae 541 (e.g., the second vertebra 541 b) are not. That is, selecting only two of the points 543 a-b can leave the registration problem under constrained. FIG. 5C, for example, illustrates an ill-registration of the initial image data 542 to the intraoperative image data 540 in which the points 543 a-b generally match one another correctly but the second vertebra 541 b is not accurately registered. Such an ill-registration can cause the display of the initial image data 542 (e.g., as described with reference to block 436 of the method 430 of FIG. 4 ) to have an implausible pose relative to the intraoperative image data 540 and the physical scene 108. For example, the initial image data 542 of the second vertebra 541 b may appear contorted—with the points 543 a-b on the second vertebra 541 b identified by the surgeon (e.g., using a tracked instrument) having small surface distances between the displayed intraoperative and initial image data 540, 542, at the expense of other regions, such as the spinous process or vertebra body, being grossly misaligned. Although such registration failures are more frequent when using a smaller number of identified regions (e.g., two screw entry points) and may be mitigated by having the surgeon identify more points (e.g., three or more points) using a tracked instrument, requiring the surgeon to identify a large number of points-per-vertebra potentially lengthens the registration procedure and distracts from the surgical workflow. [0054] Accordingly, some embodiments of the present technology can utilize additional information captured by the system 100 to reduce the likelihood of ill-registrations without requiring the surgeon or another user to provide additional inputs to the system 100 that may slow or disrupt the surgical workflow. FIG. 6 , for example, is a flow diagram of a process or method 650 for registering initial image data to/with intraoperative image data in accordance with embodiments of the present technology. In some embodiments, the method 650 can be used to register the initial image data to the intraoperative image data at block 433 of the method 430 described in detail with reference to FIG. 4 . Although some features of the method 430 are described in the context of the system 100 shown in FIGS. 1-3 for the sake of illustration, one skilled in the art will readily understand that the method 650 can be carried out using other suitable systems and/or devices described herein. [0055] At block 651, the method 650 can include registering initial image data of a single target vertebra to intraoperative image data of the target vertebra. In some embodiments, the registration is based on a comparison of common points in both data sets. For example, with reference to FIGS. 5A-5C, the registration can be for the second vertebra 541 b based on the commonly identified points 543 a-b. [0056] At block 652, the method 650 can include estimating a pose (and/or position) of at least one other vertebra of the spine, such as a vertebra adjacent to the registered target vertebra. For example, with reference to FIGS. 5A-5C together, the initial image data 542 of the first vertebra 541 a and/or the third vertebra 541 c can be used to estimate the pose of the corresponding physical vertebra in the scene based on the registration of the second vertebra 541 b. That is, the initial image data 542 of the first vertebra 541 a and/or the third vertebra 541 c can be computationally overlaid over the intraoperative image data 540 based on the registration of the target second vertebra 541 b. In some embodiments, the estimate of the pose of the at least one other vertebra is a rough estimate because the spine or other object of interest may have deformed or otherwise changed positions between initial imaging and intraoperative imaging (e.g., due to changes of the spine curvature between initial imaging conducted with the patient in a supine position and intraoperative imaging conducted with the patient in a prone position). Poulsen: [0022] The third practice station 34 c includes a cover 52 connected to the top surface of the base 32 with a connector 54 and the cover 52 may further include a knob 56 connected to the outer surface of the cover 52. The connector 54 is a fastener that permits the cover 44 to slide, swivel or pivot about the fastener along the upper surface 36 of the base 32 to uncover a cavity 58 underneath the cover 52 with an object 60 optionally located inside the cavity 58. The object 60 removed from the cavity 58 may then be placed over a peg 38 that corresponds to the shape of the object 60. The cavity 58 is formed in the base 32 and may be any shape or size and depth. A deeper cavity may increase the difficulty level of extracting the object 60. The cavity 58 may also be sinuous or have a portion hidden from view by a wall such that the user would have to reach around the obstruction with an instrument to see if an object 60 is hiding behind the wall inside the cavity 58. The cavity 58 opens to the upper surface 36 such that when the cover 52 is moved from a first position covering the cavity 58 to a second position uncovering the cavity 58, the opening to the cavity 58 as well as the object 60 disposed inside the cavity 58 is revealed to the user. In this exercise, the user uses a surgical instrument such as a laparoscopic grasper inserted through an aperture 22, simulated tissue penetration region 14 or side of the trainer 10 to grab the knob 56 or cover 52 to turn it or slide it from a first closed position to a second open position. If a cavity 58 is provided underneath the cover 52, the user can then reach inside the cavity 58 and grab the object 60 with the same grasper or another grasper held in an opposite hand of the user and remove the object 60 from the cavity 58 and place it to the side or remove it from the trainer 10. Alternatively, the object 60 may then be placed over a peg 38 at the first station 34 a such that object 60 is placed over the peg 38 having a shape 76 that corresponds with the shape associated with the object 60.)
Consider Claim 4.
The combination of Paulsen and Grupp teaches:
Claim 4. (Original) The surgery support system according to claim 3, wherein the position indication image is an image indicating a contour of the non-treatment part.(Grupp: [0056] At block 652, the method 650 can include estimating a pose (and/or position) of at least one other vertebra of the spine, such as a vertebra adjacent to the registered target vertebra. For example, with reference to FIGS. 5A-5C together, the initial image data 542 of the first vertebra 541 a and/or the third vertebra 541 c can be used to estimate the pose of the corresponding physical vertebra in the scene based on the registration of the second vertebra 541 b. That is, the initial image data 542 of the first vertebra 541 a and/or the third vertebra 541 c can be computationally overlaid over the intraoperative image data 540 based on the registration of the target second vertebra 541 b. In some embodiments, the estimate of the pose of the at least one other vertebra is a rough estimate because the spine or other object of interest may have deformed or otherwise changed positions between initial imaging and intraoperative imaging (e.g., due to changes of the spine curvature between initial imaging conducted with the patient in a supine position and intraoperative imaging conducted with the patient in a prone position). [0089] The multiple vertebrae can be adjacent to one another (e.g., in either direction) or can be non-adjacent to one another. At this stage, the initial image data provides estimated poses of the multiple vertebrae based on the initial labeling of the 3D surface reconstruction and the model of anatomical interaction. [0090] At block 1106, the method 1100 can include relabeling the one or more regions of the 3D surface reconstruction based on the estimated poses of the multiple vertebrae. For example, regions of the 3D surface reconstruction that fall within the aligned initial image data and that agree with the model of anatomical interaction can be relabeled as “bone” where the initial image data comprises a segmented CT scan or other 3D representation of the spine. [0091]-[0092] Poulsen: [0017] When assembled, the top cover 16 is positioned directly above the base 18 with the legs 20 located substantially around the periphery and interconnected between the top cover 16 and base 18. The top cover 16 and base 18 are substantially the same shape and size and have substantially the same peripheral outline. [0022] The third practice station 34 c includes a cover 52 connected to the top surface of the base 32 with a connector 54 and the cover 52 may further include a knob 56 connected to the outer surface of the cover 52. The connector 54 is a fastener that permits the cover 44 to slide, swivel or pivot about the fastener along the upper surface 36 of the base 32 to uncover a cavity 58 underneath the cover 52 with an object 60 optionally located inside the cavity 58. The object 60 removed from the cavity 58 may then be placed over a peg 38 that corresponds to the shape of the object 60. The cavity 58 is formed in the base 32 and may be any shape or size and depth. A deeper cavity may increase the difficulty level of extracting the object 60. The cavity 58 may also be sinuous or have a portion hidden from view by a wall such that the user would have to reach around the obstruction with an instrument to see if an object 60 is hiding behind the wall inside the cavity 58. The cavity 58 opens to the upper surface 36 such that when the cover 52 is moved from a first position covering the cavity 58 to a second position uncovering the cavity 58, the opening to the cavity 58 as well as the object 60 disposed inside the cavity 58 is revealed to the user. In this exercise, the user uses a surgical instrument such as a laparoscopic grasper inserted through an aperture 22, simulated tissue penetration region 14 or side of the trainer 10 to grab the knob 56 or cover 52 to turn it or slide it from a first closed position to a second open position. If a cavity 58 is provided underneath the cover 52, the user can then reach inside the cavity 58 and grab the object 60 with the same grasper or another grasper held in an opposite hand of the user and remove the object 60 from the cavity 58 and place it to the side or remove it from the trainer 10. Alternatively, the object 60 may then be placed over a peg 38 at the first station 34 a such that object 60 is placed over the peg 38 having a shape 76 that corresponds with the shape associated with the object 60.)
Consider Claim 5.
The combination of Paulsen and Grupp teaches:
Claim 5. (Currently Amended) The surgery support system according to claim 1 further comprising an autofocus unit configured to adjust focal distances of the image capturing units individually. (Grupp: [0031] The image processing device 103 can synthesize the output image from images captured by a subset (e.g., two or more) of the cameras 112 in the camera array 110, and does not necessarily utilize images from all of the cameras 112. For example, for a given virtual camera perspective, the processing device 102 can select a stereoscopic pair of images from two of the cameras 112. In some embodiments, such a stereoscopic pair can be selected to be positioned and oriented to most closely match the virtual camera perspective. In some embodiments, the image processing device 103 (and/or the depth sensor 114) estimates a depth for each surface point of the scene 108 relative to a common origin to generate a point cloud and/or a 3D mesh that represents the surface geometry of the scene 108. Such a representation of the surface geometry can be referred to as a surface reconstruction, a 3D reconstruction, a 3D volume reconstruction, a volume reconstruction, a 3D surface reconstruction, a depth map, a depth surface, and/or the like. In some embodiments, the depth cameras 118 of the depth sensor 114 detect the structured light projected onto the scene 108 by the projector 116 to estimate depth information of the scene 108. In some embodiments, the image processing device 103 estimates depth from multiview image data from the cameras 112 using techniques such as light field correspondence, stereo block matching, photometric symmetry, correspondence, defocus, block matching, texture-assisted block matching, structured light, and the like, with or without utilizing information collected by the depth sensor 114. In other embodiments, depth may be acquired by a specialized set of the cameras 112 performing the aforementioned methods in another wavelength.)
Consider Claim 6.
The combination of Paulsen and Grupp teaches:
Claim 6. (Currently Amended) The surgery support system according to claim 1 further comprising a depth estimation unit configured to estimate a depth to a point of interest inside the body cavity. (Poulsen: [0021] The second practice station 34 b includes a cover 44. The cover 44 is a rigid door 44 that is connected to the base 32 via a hinge 46 and the door 44 may further include a knob 48 connected to the outer surface of the door 44. The hinge 46 may be spring loaded such that the door 44 is biased in the closed position. Underneath the door 44 is a cavity (not shown) that may include an object (not shown) that is hidden from view when the door 44 is closed. The cavity is formed in the base 32 and may be any shape or size and depth. The cavity opens to the upper surface 36 such that when the door 44 is moved, the cavity beneath the door 44 is uncovered. In this exercise, the user employs a surgical instrument such as a laparoscopic grasper inserted through an aperture 22, simulated tissue penetration region 14 or side of the trainer 10 to grab the knob 48 or door 44 to swing it open from a closed position. If a cavity is provided underneath the door 44, the user must maintain the door 44 in the open position while another instrument in another hand is used to grab an object located inside the cavity and remove it. [0029] In the illustrated embodiment, the camera array 110 further includes a depth sensor 114. In some embodiments, the depth sensor 114 includes (i) one or more projectors 116 that project a structured light pattern onto/into the scene 108 and (ii) one or more depth cameras 118 (which can also be referred to as second cameras) that capture second image data of the scene 108 including the structured light projected onto the scene 108 by the projector 116. The projector 116 and the depth cameras 118 can operate in the same wavelength and, in some embodiments, can operate in a wavelength different than the cameras 112. For example, the cameras 112 can capture the first image data in the visible spectrum, while the depth cameras 118 capture the second image data in the infrared spectrum. In some embodiments, the depth cameras 118 have a resolution that is less than a resolution of the cameras 112. For example, the depth cameras 118 can have a resolution that is less than 70%, 60%, 50%, 40%, 30%, or 20% of the resolution of the cameras 112. In other embodiments, the depth sensor 114 can include other types of dedicated depth detection hardware (e.g., a LiDAR detector) for determining the surface geometry of the scene 108. In other embodiments, the camera array 110 can omit the projector 116 and/or the depth cameras 118.)
Consider Claim 8, 19 and 20.
The combination of Paulsen and Grupp teaches:
Claim 8. (Currently Amended) A program used in the surgery support system according to claim 1. / Claim 19. (New) A program used in the surgery support system according to claim 2. / Claim 20. (New) A program used in the surgery support device according to claim 7. (Poulsen: [0016] A video display monitor 28 that is hinged to the top cover 16 is shown in a closed orientation in FIG. 1. The video monitor 62 is connectable to a variety of visual systems for delivering an image to the monitor. For example, a scope inserted through one of the pre-established apertures 22 or a webcam located in the cavity 12 and used to observe the simulated procedure can be connected to the video monitor 28 and/or a mobile computing device to provide an image to the user. Also, audio recording or delivery means may also be provided and integrated with the trainer 10 to provide audio and visual capabilities. Means for connecting a portable memory storage device such as a flash drive, smart phone, digital audio or video player, or other digital mobile device is also provided to record training procedures and/or play back pre-recorded videos on the monitor for demonstration purposes. Of course, connection means for providing an audio visual output to a larger screen other than the monitor is provided. In another variation, the top cover 10 does not include a video display but includes means for supporting a laptop computer, a mobile digital device or tablet such as an IPAD® and connecting it by wire or wirelessly to the trainer. Grupp: [0035] The processing device 102 can comprise a processor and a non-transitory computer-readable storage medium that stores instructions that when executed by the processor, carry out the functions attributed to the processing device 102 as described herein. Although not required, aspects and embodiments of the present technology can be described in the general context of computer-executable instructions, such as routines executed by a general-purpose computer, e.g., a server or personal computer. Those skilled in the relevant art will appreciate that the present technology can be practiced with other computer system configurations, including Internet appliances, hand-held devices, wearable computers, cellular or mobile phones, multi-processor systems, microprocessor-based or programmable consumer electronics, set-top boxes, network PCs, mini-computers, mainframe computers and the like. The present technology can be embodied in a special purpose computer or data processor that is specifically programmed, configured or constructed to perform one or more of the computer-executable instructions explained in detail below. Indeed, the term “computer” (and like terms), as used generally herein, refers to any of the above devices, as well as any data processor or any device capable of communicating with a network, including consumer electronic goods such as game devices, cameras, or other electronic devices having a processor and other components, e.g., network communication circuitry.)
Consider Claim 9 and 11.
The combination of Paulsen and Grupp teaches:
Claim 9. (New) The surgery support system according to claim 1 wherein the surgical assistance device includes holding parts and a base member, the holding parts holding the image capturing units, respectively, wherein the base member is formed in a tube-like shape, and the image capturing units are attached to the base member through the holding parts, wherein three or more of the image capturing units are provided, and each of the image capturing units is arranged so as to be directed to outside of the base member at an angle of 0 degree to 10 degrees, and wherein each of the holding parts is formed so as to penetrate through the base member. / Claim 11. (New) The surgery support device according to claim 7 wherein the surgical assistance device includes holding parts and a base member, the holding parts holding the image capturing units, respectively, wherein the base member is formed in a tube-like shape, and the image capturing units are attached to the base member through the holding parts, wherein three or more of the image capturing units are provided, and each of the image capturing units is arranged so as to be directed to outside of the base member at an angle of 0 degree to 10 degrees, and wherein each of the holding parts is formed so as to penetrate through the base member. (Grupp: [0026] In the illustrated embodiment, the camera array 110 includes a plurality of cameras 112 (identified individually as cameras 112 a-112 n; which can also be referred to as first cameras) that can each capture images of a scene 108 (e.g., first image data) from a different perspective. The scene 108 can include for example, a patient undergoing surgery (e.g., spinal surgery) and/or another medical procedure. In other embodiments, the scene 108 can be another type of scene. The camera array 110 can further include dedicated object tracking hardware 113 (e.g., including individually identified trackers 113 a-113 n) that captures positional data of one more objects, such as an instrument 101 (e.g., a surgical instrument or tool) having a tip 109, to track the movement and/or orientation of the objects through/in the scene 108. In some embodiments, the cameras 112 and the trackers 113 are positioned at fixed locations and orientations (e.g., poses) relative to one another. For example, the cameras 112 and the trackers 113 can be structurally secured by/to a mounting structure (e.g., a frame) at predefined fixed locations and orientations. In some embodiments, the cameras 112 are positioned such that neighboring cameras 112 share overlapping views of the scene 108. In general, the position of the cameras 112 can be selected to maximize clear and accurate capture of all or a selected portion of the scene 108. Likewise, the trackers 113 can be positioned such that neighboring trackers 113 share overlapping views of the scene 108. Therefore, all or a subset of the cameras 112 and the trackers 113 can have different extrinsic parameters, such as position and orientation. [0027] In some embodiments, the cameras 112 in the camera array 110 are synchronized to capture images of the scene 108 simultaneously (within a threshold temporal error). In some embodiments, all or a subset of the cameras 112 are light field, plenoptic, and/or RGB cameras that capture information about the light field emanating from the scene 108 (e.g., information about the intensity of light rays in the scene 108 and also information about a direction the light rays are traveling through space). In some embodiments, image data from the cameras 112 can be used to reconstruct a light field of the scene 108. Therefore, in some embodiments the images captured by the cameras 112 encode depth information representing a surface geometry of the scene 108. In some embodiments, the cameras 112 are substantially identical. In other embodiments, the cameras 112 include multiple cameras of different types. For example, different subsets of the cameras 112 can have different intrinsic parameters such as focal length, sensor type, optical components, and the like. The cameras 112 can have charge-coupled device (CCD) and/or complementary metal-oxide semiconductor (CMOS) image sensors and associated optics. Such optics can include a variety of configurations including lensed or bare individual image sensors in combination with larger macro lenses, micro-lens arrays, prisms, and/or negative lenses. For example, the cameras 112 can be separate light field cameras each having their own image sensors and optics. In other embodiments, some or all of the cameras 112 can comprise separate microlenslets (e.g., lenslets, lenses, microlenses) of a microlens array (MLA) that share a common image sensor. In other embodiments, some or all of the cameras 112 can be RGB (e.g., color) cameras having visible imaging sensors.)
Consider Claim 10 and 12.
The combination of Paulsen and Grupp teaches:
Claim 10. (New) The surgery support system according to claim 1 wherein the treatment tool has a treatment part used for treatment inside the body cavity, anda non-treatment part continuous to the treatment part, and wherein the image processing unit is configured to select and perform any of an obstructing object removal process of removing the entire treatment tool, and an obstructing object removal process of removing only the non-treatment part. / Claim 12. (New) The surgery support device according to claim 7 wherein the treatment tool has a treatment part used for treatment inside the body cavity, anda non-treatment part continuous to the treatment part, andwherein the image processing unit is configured to select and perform any of an obstructing object removal process of removing the entire treatment tool,and an obstructing object removal process of removing only the non-treatment part. (Poulsen: [0022] The third practice station 34 c includes a cover 52 connected to the top surface of the base 32 with a connector 54 and the cover 52 may further include a knob 56 connected to the outer surface of the cover 52. The connector 54 is a fastener that permits the cover 44 to slide, swivel or pivot about the fastener along the upper surface 36 of the base 32 to uncover a cavity 58 underneath the cover 52 with an object 60 optionally located inside the cavity 58. The object 60 removed from the cavity 58 may then be placed over a peg 38 that corresponds to the shape of the object 60. The cavity 58 is formed in the base 32 and may be any shape or size and depth. A deeper cavity may increase the difficulty level of extracting the object 60. The cavity 58 may also be sinuous or have a portion hidden from view by a wall such that the user would have to reach around the obstruction with an instrument to see if an object 60 is hiding behind the wall inside the cavity 58. The cavity 58 opens to the upper surface 36 such that when the cover 52 is moved from a first position covering the cavity 58 to a second position uncovering the cavity 58, the opening to the cavity 58 as well as the object 60 disposed inside the cavity 58 is revealed to the user. In this exercise, the user uses a surgical instrument such as a laparoscopic grasper inserted through an aperture 22, simulated tissue penetration region 14 or side of the trainer 10 to grab the knob 56 or cover 52 to turn it or slide it from a first closed position to a second open position. If a cavity 58 is provided underneath the cover 52, the user can then reach inside the cavity 58 and grab the object 60 with the same grasper or another grasper held in an opposite hand of the user and remove the object 60 from the cavity 58 and place it to the side or remove it from the trainer 10. Alternatively, the object 60 may then be placed over a peg 38 at the first station 34 a such that object 60 is placed over the peg 38 having a shape 76 that corresponds with the shape associated with the object 60.)
Consider Claim 13, 14 and 15.
The combination of Paulsen and Grupp teaches:
Claim 13. (New) The surgery support system according to claim 2 further comprising an autofocus unit configured to adjust focal distances of the image capturing units individually.
Claim 14. (New) The surgery support system according to claim 3 further comprising an autofocus unit configured to adjust focal distances of the image capturing units individually.
Claim 15. (new) The surgery support system according to claim 4 further comprising an autofocus unit configured to adjust focal distances of the image capturing units individually. (Grupp: [0031] The image processing device 103 can synthesize the output image from images captured by a subset (e.g., two or more) of the cameras 112 in the camera array 110, and does not necessarily utilize images from all of the cameras 112. For example, for a given virtual camera perspective, the processing device 102 can select a stereoscopic pair of images from two of the cameras 112. In some embodiments, such a stereoscopic pair can be selected to be positioned and oriented to most closely match the virtual camera perspective. In some embodiments, the image processing device 103 (and/or the depth sensor 114) estimates a depth for each surface point of the scene 108 relative to a common origin to generate a point cloud and/or a 3D mesh that represents the surface geometry of the scene 108. Such a representation of the surface geometry can be referred to as a surface reconstruction, a 3D reconstruction, a 3D volume reconstruction, a volume reconstruction, a 3D surface reconstruction, a depth map, a depth surface, and/or the like. In some embodiments, the depth cameras 118 of the depth sensor 114 detect the structured light projected onto the scene 108 by the projector 116 to estimate depth information of the scene 108. In some embodiments, the image processing device 103 estimates depth from multiview image data from the cameras 112 using techniques such as light field correspondence, stereo block matching, photometric symmetry, correspondence, defocus, block matching, texture-assisted block matching, structured light, and the like, with or without utilizing information collected by the depth sensor 114. In other embodiments, depth may be acquired by a specialized set of the cameras 112 performing the aforementioned methods in another wavelength. [0037] The virtual camera perspective is controlled by an input controller 106 that can update the virtual camera perspective based on user driven changes to the camera's position and rotation. The output images corresponding to the virtual camera perspective can be outputted to the display device 104. In some embodiments, the image processing device 103 can vary the perspective, the depth of field (e.g., aperture), the focus plane, and/or another parameter of the virtual camera (e.g., based on an input from the input controller) to generate different 3D output images without physically moving the camera array 110. The display device 104 can receive output images (e.g., the synthesized 3D rendering of the scene 108) and display the output images for viewing by one or more viewers. In some embodiments, the processing device 102 receives and processes inputs from the input controller 106 and processes the captured images from the camera array 110 to generate output images corresponding to the virtual perspective in substantially real-time or near real-time as perceived by a viewer of the display device 104 (e.g., at least as fast as the frame rate of the camera array 110).)
Consider Claim 16, 17 and 18.
The combination of Paulsen and Grupp teaches:
Claim 16. (New) The surgery support system according to claim 2 further comprising a depth estimation unit configured to estimate a depth to a point of interest inside the body cavity.
Claim 17. (New) The surgery support system according to claim 3 further comprising a depth estimation unit configured to estimate a depth to a point of interest inside the body cavity.
Claim 18. (New) The surgery support system according to claim 4 further comprising a depth estimation unit configured to estimate a depth to a point of interest inside the body cavity. (Grupp: [0030] The image processing device 103 can (i) receive the first image data captured by the cameras 112 (e.g., light field images, light field image data, RGB images) and depth information from the depth sensor 114 (e.g., the second image data captured by the depth cameras 118), and (ii) process the image data and depth information to synthesize (e.g., generate, reconstruct, render) a three-dimensional (3D) output image of the scene 108 corresponding to a virtual camera perspective. The output image can correspond to an approximation of an image of the scene 108 that would be captured by a camera placed at an arbitrary position and orientation corresponding to the virtual camera perspective. In some embodiments, the image processing device 103 can further receive and/or store calibration data for the cameras 112 and/or the depth cameras 118 and synthesize the output image based on the image data, the depth information, and/or the calibration data. More specifically, the depth information and the calibration data can be used/combined with the images from the cameras 112 to synthesize the output image as a 3D (or stereoscopic 2D) rendering of the scene 108 as viewed from the virtual camera perspective. In some embodiments, the image processing device 103 can synthesize the output image using any of the methods disclosed in U.S. patent application Ser. No. 16/457,780, titled “SYNTHESIZING AN IMAGE FROM A VIRTUAL PERSPECTIVE USING PIXELS FROM A PHYSICAL IMAGER ARRAY WEIGHTED BASED ON DEPTH ERROR SENSITIVITY,” and filed Jun. 28, 2019, which is incorporated herein by reference in its entirety. In other embodiments, the image processing device 103 can generate the virtual camera perspective based only on the images captured by the cameras 112—without utilizing depth information from the depth sensor 114. For example, the image processing device 103 can generate the virtual camera perspective by interpolating between the different images captured by one or more of the cameras 112. [0031] The image processing device 103 can synthesize the output image from images captured by a subset (e.g., two or more) of the cameras 112 in the camera array 110, and does not necessarily utilize images from all of the cameras 112. For example, for a given virtual camera perspective, the processing device 102 can select a stereoscopic pair of images from two of the cameras 112. In some embodiments, such a stereoscopic pair can be selected to be positioned and oriented to most closely match the virtual camera perspective. In some embodiments, the image processing device 103 (and/or the depth sensor 114) estimates a depth for each surface point of the scene 108 relative to a common origin to generate a point cloud and/or a 3D mesh that represents the surface geometry of the scene 108. Such a representation of the surface geometry can be referred to as a surface reconstruction, a 3D reconstruction, a 3D volume reconstruction, a volume reconstruction, a 3D surface reconstruction, a depth map, a depth surface, and/or the like. In some embodiments, the depth cameras 118 of the depth sensor 114 detect the structured light projected onto the scene 108 by the projector 116 to estimate depth information of the scene 108. In some embodiments, the image processing device 103 estimates depth from multiview image data from the cameras 112 using techniques such as light field correspondence, stereo block matching, photometric symmetry, correspondence, defocus, block matching, texture-assisted block matching, structured light, and the like, with or without utilizing information collected by the depth sensor 114. In other embodiments, depth may be acquired by a specialized set of the cameras 112 performing the aforementioned methods in another wavelength. [0032])
Conclusion
The prior art made of record in form PTO-892 and not relied upon is considered pertinent to applicant's disclosure.
PNG
media_image1.png
117
900
media_image1.png
Greyscale
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAHMINA ANSARI whose telephone number is 571-270-3379. The examiner can normally be reached on IFP Flex - Monday through Friday 9 to 5.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’NEAL MISTRY can be reached on 313-446-4912. The fax phone numbers for the organization where this application or proceeding is assigned are 571-273-8300 for regular communications and 571-273-8300 for After Final communications. TC 2600’s customer service number is 571-272-2600.
Any inquiry of a general nature or relating to the status of this application or proceeding should be directed to the receptionist whose telephone number is 571-272-2600.
2674
/Tahmina Ansari/
August 8, 2026
/TAHMINA N ANSARI/Primary Examiner, Art Unit 2674
Prosecution Insights