DETAILED ACTION
This Office action is responsive to communications filed on 02/17/2026. Claims 1, 19, & 37 have been amended. Claims 6-7 are withdrawn. Claims 8, 18, 20-36, & 39 canceled. Presently, Claims 1-5, 9-17, 19, & 37-38 remain pending and are hereinafter examined on the merits.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 02/17/2026 has been entered.
Response to Arguments
The Applicant’s arguments with respect to rejections under 35 USC § 101 have been fully, considered, but are not persuasive.
The Applicant argues the claimed subject matter is directed to a mental process because a human does not possess a depth sensor and cannot scan an internal space of a patient or obtain internal depth data with the same precision as a sensor. This argument does not commensurate with the scope of the claims. The claims do not require any particular manner of processing the obtained data nor any specific algorithmic implementation for identifying the port location. The identified abstract idea resides in the step of performing an operation that includes identifying a port location based on the external body wall data and the internal depth data. As written, the claim only requires that the location be identified “based on” the data. The claim does not recite how the data is analyzed or processed to determine the location. Accordingly, the step remains a mental evaluation or decision that can be conceptually performed using observed information about the patient. The presence of data acquisition hardware does not remove the mental character of the decision-making step that constitutes the judicial exception.
The Applicant further argues that identifying a port location for insertion of a surgical instrument cannot practically be performed mentally to the precision required in surgery. The claims do not recite any requirement regarding surgical precision, accuracy thresholds, required to determine the port location. The claims merely require identifying a location through which an instrument is “to” be inserted. Arguments relying on surgical precision or asserted practical constraints are therefore directed to features not recited in the claim and are not persuasive.
The Applicant contends that the claims integrate the alleged exception into a practical application because the identified port location is used for a surgical procedure. The claims, however, do not require the surgical system to perform any surgical action. The claims only require identifying a port location associated with a computer-assisted surgical system that is configured to perform a procedure. The surgical procedure “itself” is not performed or controlled by the claimed steps. Thus the recited context of surgery merely provides an environment for the abstract idea and does not impose any meaningful limits on the judicial exception.
With respect to Step 2A, Prong Two, the Applicant argues that the use of a depth sensor, time-of-flight sensor, and a visible light camera integrates the exception into a practical application. The Examiner disagrees. The claim recites these elements only as tools for obtaining external wall data and internal depth data. These steps constitute data gathering activities performed before the abstract decision of identifying a port location.
The Applicant asserts that using a single imaging device to obtain external and internal depth data provides a technical advantage, including precision and a shared coordinate system. These advantages are not reflected in the claims. The claims do not require that the external and internal data share a coordinate system, nor do they recite any particular fusion processing technique that produces an improvement, nor do the claim specify a single imaging device. The claim requires obtaining two types od data and using them as a basis for identifying a location.
The Applicant argues that implementing the depth sensor as a time-of-flight sensor in combination with a visible light camera process a technical solution. However, the claims do not recite any operation of these components. The are recited only as generic imaging hardware used to collect data. As explained in the rejection below, these additional elements, do not provide an inventive concept under Step 2B, nor do these claimed elements specify how the subcomponents are linked to obtaining of the external and internal data. In addition, the claims do not recite any particular improvement to sensor technology, imaging processing, or computer functionality. Instead, the elements merely supply data to the abstract evaluation of identifying a port location.
The Applicant also asserts that dependent claims further integrate the invention into a practical application. However, the Applicant did not identify specific additional limitations in those claims that would alter the eligibility analysis. The dependent claims continue to reply on the same abstract determinations.
For these reasons, the Examiner maintained that the claims are indeed directed to a mental process and that the additional elements, individually and in combination, do not integrate the judicial exception into a practical application and do not amount to significantly more than the abstract idea. Accordingly, the rejections under 35 USC § 101 is maintained.
Applicant’s arguments with respect to claim(s) rejected under 35 USC § 103 have been considered but are moot because the new ground of rejection does not rely on Clavin et al (US 2013/0296682) in view of Coste-Maniere et al (US 8,170,716 B2) applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. The new grounds of rejection now relies on Coste-Maniere et al (US 8170716 B2) in view of Casas (US 20160191887 A1) in view of Refai et al (US 2019/0231220 A1).
However, the Applicants arguments have been considered but are not persuasive.
First, the Applicant argues, none of the cited references such as Refai et al disclosed, “an imaging device” including a depth sensor and a visible light camera, and wherein the depth sensor is implemented by a time-of-flight sensor. This argument is not commensurate with the scope of the claim or with the new grounds of rejection. The claim recites, “an imaging device” where the indefinite article “an” encompasses one or more devices. The rejection now relies on Casas for obtaining the external body wall data using a 3D scanner positioned outside the patient, and on Refai for obtaining internal depth data and for the use of a time-of-flight sensing system with visible light imaging components. Coste-Maniere supplies the surgical planning system that identifies entry port locations based on patient models using external body wall data and internal depth data.
The Applicant argues no one reference teaches or discloses a single imaging device capable of performing both external and internal scanning. The claim does not require or specify such a configuration. The claim language permits one or more imaging devices. The rejection does not require the physical merging of the external scanner of Casas with the endoscopic system of Refai into a single structure. Even if the claim(s) required a single imaging device, which the claims do not specify – MPEP 2144.04(V)(B), - states, “In re Larson, 340 F.2d 965, 968, 144 USPQ 347, 349 (CCPA 1965) (A claim to a fluid transporting vehicle was rejected as obvious over a prior art reference which differed from the prior art in claiming a brake drum integral with a clamping means, whereas the brake disc and clamp of the prior art comprise several parts rigidly secured together as a single unit. The court affirmed the rejection holding, among other reasons, "that the use of a one piece construction instead of the structure disclosed in [the prior art] would be merely a matter of obvious engineering choice."), which establishes that combining of the two imaging devices into a single structure is merely a matter of obvious engineering choice that a person of ordinary skill in the art would be motivated to do so. The prior arts Casas et al and Refai et al already comprise several parts rigidly secured together as a single unit, simply manufacturing them as a single integral piece is just an expected, uninventive engineering steps, exactly as the court found in In re Larson.
Applicant also contends that Refai does not disclose an imaging device including both a time-of-flight depth sensor and a visible light camera. The Examiner disagrees. Refai discloses, “wherein the imaging device includes a depth sensor and a visible light camera, and wherein the depth sensor is implemented by a time-of-flight sensor.”. Refai describes at ¶0143 the integrated system (130e and 130f) include a visible light source and one or more cameras with visible RBG light sensitivity. Refai explicitly highlights that these integrated systems can include a “forward facing medical scanning and mapping system 10, 10 a, 10 b, 10 c, and/or 10 d”. “10c” is defined by Refai as utilizing the time-of-flight camera and modulated infrared optical source to extract 3D depth information, ¶0113. Therefore, by the explicit teaching that the integrated system 130e/130f can simultaneously house the visible light camera components alongside the forward-facing system 10c, Refai teaches the imaging device includes a depth sensor and a visible light camera, and wherein the depth sensor is implemented by a time-of-flight sensor, “as claimed”.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description: FIG. 14 – element 1424, FIG. 18 - elements 1802, 1804, 1806, 1808, 1810, 1812. Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because of the following informalities and should recite:
¶0043, “the system [[300]]100”.
¶0069, “camera head [[206]]802”.
¶0075, “the imaging device [[204]]304”.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-5, 9-17, 19, & 37 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 of the subject matter eligibility test (see MPEP 2106.03).
Claims 1-5, 9-17, & 38 are drawn to a “system” which describes one of the four statutory categories, i.e., a machine.
Claim 19 is directed to a “method” which describes one of the four statutory categories of patentable subject matter, i.e., a process.
Claims 37 are drawn to a “non-transitory computer-readable medium” which describes one of the four statutory categories, i.e., a manufacture.
Step 2A of the subject matter eligibility test (see MPEP 2106.04).
Prong One:
Claim 1 recite (“sets forth” or “describes”) the abstract idea of “a mental process” (MPEP 2106.04(a)(2).III.), substantially as follows:
“perform, based on the external body wall data and the internal depth data, an operation associated with a computer-assisted surgical system configured to perform a procedure with respect to the patient, wherein the performing of the operation comprises identifying a port location on the external body wall of the patient ”
Claim 19 recite (“sets forth” or “describes”) the abstract idea of “a mental process” (MPEP 2106.04(a)(2).III.), substantially as follows:
“performing, on the external body wall data and the internal depth data, an operation associated with a computer-assisted surgical system configured to perform a procedure with respect to the patient, wherein the performing of the operation comprises identifying a port location on the external body wall of the patient”
Claim 37 recite (“sets forth” or “describes”) the abstract idea of “a mental process” (MPEP 2106.04(a)(2).III.), substantially as follows:
“perform, based on the external body wall data and the internal depth data, an operation associated with a computer-assisted surgical system configured to perform a procedure with respect to the patient, wherein the performing of the operation comprises identifying a port location on the external body wall of the patient through”
In claims (1, 19, 37), while this limitation references a computer-assisted surgical system, it only recites that some unspecified operation is performed “based on” the acquired data. The claim does not describe how the data is processed, how the operation is executed, or what is involved in associating the data with the surgical system. This step recited a mental process because the “operation” is understood in context to be an act such as identifying a port location through which to insert a surgical instrument, an act that can be performed in the human mind. The act of the operation “perform[...] an operation” is not indicative of performing a surgery, and claimed recitations identified above sustain that it is something that can be done mentally. Specifically, the above noted recitation reflects the surgeon conceptualization of what operation will be performed. Similarly, the step of "identifying a port location on the external body wall of the patient" can be done by the human mind through ordinary visual observation, judgment, and reasoning (i.e., looking at the patient's anatomy, recalling prior knowledge of suitable port placements, and deciding mentally where such a port should be located). The additional mention of a “computer-assisted surgical system” is not tied to any specific structural implementation or technological improvement, and thus is merely a nominal context. There is nothing recited in the claim to suggest an undue level of complexity in the performing of the operation. Accordingly, the claim recites a mental process falling within the “Mental Processes” grouping of abstract ideas, and therefore, the claim recites an abstract idea.
Prong Two: Claims (1, 19, 37) do not include additional elements that integrate the mental process into a practical application.
This judicial exception is not integrated into a practical application. In particular, the claims recites
[1] additional steps of
A. “a memory storing instructions; and a processor communicatively coupled to the memory and configured to execute the instructions to: obtain, by directing an imaging device to scan an external body wall of a patient while the imaging device is external to the patient, [...]; obtain, by directing the imaging device to scan an internal space of the patient, internal depth data [...] ” - (claim 1),
B. “obtaining, by directing an imaging device to scan an external body wall of a patient while the imaging device is external to the patient, external body wall data [...]; obtaining, by directing imaging device to scan an internal space of the patient, internal depth data [...] ” -(claim 19),
C. “A non-transitory computer-readable medium storing instructions that, when executed, direct a processor to: obtain, by directing an imaging device to scan an external body wall of a patient while the imaging device is external to the patient, external body wall data [...]; obtain, by directing the imaging device to scan an internal space of the patient, internal depth data [...]” -(claim 37); and
[2] further an addition step of
A. “through which the computer-assisted surgical system is to insert a surgical instrument into the internal space of the patient, and”-(claim 1),
B. “through which the computer-assisted surgical system is to insert a surgical instrument into the internal space of the patient, and” -(claim 19),
C. “through which the computer-assisted surgical system is to insert a surgical instrument into the internal space of the patient, and” -(claim 37),
The steps in (1) represent merely data gathering or pre-solution activities that are necessary for use of the recited judicial exception and are recited at a high level of generality with conventionally used tools (see below Step IIB for further details). Data gathering and mere instructions to implement an abstract idea on a computer do not integrate a judicial exception into a practical application (MPEP 2106.05 (f and g)).
Regarding the processor language written at such a high level of generality of structural limitations, the processor language amounts to a generic computer component with mere instructions to implement the abstract idea on a computer.
The step in (2) represents merely post-solution activity and is recited at a high level of generality, not a practical application, nor a technological solution that solves a technological application in a meaningful way.
As a whole, the additional elements merely serve to gather and feed information to the abstract idea and to output a notification based on the abstract idea, while generically implementing it on conventionally used tools. There is no practical application because the abstract idea is not applied, relied on, or used in a meaningful way. No improvement to the technology is evident, and the estimated image information is not outputted in any way such that a practical benefit is realized. Therefore, the additional elements, alone or in combination, do not integrate the abstract idea into a practical application.
Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. Further, there is no evidence of record that would support the assertion that this step is an improvement to a computer or technological solution to a technological problem. Ultimately, the Applicant’s describe improvement in the process of using imaging techniques, but this is not an improvement in the function of a computer or other technology (See MPEP 2106.05(a)(ii); “the court determined that the claimed user interface simply provided a trader with more information to facilitate market trades, which improved the business process of market trading but did not improve computers or technology”; See MPEP 2106.04(d)(1); 2106.05(a); and 2106.05(f)). The claims are directed to the abstract idea. Also, there does not appear to be any particular structure or machine, treatment or prophylaxis, transformation, or any other meaningful application that would render the claim eligible at step 2A, prong 2.
Step 2B of the subject matter eligibility test (see MPEP 2106.05).
Claims (1, 19, 37) do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above, the claims recite additional steps of obtaining internal and external data with an imaging device. These steps represents mere data gathering, data outputting or pre/post/extra-solution activities that are necessary for use of the recited judicial exception and are recited at a high level of generality. Furthermore, as discussed above, limitations with respect to the processor languages/terms, respectively, amount to mere instructions to implement the abstract idea on a computer. As discussed with respect to Step 2A Prong Two, the additional elements in the claims amount to no more than insignificant extra solution activity and mere instructions to apply the exception using a generic computer component. The same analysis applies here in 2B and does not provide an inventive concept. The data gathering steps that were considered insignificant extra-solution activity in Step 2A Prong Two, have been re-evaluated in Step 2B and determined to be well-understood, routine, conventional activity in the field.
As an evidence, Lang (US 20170258526 A1) discloses:
¶0207, ‘surface registration algorithms such as the Iterative Closest Point algorithm, statistical models, Active Shape Models, mutual information-based or other volume registration algorithms, object recognition, pattern recognition or computer vision techniques, deep learning or other artificial intelligence methods. The processed data can, for example, consist of mesh data, parametric surface data, point cloud data, volume data or a combination thereof. These methods are known in the art and have been implemented in publicly and/or commercially available code libraries and application programming interfaces (API's), such as the Insight Segmentation and Registration Toolkit (ITK), the open-source computer vision library OpenCV, Elastix, Plastimatch, or the Medical Image Registration Toolkit (MIRTK).’
¶1185, ‘3D laser scanners or depth sensors known in the art, such as, for example, the Structure laser scanner provided by Occipital Inc., can be used to image the surface of the patient's bone and/or cartilage and/or tissue and/or ligaments and/or menisci and/or organ. Other 3D scanners known in the art can be used. Any laser scanning, optical or light scanning technique known in the art for determining, estimating or deriving the 3D volume, 3D surface or 3D shape of a structure known in the art can be used.’
As an evidence, Ramachandra et al (US 20150049172 A1) discloses:
¶0064, ‘The steps of a method or process described in connection with the implementations disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of non-transitory storage medium known in the art.’
For similar reasons set forth in Step 2A, Prong Two above, the additional elements of “ obtain, [...] external body wall data representative of a three-dimensional model of the external body wall of the patient and based on external depth data for the external body wall; obtain, [...] internal depth data representative of a depth map for the internal space of the patient; and [...] wherein the imaging device includes a depth sensor and a visible light camera, and wherein the depth sensor is implemented by a time-of-flight sensor.”-(claim 1), “obtaining, [...] external body wall data representative of a three-dimensional model of the external body wall of the patient and based on external depth data for the external body wall; obtaining, [...] internal depth data representative of a depth map for the internal space of the patient; and [...] wherein the imaging device includes a depth sensor and a visible light camera, and wherein the depth sensor is implemented by a time-of-flight sensor.” -(claim 19), & “obtain, [...], external body wall data representative of a three-dimensional model of the external body wall of the patient and based on external depth data for the external body wall; obtain, [...], internal depth data representative of a depth map for the internal space of the patient; and [...] wherein the imaging device includes a depth sensor and a visible light camera, and wherein the depth sensor is implemented by a time-of-flight sensor.” -(claim 37); do not provide an inventive concept under Step 2B.
For these reasons, there is no inventive concept. The claim is not patent eligible. Even when viewed as a whole, nothing in the claim adds significantly more to the abstract idea.
Dependent Claims
The following dependent claims merely further define the abstract idea and are, therefore, directed to an abstract idea for similar reasons:
Defining wherein the identifying of the port location comprises using the external body wall data and the internal depth data to identify a positioning on the external body wall for the port location that allows the surgical instrument to access, through the port location, a structure within the internal space while avoiding collision with an additional surgical instrument.– (claim 9).
Defining wherein the identifying of the port location comprises identifying the port location such that the surgical instrument is configured to access a structure within the internal space without the manipulator arm colliding with a different manipulator arm.– (claim 12).
Defining wherein the identifying the port location comprises identifying the port location such that the surgical instrument and the manipulator arm avoid unintentional contact with the patient.– (claim 13)
Defining to execute the instructions to determine, for a candidate port location, at least one of a reachability metric indicating an ability of the surgical instrument to reach a target structure located in the internal space of the patient using [[a]]the candidate port location, an anthropomorphic metric indicating an ease with which a user may manipulate the surgical instrument introduced into the internal space of the patient through the candidate port location, a collision volume for portions of the computer-assisted surgical system proximal to the candidate port location, the collision volume corresponding to a volume swept by the portions of the computer-assisted surgical system proximal to the candidate port location, or a collision metric indicating a likelihood of a collision between portions of the computer-assisted surgical system proximal to the candidate port location; and the identifying of the port location is further based on at least one of the reachability metric, the anthropomorphic metric, the collision volume, or the collision metric – (claim 15). Although, the claim 15 recites the processor configured to perform the steps, its merely a generic computer implementation, which falls under mere instructions to apply the abstract idea on a computer and therefore does not place the abstract idea into a practical application that solves a technological solution in a meaningful way or improve the functionality of the technology or generic computer “itself”. Simply, it’s a generic computer implementation of a mental process rather than a meaningful limitation. Regarding the processor language written at such a high level of generality of structural limitations, the processor language amounts to a generic computer component with mere instructions to implement the abstract idea on a computer.
Defining wherein the performing of the operation comprises identifying, based on the external body wall data and the internal depth data, a set-up position for a manipulator arm of the computer-assisted surgical system. -(claim 16).
The following dependent claims merely further describe the extra-solution activities and therefore, do not amount to significantly more than the judicial exception or integrate the abstract idea into a practical application for similar reasons:
Describing wherein the obtaining of the internal depth data comprises either: i) inserting a distal end of the imaging device into the intern il space through a camera port formed through the external body wall of the patient, or ii) aiming the imaging device at the internal space through the camera port.–(claim 2). The data gathering steps and pre-solution activity are conventional and recited at high level of generality. As such, the abstract idea is not applied, relied on, or used in a meaningful way. No improved to the technology is evident, and the determined visualization of context is not outputted in any way such that the practical benefit is realized.
Describing wherein the obtaining of the external body wall data comprises: receiving the external depth data from the depth sensor; and using the external depth data as the external body wall data representative of the three- dimensional model of the external body wall of the patient.–(claim 3). The data gathering steps and pre-solution activity are conventional and recited at high level of generality. As such, the abstract idea is not applied, relied on, or used in a meaningful way. No improved to the technology is evident, and the determined visualization of context is not outputted in any way such that the practical benefit is realized.
Describing wherein: the imaging device is attached to a manipulator arm of the computer-assisted surgical system while the depth sensor acquires the external depth data and the internal depth data; the computer-assisted surgical system is configured to generate kinematics data for the imaging device while the depth sensor acquires the external depth data and the internal depth data; the processor is further configured to execute the instructions to register, based on the kinematics data, the external body wall data with the internal depth data; and the performing of the operation is based on the registration of the external body wall data with the internal depth data.– (claim 4). The data gathering steps and pre-solution activity are conventional and recited at high level of generality. As such, the abstract idea is not applied, relied on, or used in a meaningful way. No improved to the technology is evident, and the determined visualization of context is not outputted in any way such that the practical benefit is realized.
Describing wherein the visible light camera is configured to acquire a visible light image of the internal space.– (claim 5). The data gathering steps and pre-solution activity are conventional and recited at high level of generality. As such, the abstract idea is not applied, relied on, or used in a meaningful way. No improved to the technology is evident, and the determined visualization of context is not outputted in any way such that the practical benefit is realized.
Defining wherein the identifying of the port location is further based on kinematics data generated by the computer-assisted surgical system.– (Claim 10).
Defining wherein the surgical instrument is attached to a manipulator arm of the computer-assisted surgical system. – (Claim 11)
Describing wherein the processor is further configured to execute the instructions to direct a display to display a graphical representation of the port location.– (claim 14). The data gathering steps and post-solution activity are conventional and recited at high level of generality. As such, the abstract idea is not applied, relied on, or used in a meaningful way. No improved to the technology is evident, and the determined visualization of context is not outputted in any way such that the practical benefit is realized.
Describing wherein the identifying of the set- up position for the manipulator arm is further based on kinematics data generated by the computer-assisted surgical system.– (Claim 17). The data gathering steps and pre-solution activity are conventional and recited at high level of generality. As such, the abstract idea is not applied, relied on, or used in a meaningful way. No improved to the technology is evident, and the determined visualization of context is not outputted in any way such that the practical benefit is realized.
Describing wherein the external depth data and the internal depth data are both generated during a surgical procedure during which the computer-assisted surgical system is configured to perform the procedure with respect to the patient.– (claim 38). The data gathering steps and pre- & post-solution activity are conventional and recited at high level of generality. As such, the abstract idea is not applied, relied on, or used in a meaningful way. No improved to the technology is evident, and the determined visualization of context is not outputted in any way such that the practical benefit is realized.
Taken alone and in combination, the additional elements do not integrate the judicial exception into a practical application at least because the abstract idea is not applied, relied on, or used in a meaningful way. They also do not add anything significantly more than the abstract idea. Their collective functions merely provide computer/electronic implementation and processing, and no additional elements beyond those of the abstract idea. Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements individually. There is no indication that the combination of elements improves the functioning of a computer, output device, improves technology other than the technical field of the claimed invention, etc. Therefore, the claims are rejected as being directed to non-statutory subject matter.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 3-5, 9-19, & 37 are rejected under 35 U.S.C. 103 as being unpatentable over Coste-Maniere et al (US 8170716 B2) in view of Casas (US 20160191887 A1) in view of Refai et al (US 2019/0231220 A1).
Claim 1: Coste-Maniere discloses, A system (FIG. 1, ¶Abstract) comprising:
a memory storing instructions; and a processor communicatively coupled to the memory and configured to execute the instructions to:
-Coste-Maniere discloses a processor coupled with a tele-surgical system and a tangible medium embodying machine-readable code, or software, [Col. 7 l.61-67 to Col. 8 l.1-9], ‘a processor 400 coupled with master control station 200 and cart 300 and a tangible medium 410 embodying machine readable code, or software. The software typically includes instructions which enable various embodiments of the methods of the present invention. The tangible medium 410 may be coupled with the processor 400 for use. Generally, the software may be used with any suitable hardware, such as a personal computer work station with graphics capabilities, such as but not limited to a PENTIUM III.RTM. or equivalent processor with a GEFORCE2.RTM. graphics card [...]’. This tangible medium functions as a memory, storing instructions that enable the various methods of the invention, and it’s coupled with the processor for execution.
perform, based on the external body wall data and the internal depth data, an operation associated with a computer-assisted surgical system configured to perform a procedure with respect to the patient, wherein the performing of the operation comprises identifying a port location on the external body wall of the patient through which the computer-assisted surgical system is to insert a surgical instrument into the internal space of the patient,
-Coste-Maniere describes the processor and software in conjunction with a minimally invasive robotic surgical system, [Col. 2 l.40-54]. The robotic surgical system acts as the computer-assisted surgical system, comprising a robotic manipulator arms configured to hold and articulate surgical instruments in response to a surgeon’s input at a master control station to perform a procedure on a patient, [Col. 7 l.50-60], Claim 1. Therefore, Coste-Maniere teaches to perform an operation associated with a computer-assisted surgical system configured to perform a procedure with respect to the patient.
-Coste-Maniere teaches that the performing of the operation is based on the external body wall data and the internal depth data. Coste-Maniere describes that the system performs surgical planning operations based on comprehensive patient models constructed from acquired data such as CT or MRI scans, [Col. 3 l.56-66 to Col. 4 l.1-14], [Col. 9 l.16-37]. The system utilizes external body wall data by defining “admissible points of entry” which are sites on the external surface of the patient’s volume, [Col. 9 l.57-66 to Col. 10 l.1-4]. Admissible points are defined as specific sites on the surface of the volume of the patient’s body that can safely allow the instruction of robotic arms, endoscopes or other surgical tools without attempting to pass through impassible barriers like bone. Coste-Maniere teaches that this external body wall data utilized is based upon and derived directly from acquired imaging data. The acquired imaging data such as CT or MRI scans are utilized to create a three-dimensional model of the patient’s surgical site, [Col. 4 l.15-24], ‘the operation comprises a surgical operation on a body of a patient and the defined volume comprises a volume of at least a portion of the body. In other embodiments, the operation comprises an operation on a munitions material, the operation including at least one of inspection, maintenance, disabling, and mechanical interaction. Typically, the acquired data comprises imaging data acquired using at least one of computed tomography and magnetic resonance imaging, though other modalities may be used.’, see also [Col. 9 l.16-37]. This imaging data is segmented and reconstructed to defined the closed surface or external boundaries of the patient, which the system uses to evaluate and select the admissible entry port locations on the body wall for the surgical instruments, [Col 3 l.56-67 to Col. 4. 1-14], [Col. 9 l.57-66 to Col. 10 l.1-25]. Coste-Maniere teaches relying on internal depth data to perform its operations, specifically, a surgeon or the system defines internal target areas or target points within the patient, [Col 3 l.56-67 to Col. 4. 1-14]. Coste-Maniere specifically discloses, the imaging data from the CT or MRI scans capture the internal anatomical volume of the patient, [Col. 9 l.16-37]. This imaging data is segmented and reconstructed into a three-dimensional model of the patient’s internal structures and internal target areas, [Col. 9 l.38-56], [Col. 10 l.5-25]. The system utilizes this 3D model, which originates directly from the acquired imaging data, to calculate internal depth measurements, such as evaluating port to target distance between the external entry ports and the internal target, [Col. 15].
-The fundamental purpose of the processor of Coste-Maniere is “identifying advantageous locations for placement of two or more entry ports for performing a surgical operation on a body of a patient”, [Col 3. L40-66 to Col. 4. L.1-66]., The system lists the possible port locations disposed on the surface of the body corresponding to the external body wall. These identified port location are chosen to allow the introduction of the robotic arms, endoscope or any other tools into the internal volume of the patient, [Col. 9 l.57-66 to Col. 10 l.1-4]. Hence, Coste-Maniere does teach, wherein the performing of the operation comprises identifying a port location on the external body wall of the patient through which the computer-assisted surgical system is to insert a surgical instrument into the internal space of the patient.
Coste-Maniere fails to disclose that the external body wall is obtain[ed], by directing an imaging device to scan an external body wall of a patient while the imaging device is external to the patient, external body wall data representative of a three-dimensional model of the external body wall of the patient and based on external depth data for the external body wall;
Note; as written “a” or “an” means one or more. The claim does not require a single imaging device.
Therefore, Casas is relied upon in the context of image guided surgery discloses, obtain, by directing an imaging device (3D scanner) to scan an external body wall of a patient while the imaging device is external to the patient, external body wall data representative of a three-dimensional model of the external body wall of the patient and based on external depth data for the external body wall;
-Casas teaches utilizing a 3D scanner system 110 (i.e., “an” imaging device), which can include a time-of-flight camera, laser scanner, and/or depth camera to capture a 3D surface image of a target portion of the patient, ¶0078, ‘a 3D scanner system 110 may include [...], a time-of-flight 3D laser scanner, [...], a time-of-flight camera, a depth camera, or a combination of these or other devices.’;
-This 3D scanner is externally located to the patient such as being configured to be hand-held, ¶0082, ‘a handheld 3D scanner is used, e.g. a portable 3D scanner forming part of the 3D scanner system 110 is held in the hand and moved around the target portion of the patient, and once the desired surface is obtained, it is placed in a previously defined fixed position.’. Casas explicitly identifies that the targeted surface reconstruction can be superficial skin of the patient, ¶0091, ‘[...] the 3D scanner system 110 obtains the surface reconstruction 112 of a target portion of the patient 118, the surface being e.g. the superficial skin of the patient 118, and the 3D surface is blended by computer means 100 with the 3D volume of the preoperative imaging 102 (e.g. CT scan) of the patient 118.’. The obtained data is representative of a three-dimensional model, because the scanning process generates a dense 3D point cloud representing the surface of the patient, ¶0079, ‘The 3D scanner 110 may be moved around the target portion of the patient 118 to obtain a precise 3D surface image of it through surface reconstruction 112. For example, computer means 100 may receive a dense 3D point cloud provided by the 3D scanning process, that represents the surface of the target portion of the patient 118 by a point cloud construction algorithm,’. This 3D point cloud (i.e., external depth data of the external body wall), which the computer converts into a 3D surface model, ¶0080, ‘computer means 100 make use of techniques of surface reconstruction 112 (e.g. Delaunay triangulation, alpha shapes, ball pivoting, etc.) for converting the point cloud to a 3D surface model (e.g. polygon mesh models, surface models, or solid computer-aided design models).’.
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the obtaining of the external body wall of Coste-Maniere to be obtained by directing an imaging device to scan an external body wall of a patient while the imaging device is external to the patient, external body wall data representative of a three-dimensional model of the external body wall of the patient and based on external depth data for the external body wall as taught by Casas. The motivation to do this yield predictable results such as to provide more accurate, quicker and real-time registration within image-guided navigation systems, as suggested by Casas, ¶0193.
Coste-Maniere in view of Casas fails to disclose:
obtain, by directing the imaging device to scan an internal space of the patient, internal depth data representative of a depth map for the internal space of the patient; and
wherein the imaging device includes a depth sensor and a visible light camera, and
wherein the depth sensor is implemented by a time-of-flight sensor.
However, Refai in the context of three-dimensional scanning and mapping systems’ discloses,
obtain, by directing the imaging device to scan an internal space of the patient, internal depth data representative of a depth map for the internal space of the patient; and
-Refai teaches during a procedure, the endoscopic moves through the surgical environment to scan the area and capture images of the internal space of the patient, ¶0123-0124. As the imaging device scans the internal tissue, cameras (i.e., the time-of-flight camera) capture light reflecting off the tissue, ¶0049, ¶0117. The system’s image reconstruction software then processes this data to extract depth information and 3D position data for each point, ¶0050, ¶0118. Finally, the software uses this 3D position information to construct a 3D image depth map of the entire image, ¶0050, ¶0118.
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the obtained internal depth data of modified Coste-Maniere to be obtained, by directing the imaging device to scan an internal space of the patient, internal depth data representative of a depth map for the internal space of the patient as taught by Refai. The motivation to do this yield predictable results such as to “provide additional information to increase accuracy, improve ability to detect objects of interest that one configuration may have difficulty in detecting but another configuration detects easily, and/or provide supplemental data to inform and/or determine control signals for autonomous or semi-autonomous operation of surgical and endoscopic instruments.”, as suggested by Refai, ¶0119.
wherein the imaging device includes a depth sensor and a visible light camera, and wherein the depth sensor is implemented by a time-of-flight sensor (¶0143, ¶0113).
-Refai describes at ¶0143 the integrated system (130e and 130f) include a visible light source and one or more cameras with visible RBG light sensitivity. Refai explicitly highlights that these integrated systems can include a “forward facing medical scanning and mapping system 10, 10 a, 10 b, 10 c, and/or 10 d”. “10c” is defined by Refai as utilizing the time-of-flight camera and modulated infrared optical source to extract 3D depth information, ¶0113. Therefore, by the explicit teaching that the integrated system 130e/130f can simultaneously house the visible light camera components alongside the forward-facing system 10c, Refai teaches the imaging device includes a depth sensor and a visible light camera, and wherein the depth sensor is implemented by a time-of-flight sensor, “as claimed”.
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the imaging device of modified Coste-Maniere to include a depth sensor and a visible light camera, and wherein the depth sensor is implemented by a time-of-flight sensor as taught by Rafei for the advantage of providing an improved system with such a system being able to obtain the depth of tissue for which the target resolution is achieved, the field of view over which the system can make measurements, and the data made available to the surgeon or control systems for guiding the surgical or endoscopic procedure, as suggested by Refai, ¶0113.
Claim 3: Modified Coste-Maniere discloses all the elements above in claim 1, Coste-Maniere fails to disclose: wherein the obtaining of the external body wall data comprises: receiving the external depth data from the depth sensor; and using the external depth data as the external body wall data representative of the three-dimensional model of the external body wall of the patient.
However, Casas is relied upon above discloses: wherein the obtaining of the external body wall data comprises: receiving the external depth data from the depth sensor; and using the external depth data as the external body wall data representative of the three-dimensional model of the external body wall of the patient.
-Casas teaches utilizing a 3D scanner system 110 (i.e., “an” imaging device), which can include a time-of-flight camera, laser scanner, and/or depth camera to capture a 3D surface image of a target portion of the patient, ¶0078, ‘a 3D scanner system 110 may include [...], a time-of-flight 3D laser scanner, [...], a time-of-flight camera, a depth camera, or a combination of these or other devices.’;
-This 3D scanner is externally located to the patient such as being configured to be hand-held, ¶0082, ‘a handheld 3D scanner is used, e.g. a portable 3D scanner forming part of the 3D scanner system 110 is held in the hand and moved around the target portion of the patient, and once the desired surface is obtained, it is placed in a previously defined fixed position.’. Casas explicitly identifies that the targeted surface reconstruction can be superficial skin of the patient, ¶0091, ‘[...] the 3D scanner system 110 obtains the surface reconstruction 112 of a target portion of the patient 118, the surface being e.g. the superficial skin of the patient 118, and the 3D surface is blended by computer means 100 with the 3D volume of the preoperative imaging 102 (e.g. CT scan) of the patient 118.’. The obtained data is representative of a three-dimensional model, because the scanning process generates a dense 3D point cloud representing the surface of the patient, ¶0079, ‘The 3D scanner 110 may be moved around the target portion of the patient 118 to obtain a precise 3D surface image of it through surface reconstruction 112. For example, computer means 100 may receive a dense 3D point cloud provided by the 3D scanning process, that represents the surface of the target portion of the patient 118 by a point cloud construction algorithm,’. This 3D point cloud (i.e., external depth data of the external body wall), which the computer converts into a 3D surface model, ¶0080, ‘computer means 100 make use of techniques of surface reconstruction 112 (e.g. Delaunay triangulation, alpha shapes, ball pivoting, etc.) for converting the point cloud to a 3D surface model (e.g. polygon mesh models, surface models, or solid computer-aided design models).’.
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the obtaining of the external body wall of modified Coste-Maniere to be obtained by directing an imaging device to scan an external body wall of a patient while the imaging device is external to the patient, external body wall data representative of a three-dimensional model of the external body wall of the patient and based on external depth data for the external body wall as taught by Casas. The motivation to do this yield predictable results such as to provide more accurate, quicker and real-time registration within image-guided navigation systems, as suggested by Casas, ¶0193.
Claim 4: Modified Coste-Maniere discloses all the elements above in claim 3, Coste-Maniere discloses, the processor is further configured to execute the instructions to register, based on the kinematics data, the external body wall data with the internal depth data; and (Abstract, [Col 4 l.1-9]; [Col 4 l.25-33], & [Col 16 l.41-64]-(step 12))
the performing of the operation is based on the registration of the external body wall data with the internal depth data.
-Coste-Maniere describes the processor and software in conjunction with a minimally invasive robotic surgical system, [Col. 2 l.40-54]. The robotic surgical system acts as the computer-assisted surgical system, comprising a robotic manipulator arms configured to hold and articulate surgical instruments in response to a surgeon’s input at a master control station to perform a procedure on a patient, [Col. 7 l.50-60], Claim 1. Therefore, Coste-Maniere teaches to perform an operation associated with a computer-assisted surgical system configured to perform a procedure with respect to the patient.
-Coste-Maniere teaches that the performing of the operation is based on the external body wall data and the internal depth data. Coste-Maniere describes that the system performs surgical planning operations based on comprehensive patient models constructed from acquired data such as CT or MRI scans, [Col. 3 l.56-66 to Col. 4 l.1-14], [Col. 9 l.16-37]. The system utilizes external body wall data by defining “admissible points of entry” which are sites on the external surface of the patient’s volume, [Col. 9 l.57-66 to Col. 10 l.1-4]. Admissible points are defined as specific sites on the surface of the volume of the patient’s body that can safely allow the instruction of robotic arms, endoscopes or other surgical tools without attempting to pass through impassible barriers like bone. Coste-Maniere teaches that this external body wall data utilized is based upon and derived directly from acquired imaging data. The acquired imaging data such as CT or MRI scans are utilized to create a three-dimensional model of the patient’s surgical site, [Col. 4 l.15-24], ‘the operation comprises a surgical operation on a body of a patient and the defined volume comprises a volume of at least a portion of the body. In other embodiments, the operation comprises an operation on a munitions material, the operation including at least one of inspection, maintenance, disabling, and mechanical interaction. Typically, the acquired data comprises imaging data acquired using at least one of computed tomography and magnetic resonance imaging, though other modalities may be used.’, see also [Col. 9 l.16-37]. This imaging data is segmented and reconstructed to defined the closed surface or external boundaries of the patient, which the system uses to evaluate and select the admissible entry port locations on the body wall for the surgical instruments, [Col 3 l.56-67 to Col. 4. 1-14], [Col. 9 l.57-66 to Col. 10 l.1-25]. Coste-Maniere teaches relying on internal depth data to perform its operations, specifically, a surgeon or the system defines internal target areas or target points within the patient, [Col 3 l.56-67 to Col. 4. 1-14]. Coste-Maniere specifically discloses, the imaging data from the CT or MRI scans capture the internal anatomical volume of the patient, [Col. 9 l.16-37]. This imaging data is segmented and reconstructed into a three-dimensional model of the patient’s internal structures and internal target areas, [Col. 9 l.38-56], [Col. 10 l.5-25]. The system utilizes this 3D model, which originates directly from the acquired imaging data, to calculate internal depth measurements, such as evaluating port to target distance between the external entry ports and the internal target, [Col. 15].
-The fundamental purpose of the processor of Coste-Maniere is “identifying advantageous locations for placement of two or more entry ports for performing a surgical operation on a body of a patient”, [Col 3. L40-66 to Col. 4. L.1-66]., The system lists the possible port locations disposed on the surface of the body corresponding to the external body wall. These identified port location are chosen to allow the introduction of the robotic arms, endoscope or any other tools into the internal volume of the patient, [Col. 9 l.57-66 to Col. 10 l.1-4]. Hence, Coste-Maniere does teach, wherein the performing of the operation comprises identifying a port location on the external body wall of the patient through which the computer-assisted surgical system is to insert a surgical instrument into the internal space of the patient.
Coste-Maniere fails to disclose: wherein: the imaging device is attached to a manipulator arm of the computer-assisted surgical system while the depth sensor acquires the external depth data and the internal depth data; the computer-assisted surgical system is configured to generate kinematics data for the imaging device while the depth sensor acquires the external depth data and the internal depth data;
However, Casas is relied upon above discloses: wherein: the imaging device is attached to a manipulator arm of the computer-assisted surgical system while the depth sensor acquires the external depth data; the computer-assisted surgical system is configured to generate kinematics data for the imaging device while the depth sensor acquires the external depth data and the internal depth data;
-Casas disclose, 3D scanner system 110 is attached to an articulated mechanical arm, ¶0036, ¶0066. The computer system controls the movement of this arm during the surgery, ¶0068. This 3D scanner system captures 3D surfaces of both the external superficial skin, ¶0091, and the exposed internal soft tissue and bony structures, ¶0092, ¶0097. Therefore, Casas teaches that the 3D scanner system is attached to the articulated mechanical arm while acquiring both external and internal depth data. See ¶0109 with regard the kinematics data, the computer uses this data from the joint encodes along with the tracking to provide the location and orientation of the 3D scanner system, ¶0035, ¶0058, ¶0104, ¶0110, because this tracking continuously monitors the pose of the 3D scanner throughout the surgical procedure to accurately register the 3D surfaces, the system generates this mechanical joint tracking (i.e., kinematic data) while the depth sensor acquires the external and internal depth data.
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the imaging device of modified Coste-Maniere such that it is attached to a manipulator arm of the computer-assisted surgical system while the depth sensor acquires the external depth data; the computer-assisted surgical system is configured to generate kinematics data for the imaging device while the depth sensor acquires the external depth data and the internal depth data as taught by Casas. The motivation to do this yield predictable results such as to provide more accurate, quicker and real-time registration within image-guided navigation systems, as suggested by Casas, ¶0193.
Claim 5: Modified Coste-Maniere discloses all the elements above in claim 1, Coste-Maniere fails to disclose: wherein the visible light camera is configured to acquire a visible light image of the internal space.
However, Refai is relied upon above discloses, wherein the visible light camera is configured to acquire a visible light image of the internal space.
-Refai teaches during a procedure, the endoscopic moves through the surgical environment to scan the area and capture images of the internal space of the patient, ¶0123-0124. As the imaging device scans the internal tissue, cameras (i.e., the time-of-flight camera) capture light reflecting off the tissue, ¶0049, ¶0117. The system’s image reconstruction software then processes this data to extract depth information and 3D position data for each point, ¶0050, ¶0118. Finally, the software uses this 3D position information to construct a 3D image depth map of the entire image, ¶0050, ¶0118.
-Refai describes at ¶0143 the integrated system (130e and 130f) include a visible light source and one or more cameras with visible RBG light sensitivity. Refai explicitly highlights that these integrated systems can include a “forward facing medical scanning and mapping system 10, 10 a, 10 b, 10 c, and/or 10 d”. “10c” is defined by Refai as utilizing the time-of-flight camera and modulated infrared optical source to extract 3D depth information, ¶0113. Therefore, by the explicit teaching that the integrated system 130e/130f can simultaneously house the visible light camera components alongside the forward-facing system 10c, Refai teaches the imaging device includes a depth sensor and a visible light camera, and wherein the depth sensor is implemented by a time-of-flight sensor, “as claimed”. “The visible light source 138 e and camera 144 components may be integrated with the medical scanning and mapping system 10 to allow capture of visible light images of the tissues scanned by the system 10”-¶0143.
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify visible light camera of modified Coste-Maniere to be configured to acquire a visible light image of the internal space as taught by Rafei for the advantage of providing an improved system with such a system being able to obtain the depth of tissue for which the target resolution is achieved, the field of view over which the system can make measurements, and the data made available to the surgeon or control systems for guiding the surgical or endoscopic procedure, as suggested by Refai, ¶0113.
Claim 9: Modified Coste-Maniere discloses all the elements above in claim 1, Coste-Maniere i discloses: wherein the identifying of the port location comprises using imaging data to identify a positioning on the external body wall for the port location that allows the surgical instrument to access, through the port location, a structure within the internal space while avoiding collision with an additional surgical instrument. (¶Abstract, ‘Methods and apparatus for enhancing surgical planning provide enhanced planning of entry port placement and/or robot position for laparoscopic, robotic, and other minimally invasive surgery. Various embodiments may be used in robotic surgery systems to identify advantageous entry ports for multiple robotic surgical tools into a patient to access a surgical site. Generally, data such as imaging data is processed and used to create a model of a surgical site, which can then be used to select advantageous entry port sites for two or more surgical tools based on multiple criteria. Advantageous robot positioning may also be determined, based on the entry port locations and other factors. Validation and simulation may then be provided to ensure feasibility of the selected port placements and/or robot positions. Such methods, apparatus and systems may also be used in non-surgical contexts, such as for robotic port placement in munitions diffusion or hazardous waste handling.’; Claim 9, ‘wherein the processor is further configured to determine a plurality of entry points into the patient for inserting surgical devices held by the plurality of robotic arms to avoid collisions between any two or more of the plurality of robotic arms during the surgical procedure.’)
-Coste-Maniere describes the processor and software in conjunction with a minimally invasive robotic surgical system, [Col. 2 l.40-54]. The robotic surgical system acts as the computer-assisted surgical system, comprising a robotic manipulator arms configured to hold and articulate surgical instruments in response to a surgeon’s input at a master control station to perform a procedure on a patient, [Col. 7 l.50-60], Claim 1. Therefore, Coste-Maniere teaches to perform an operation associated with a computer-assisted surgical system configured to perform a procedure with respect to the patient.
-Coste-Maniere teaches that the performing of the operation is based on the external body wall data and the internal depth data. Coste-Maniere describes that the system performs surgical planning operations based on comprehensive patient models constructed from acquired data such as CT or MRI scans, [Col. 3 l.56-66 to Col. 4 l.1-14], [Col. 9 l.16-37]. The system utilizes external body wall data by defining “admissible points of entry” which are sites on the external surface of the patient’s volume, [Col. 9 l.57-66 to Col. 10 l.1-4]. Admissible points are defined as specific sites on the surface of the volume of the patient’s body that can safely allow the instruction of robotic arms, endoscopes or other surgical tools without attempting to pass through impassible barriers like bone. Coste-Maniere teaches that this external body wall data utilized is based upon and derived directly from acquired imaging data. The acquired imaging data such as CT or MRI scans are utilized to create a three-dimensional model of the patient’s surgical site, [Col. 4 l.15-24], ‘the operation comprises a surgical operation on a body of a patient and the defined volume comprises a volume of at least a portion of the body. In other embodiments, the operation comprises an operation on a munitions material, the operation including at least one of inspection, maintenance, disabling, and mechanical interaction. Typically, the acquired data comprises imaging data acquired using at least one of computed tomography and magnetic resonance imaging, though other modalities may be used.’, see also [Col. 9 l.16-37]. This imaging data is segmented and reconstructed to defined the closed surface or external boundaries of the patient, which the system uses to evaluate and select the admissible entry port locations on the body wall for the surgical instruments, [Col 3 l.56-67 to Col. 4. 1-14], [Col. 9 l.57-66 to Col. 10 l.1-25]. Coste-Maniere teaches relying on internal depth data to perform its operations, specifically, a surgeon or the system defines internal target areas or target points within the patient, [Col 3 l.56-67 to Col. 4. 1-14]. Coste-Maniere specifically discloses, the imaging data from the CT or MRI scans capture the internal anatomical volume of the patient, [Col. 9 l.16-37]. This imaging data is segmented and reconstructed into a three-dimensional model of the patient’s internal structures and internal target areas, [Col. 9 l.38-56], [Col. 10 l.5-25]. The system utilizes this 3D model, which originates directly from the acquired imaging data, to calculate internal depth measurements, such as evaluating port to target distance between the external entry ports and the internal target, [Col. 15].
-The fundamental purpose of the processor of Coste-Maniere is “identifying advantageous locations for placement of two or more entry ports for performing a surgical operation on a body of a patient”, [Col 3. L40-66 to Col. 4. L.1-66]., The system lists the possible port locations disposed on the surface of the body corresponding to the external body wall. These identified port location are chosen to allow the introduction of the robotic arms, endoscope or any other tools into the internal volume of the patient, [Col. 9 l.57-66 to Col. 10 l.1-4]. Hence, Coste-Maniere does teach, wherein the performing of the operation comprises identifying a port location on the external body wall of the patient through which the computer-assisted surgical system is to insert a surgical instrument into the internal space of the patient.
Claim 10: Modified Coste-Maniere discloses all the elements above in claim 1, Coste-Maniere discloses: wherein the identifying of the port location is further based on kinematics data generated by the computer-assisted surgical system. (Abstract, ‘Methods and apparatus for enhancing surgical planning provide enhanced planning of entry port placement and/or robot position for laparoscopic, robotic, and other minimally invasive surgery. Various embodiments may be used in robotic surgery systems to identify advantageous entry ports for multiple robotic surgical tools into a patient to access a surgical site. Generally, data such as imaging data is processed and used to create a model of a surgical site, which can then be used to select advantageous entry port sites for two or more surgical tools based on multiple criteria. Advantageous robot positioning may also be determined, based on the entry port locations and other factors. Validation and simulation may then be provided to ensure feasibility of the selected port placements and/or robot positions. Such methods, apparatus and systems may also be used in non-surgical contexts, such as for robotic port placement in munitions diffusion or hazardous waste handling.’; Claim 9, ‘wherein the processor is further configured to determine a plurality of entry points into the patient for inserting surgical devices held by the plurality of robotic arms to avoid collisions between any two or more of the plurality of robotic arms during the surgical procedure.’; [Col 4 l.1-9], ‘selecting an advantageous location for placement of each entry port from the list of possible locations for each entry port. In such embodiments, selecting the advantageous locations may be based at least in part on a set of criteria, the criteria including at least one of robot kinematics, robot kinetics, robot work range, deviation of tool entry angle from normal, organ geometry, surgeon defined constraints, robot force limitations, and patient force limitations’; [Col 5 l.9-15], ‘an apparatus for identifying advantageous locations for placement of two or more entry ports for performing an operation within a defined volume having a closed surface includes a computer software module for identifying the advantageous locations, and a computerized simulation device for simulating the operation using the computer software and the advantageous locations’)
Claim 11: Modified Coste-Maniere discloses all the elements above in claim 1, Coste-Maniere discloses, wherein the surgical instrument is attached to a manipulator arm of the computer-assisted surgical system. (¶Abstract, ‘Methods and apparatus for enhancing surgical planning provide enhanced planning of entry port placement and/or robot position for laparoscopic, robotic, and other minimally invasive surgery. Various embodiments may be used in robotic surgery systems to identify advantageous entry ports for multiple robotic surgical tools into a patient to access a surgical site. Generally, data such as imaging data is processed and used to create a model of a surgical site, which can then be used to select advantageous entry port sites for two or more surgical tools based on multiple criteria. Advantageous robot positioning may also be determined, based on the entry port locations and other factors. Validation and simulation may then be provided to ensure feasibility of the selected port placements and/or robot positions. Such methods, apparatus and systems may also be used in non-surgical contexts, such as for robotic port placement in munitions diffusion or hazardous waste handling.’; Claim 9, ‘wherein the processor is further configured to determine a plurality of entry points into the patient for inserting surgical devices held by the plurality of robotic arms to avoid collisions between any two or more of the plurality of robotic arms during the surgical procedure.’)
Claim 12: Modified Coste-Maniere discloses all the elements above in claim 11, Coste-Maniere discloses: wherein the identifying of the port location comprises identifying the port location such that the surgical instrument is configured to access a structure within the internal space without the manipulator arm colliding with a different manipulator arm. (¶Abstract, ‘Methods and apparatus for enhancing surgical planning provide enhanced planning of entry port placement and/or robot position for laparoscopic, robotic, and other minimally invasive surgery. Various embodiments may be used in robotic surgery systems to identify advantageous entry ports for multiple robotic surgical tools into a patient to access a surgical site. Generally, data such as imaging data is processed and used to create a model of a surgical site, which can then be used to select advantageous entry port sites for two or more surgical tools based on multiple criteria. Advantageous robot positioning may also be determined, based on the entry port locations and other factors. Validation and simulation may then be provided to ensure feasibility of the selected port placements and/or robot positions. Such methods, apparatus and systems may also be used in non-surgical contexts, such as for robotic port placement in munitions diffusion or hazardous waste handling.’; Claim 9, ‘wherein the processor is further configured to determine a plurality of entry points into the patient for inserting surgical devices held by the plurality of robotic arms to avoid collisions between any two or more of the plurality of robotic arms during the surgical procedure.’)
Claim 13: Modified Coste-Maniere discloses all the elements above in claim 11, Coste-Maniere discloses: wherein the identifying the port location comprises identifying the port location such that the surgical instrument and the manipulator arm avoid unintentional contact with the patient. (¶Abstract, ‘Methods and apparatus for enhancing surgical planning provide enhanced planning of entry port placement and/or robot position for laparoscopic, robotic, and other minimally invasive surgery. Various embodiments may be used in robotic surgery systems to identify advantageous entry ports for multiple robotic surgical tools into a patient to access a surgical site. Generally, data such as imaging data is processed and used to create a model of a surgical site, which can then be used to select advantageous entry port sites for two or more surgical tools based on multiple criteria. Advantageous robot positioning may also be determined, based on the entry port locations and other factors. Validation and simulation may then be provided to ensure feasibility of the selected port placements and/or robot positions. Such methods, apparatus and systems may also be used in non-surgical contexts, such as for robotic port placement in munitions diffusion or hazardous waste handling.’; Claim 9, ‘wherein the processor is further configured to determine a plurality of entry points into the patient for inserting surgical devices held by the plurality of robotic arms to avoid collisions between any two or more of the plurality of robotic arms during the surgical procedure.’; [Col. 3 l.5-11], ‘robot positioning must also be determined, usually based at least in part on the port placement. Robotic placement will also typically depend on multiple factors, such as robotic-arm collision avoidance, angles of entry for surgical tools, patient anatomy, and/or the like.’; [Col. 4 l. 58-61], ‘ The criteria translate features such as collision avoidance between the manipulator arms and reachability of targeted organs.’; [Col 12 l.28-39], ‘Robot positioning will typically be based on the entry port placement and the robot configuration, such that the robot is positioned in a way that avoids collisions between robot arms and allows the arms to function in performing a given operation without violating any of a number of selected constraints. Constraints may include, for example, number of robot arms and number of degrees of freedom for each arm, potential collisions between the robot arms, potential collisions between an arm and the patient, other potential collisions (e.g. with anesthesia equipment or operating room table), ‘)
Claim 14: Coste-Maniere as modified discloses all the elements above in claim 1, Coste-Maniere disclose, wherein the processor is further configured to execute the instructions to direct a display device to display a graphical representation of the port location. (FIG 8A-8B, [Col. 13 l.12-26], [Col 14 l. 6-32])
Claim 15: Modified Coste-Maniere discloses all the elements above in claim 1, Coste-Maniere discloses: the processor is further configured to execute the instructions to determine, for a candidate port location, at least one of
a reachability metric indicating an ability of the surgical instrument to reach a target structure located in the internal space of the patient using a candidate port location, an anthropomorphic metric indicating an ease with which a user may manipulate the surgical instrument introduced into the internal space of the patient through the candidate port location, the identifying of the port location is further based on at least one of the reachability metric and the anthropomorphic metric ([Col 15 l.36-67 to Col.6 l.1-64 (steps 7-12)]).
Claim 16: Modified Coste-Maniere discloses all the elements above in claim 1, Coste-Maniere discloses, wherein the performing of the operation comprises identifying, based on the imaging data, a set-up position for a manipulator arm of the computer-assisted surgical system. (¶Abstract, ‘Methods and apparatus for enhancing surgical planning provide enhanced planning of entry port placement and/or robot position for laparoscopic, robotic, and other minimally invasive surgery. Various embodiments may be used in robotic surgery systems to identify advantageous entry ports for multiple robotic surgical tools into a patient to access a surgical site. Generally, data such as imaging data is processed and used to create a model of a surgical site, which can then be used to select advantageous entry port sites for two or more surgical tools based on multiple criteria. Advantageous robot positioning may also be determined, based on the entry port locations and other factors. Validation and simulation may then be provided to ensure feasibility of the selected port placements and/or robot positions. Such methods, apparatus and systems may also be used in non-surgical contexts, such as for robotic port placement in munitions diffusion or hazardous waste handling.’; Claim 9, ‘wherein the processor is further configured to determine a plurality of entry points into the patient for inserting surgical devices held by the plurality of robotic arms to avoid collisions between any two or more of the plurality of robotic arms during the surgical procedure.’; [Col 4 l.1-9], ‘selecting an advantageous location for placement of each entry port from the list of possible locations for each entry port. In such embodiments, selecting the advantageous locations may be based at least in part on a set of criteria, the criteria including at least one of robot kinematics, robot kinetics, robot work range, deviation of tool entry angle from normal, organ geometry, surgeon defined constraints, robot force limitations, and patient force limitations’; [Col 4 l.25-33], ‘As mentioned briefly above, some embodiments include determining a position for placement of a robot relative to the defined volume for performing the operation. In such embodiments, determining the position of the robot may be based at least in part on a set of criteria, the criteria including at least one of robot kinematics, robot kinetics, robot work range, deviation of tool entry angle from normal, organ geometry, surgeon defined constraints, robot force limitations, and patient force limitations.’; & [Col 16 l.41-64], ‘(66) Step 12: Determining an advantageous robotic system pre-surgical set-up configuration. Although the robotic system pre-surgical set-up position(s) may be determined empirically, preferably optimization methods are employed according to the principals of the invention. This may include determining positions for a portion or all of the "passive" flexibility degrees of freedom (dofs) of the system (i.e., "passive" in the sense of fixed during surgical treatment manipulation), including the base support position(s), base support orientation(s), set-up joint position(s), and the like. Note that different robotic surgical systems vary considerably in the number of passive pre-surgical set-up dofs. An exemplary process (e.g., probabilistic and gradient descent) may include: 1. defining a set of constraints on the system based on port location and/or trajectory modeling the intervention; 2. defining a cost function based on a measure of goodness including, e.g., separation between the arms; separation from obstacles; maximizing dexterity and/or maneuverability at the end effector(s); 3. running probabilistic optimization to get a set of admissible (constraints realized) solutions (position/orientation of the base and/or values for set-up joints); and/or 4. running gradient descent optimization from the above initial solutions to optimize measure of goodness.’)
Claim 17: Modified Coste-Maniere discloses all the elements above in claim 16, Coste-Maniere discloses, wherein the identifying of the set-up position for the manipulator arm is further based on kinematics data generated by the computer-assisted surgical system. (Abstract, [Col 4 l.1-9]; [Col 4 l.25-33], & [Col 16 l.41-64]-(step 12))
Claim 19: Coste-Maniere A method (FIG. 1, ¶Abstract) comprising:
performing, on the external body wall data and the internal depth data, an operation associated with a computer-assisted surgical system configured to perform a procedure with respect to the patient, wherein the performing of the operation comprises identifying a port location on the external body wall of the patient through which the computer-assisted surgical system is to insert a surgical instrument into the internal space of the patient,
-Coste-Maniere discloses a processor coupled with a tele-surgical system and a tangible medium embodying machine-readable code, or software, [Col. 7 l.61-67 to Col. 8 l.1-9], ‘a processor 400 coupled with master control station 200 and cart 300 and a tangible medium 410 embodying machine readable code, or software. The software typically includes instructions which enable various embodiments of the methods of the present invention. The tangible medium 410 may be coupled with the processor 400 for use. Generally, the software may be used with any suitable hardware, such as a personal computer work station with graphics capabilities, such as but not limited to a PENTIUM III.RTM. or equivalent processor with a GEFORCE2.RTM. graphics card [...]’. This tangible medium functions as a memory, storing instructions that enable the various methods of the invention, and it’s coupled with the processor for execution.
-Coste-Maniere describes the processor and software in conjunction with a minimally invasive robotic surgical system, [Col. 2 l.40-54]. The robotic surgical system acts as the computer-assisted surgical system, comprising a robotic manipulator arms configured to hold and articulate surgical instruments in response to a surgeon’s input at a master control station to perform a procedure on a patient, [Col. 7 l.50-60], Claim 1. Therefore, Coste-Maniere teaches to perform an operation associated with a computer-assisted surgical system configured to perform a procedure with respect to the patient.
-Coste-Maniere teaches that the performing of the operation is based on the external body wall data and the internal depth data. Coste-Maniere describes that the system performs surgical planning operations based on comprehensive patient models constructed from acquired data such as CT or MRI scans, [Col. 3 l.56-66 to Col. 4 l.1-14], [Col. 9 l.16-37]. The system utilizes external body wall data by defining “admissible points of entry” which are sites on the external surface of the patient’s volume, [Col. 9 l.57-66 to Col. 10 l.1-4]. Admissible points are defined as specific sites on the surface of the volume of the patient’s body that can safely allow the instruction of robotic arms, endoscopes or other surgical tools without attempting to pass through impassible barriers like bone. Coste-Maniere teaches that this external body wall data utilized is based upon and derived directly from acquired imaging data. The acquired imaging data such as CT or MRI scans are utilized to create a three-dimensional model of the patient’s surgical site, [Col. 4 l.15-24], ‘the operation comprises a surgical operation on a body of a patient and the defined volume comprises a volume of at least a portion of the body. In other embodiments, the operation comprises an operation on a munitions material, the operation including at least one of inspection, maintenance, disabling, and mechanical interaction. Typically, the acquired data comprises imaging data acquired using at least one of computed tomography and magnetic resonance imaging, though other modalities may be used.’, see also [Col. 9 l.16-37]. This imaging data is segmented and reconstructed to defined the closed surface or external boundaries of the patient, which the system uses to evaluate and select the admissible entry port locations on the body wall for the surgical instruments, [Col 3 l.56-67 to Col. 4. 1-14], [Col. 9 l.57-66 to Col. 10 l.1-25]. Coste-Maniere teaches relying on internal depth data to perform its operations, specifically, a surgeon or the system defines internal target areas or target points within the patient, [Col 3 l.56-67 to Col. 4. 1-14]. Coste-Maniere specifically discloses, the imaging data from the CT or MRI scans capture the internal anatomical volume of the patient, [Col. 9 l.16-37]. This imaging data is segmented and reconstructed into a three-dimensional model of the patient’s internal structures and internal target areas, [Col. 9 l.38-56], [Col. 10 l.5-25]. The system utilizes this 3D model, which originates directly from the acquired imaging data, to calculate internal depth measurements, such as evaluating port to target distance between the external entry ports and the internal target, [Col. 15].
-The fundamental purpose of the processor of Coste-Maniere is “identifying advantageous locations for placement of two or more entry ports for performing a surgical operation on a body of a patient”, [Col 3. L40-66 to Col. 4. L.1-66]., The system lists the possible port locations disposed on the surface of the body corresponding to the external body wall. These identified port location are chosen to allow the introduction of the robotic arms, endoscope or any other tools into the internal volume of the patient, [Col. 9 l.57-66 to Col. 10 l.1-4]. Hence, Coste-Maniere does teach, wherein the performing of the operation comprises identifying a port location on the external body wall of the patient through which the computer-assisted surgical system is to insert a surgical instrument into the internal space of the patient.
Coste-Maniere fails to disclose that the external body wall is obtain[ed], by directing an imaging device to scan an external body wall of a patient while the imaging device is external to the patient, external body wall data representative of a three-dimensional model of the external body wall of the patient and based on external depth data for the external body wall;
Note; as written “a” or “an” means one or more. The claim does not require a single imaging device.
Therefore, Casas is relied upon in the context of image guided surgery discloses, obtaining, by directing an imaging device (3D scanner) to scan an external body wall of a patient while the imaging device is external to the patient, external body wall data representative of a three-dimensional model of the external body wall of the patient and based on external depth data for the external body wall;
-Casas teaches utilizing a 3D scanner system 110 (i.e., “an” imaging device), which can include a time-of-flight camera, laser scanner, and/or depth camera to capture a 3D surface image of a target portion of the patient, ¶0078, ‘a 3D scanner system 110 may include [...], a time-of-flight 3D laser scanner, [...], a time-of-flight camera, a depth camera, or a combination of these or other devices.’;
-This 3D scanner is externally located to the patient such as being configured to be hand-held, ¶0082, ‘a handheld 3D scanner is used, e.g. a portable 3D scanner forming part of the 3D scanner system 110 is held in the hand and moved around the target portion of the patient, and once the desired surface is obtained, it is placed in a previously defined fixed position.’. Casas explicitly identifies that the targeted surface reconstruction can be superficial skin of the patient, ¶0091, ‘[...] the 3D scanner system 110 obtains the surface reconstruction 112 of a target portion of the patient 118, the surface being e.g. the superficial skin of the patient 118, and the 3D surface is blended by computer means 100 with the 3D volume of the preoperative imaging 102 (e.g. CT scan) of the patient 118.’. The obtained data is representative of a three-dimensional model, because the scanning process generates a dense 3D point cloud representing the surface of the patient, ¶0079, ‘The 3D scanner 110 may be moved around the target portion of the patient 118 to obtain a precise 3D surface image of it through surface reconstruction 112. For example, computer means 100 may receive a dense 3D point cloud provided by the 3D scanning process, that represents the surface of the target portion of the patient 118 by a point cloud construction algorithm,’. This 3D point cloud (i.e., external depth data of the external body wall), which the computer converts into a 3D surface model, ¶0080, ‘computer means 100 make use of techniques of surface reconstruction 112 (e.g. Delaunay triangulation, alpha shapes, ball pivoting, etc.) for converting the point cloud to a 3D surface model (e.g. polygon mesh models, surface models, or solid computer-aided design models).’.
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the obtaining of the external body wall of Coste-Maniere to be obtained by directing an imaging device to scan an external body wall of a patient while the imaging device is external to the patient, external body wall data representative of a three-dimensional model of the external body wall of the patient and based on external depth data for the external body wall as taught by Casas. The motivation to do this yield predictable results such as to provide more accurate, quicker and real-time registration within image-guided navigation systems, as suggested by Casas, ¶0193.
Coste-Maniere in view of Casas fails to disclose:
obtaining, by directing imaging device to scan an internal space of the patient, internal depth data representative of a depth map for the internal space of the patient; and
wherein the imaging device includes a depth sensor and a visible light camera, and wherein the depth sensor is implemented by a time-of-flight sensor.
However, Refai in the context of three-dimensional scanning and mapping systems’ discloses,
obtaining, by directing imaging device to scan an internal space of the patient, internal depth data representative of a depth map for the internal space of the patient; and
-Refai teaches during a procedure, the endoscopic moves through the surgical environment to scan the area and capture images of the internal space of the patient, ¶0123-0124. As the imaging device scans the internal tissue, cameras (i.e., the time-of-flight camera) capture light reflecting off the tissue, ¶0049, ¶0117. The system’s image reconstruction software then processes this data to extract depth information and 3D position data for each point, ¶0050, ¶0118. Finally, the software uses this 3D position information to construct a 3D image depth map of the entire image, ¶0050, ¶0118.
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the obtained internal depth data of modified Coste-Maniere to be obtained, by directing the imaging device to scan an internal space of the patient, internal depth data representative of a depth map for the internal space of the patient as taught by Refai. The motivation to do this yield predictable results such as to “provide additional information to increase accuracy, improve ability to detect objects of interest that one configuration may have difficulty in detecting but another configuration detects easily, and/or provide supplemental data to inform and/or determine control signals for autonomous or semi-autonomous operation of surgical and endoscopic instruments.”, as suggested by Refai, ¶0119.
wherein the imaging device includes a depth sensor and a visible light camera, and wherein the depth sensor is implemented by a time-of-flight sensor. (¶0143, ¶0113).
-Refai describes at ¶0143 the integrated system (130e and 130f) include a visible light source and one or more cameras with visible RBG light sensitivity. Refai explicitly highlights that these integrated systems can include a “forward facing medical scanning and mapping system 10, 10 a, 10 b, 10 c, and/or 10 d”. “10c” is defined by Refai as utilizing the time-of-flight camera and modulated infrared optical source to extract 3D depth information, ¶0113. Therefore, by the explicit teaching that the integrated system 130e/130f can simultaneously house the visible light camera components alongside the forward-facing system 10c, Refai teaches the imaging device includes a depth sensor and a visible light camera, and wherein the depth sensor is implemented by a time-of-flight sensor, “as claimed”.
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the imaging device of modified Coste-Maniere to include a depth sensor and a visible light camera, and wherein the depth sensor is implemented by a time-of-flight sensor as taught by Rafei for the advantage of providing an improved system with such a system being able to obtain the depth of tissue for which the target resolution is achieved, the field of view over which the system can make measurements, and the data made available to the surgeon or control systems for guiding the surgical or endoscopic procedure, as suggested by Refai, ¶0113.
Claim 37: Coste-Maniere discloses, A non-transitory computer-readable medium storing instructions that, when executed, direct a processor to:
-Coste-Maniere discloses a processor coupled with a tele-surgical system and a tangible medium embodying machine-readable code, or software, [Col. 7 l.61-67 to Col. 8 l.1-9], ‘a processor 400 coupled with master control station 200 and cart 300 and a tangible medium 410 embodying machine readable code, or software. The software typically includes instructions which enable various embodiments of the methods of the present invention. The tangible medium 410 may be coupled with the processor 400 for use. Generally, the software may be used with any suitable hardware, such as a personal computer work station with graphics capabilities, such as but not limited to a PENTIUM III.RTM. or equivalent processor with a GEFORCE2.RTM. graphics card [...]’. This tangible medium functions as a memory, storing instructions that enable the various methods of the invention, and it’s coupled with the processor for execution.
perform, based on the external body wall data and the internal depth data, an operation associated with a computer-assisted surgical system configured to perform a procedure with respect to the patient, wherein the performing of the operation comprises identifying a port location on the external body wall of the patient through which the computer-assisted surgical system is to insert a surgical instrument into the internal space of the patient,
-Coste-Maniere describes the processor and software in conjunction with a minimally invasive robotic surgical system, [Col. 2 l.40-54]. The robotic surgical system acts as the computer-assisted surgical system, comprising a robotic manipulator arms configured to hold and articulate surgical instruments in response to a surgeon’s input at a master control station to perform a procedure on a patient, [Col. 7 l.50-60], Claim 1. Therefore, Coste-Maniere teaches to perform an operation associated with a computer-assisted surgical system configured to perform a procedure with respect to the patient.
-Coste-Maniere teaches that the performing of the operation is based on the external body wall data and the internal depth data. Coste-Maniere describes that the system performs surgical planning operations based on comprehensive patient models constructed from acquired data such as CT or MRI scans, [Col. 3 l.56-66 to Col. 4 l.1-14], [Col. 9 l.16-37]. The system utilizes external body wall data by defining “admissible points of entry” which are sites on the external surface of the patient’s volume, [Col. 9 l.57-66 to Col. 10 l.1-4]. Admissible points are defined as specific sites on the surface of the volume of the patient’s body that can safely allow the instruction of robotic arms, endoscopes or other surgical tools without attempting to pass through impassible barriers like bone. Coste-Maniere teaches that this external body wall data utilized is based upon and derived directly from acquired imaging data. The acquired imaging data such as CT or MRI scans are utilized to create a three-dimensional model of the patient’s surgical site, [Col. 4 l.15-24], ‘the operation comprises a surgical operation on a body of a patient and the defined volume comprises a volume of at least a portion of the body. In other embodiments, the operation comprises an operation on a munitions material, the operation including at least one of inspection, maintenance, disabling, and mechanical interaction. Typically, the acquired data comprises imaging data acquired using at least one of computed tomography and magnetic resonance imaging, though other modalities may be used.’, see also [Col. 9 l.16-37]. This imaging data is segmented and reconstructed to defined the closed surface or external boundaries of the patient, which the system uses to evaluate and select the admissible entry port locations on the body wall for the surgical instruments, [Col 3 l.56-67 to Col. 4. 1-14], [Col. 9 l.57-66 to Col. 10 l.1-25]. Coste-Maniere teaches relying on internal depth data to perform its operations, specifically, a surgeon or the system defines internal target areas or target points within the patient, [Col 3 l.56-67 to Col. 4. 1-14]. Coste-Maniere specifically discloses, the imaging data from the CT or MRI scans capture the internal anatomical volume of the patient, [Col. 9 l.16-37]. This imaging data is segmented and reconstructed into a three-dimensional model of the patient’s internal structures and internal target areas, [Col. 9 l.38-56], [Col. 10 l.5-25]. The system utilizes this 3D model, which originates directly from the acquired imaging data, to calculate internal depth measurements, such as evaluating port to target distance between the external entry ports and the internal target, [Col. 15].
-The fundamental purpose of the processor of Coste-Maniere is “identifying advantageous locations for placement of two or more entry ports for performing a surgical operation on a body of a patient”, [Col 3. L40-66 to Col. 4. L.1-66]., The system lists the possible port locations disposed on the surface of the body corresponding to the external body wall. These identified port location are chosen to allow the introduction of the robotic arms, endoscope or any other tools into the internal volume of the patient, [Col. 9 l.57-66 to Col. 10 l.1-4]. Hence, Coste-Maniere does teach, wherein the performing of the operation comprises identifying a port location on the external body wall of the patient through which the computer-assisted surgical system is to insert a surgical instrument into the internal space of the patient.
Coste-Maniere fails to disclose that the external body wall is obtain[ed], by directing an imaging device to scan an external body wall of a patient while the imaging device is external to the patient, external body wall data representative of a three-dimensional model of the external body wall of the patient and based on external depth data for the external body wall;
Note; as written “a” or “an” means one or more. The claim does not require a single imaging device.
Therefore, Casas is relied upon in the context of image guided surgery discloses, obtain, by directing an imaging device (3D scanner) to scan an external body wall of a patient while the imaging device is external to the patient, external body wall data representative of a three-dimensional model of the external body wall of the patient and based on external depth data for the external body wall;
-Casas teaches utilizing a 3D scanner system 110 (i.e., “an” imaging device), which can include a time-of-flight camera, laser scanner, and/or depth camera to capture a 3D surface image of a target portion of the patient, ¶0078, ‘a 3D scanner system 110 may include [...], a time-of-flight 3D laser scanner, [...], a time-of-flight camera, a depth camera, or a combination of these or other devices.’;
-This 3D scanner is externally located to the patient such as being configured to be hand-held, ¶0082, ‘a handheld 3D scanner is used, e.g. a portable 3D scanner forming part of the 3D scanner system 110 is held in the hand and moved around the target portion of the patient, and once the desired surface is obtained, it is placed in a previously defined fixed position.’. Casas explicitly identifies that the targeted surface reconstruction can be superficial skin of the patient, ¶0091, ‘[...] the 3D scanner system 110 obtains the surface reconstruction 112 of a target portion of the patient 118, the surface being e.g. the superficial skin of the patient 118, and the 3D surface is blended by computer means 100 with the 3D volume of the preoperative imaging 102 (e.g. CT scan) of the patient 118.’. The obtained data is representative of a three-dimensional model, because the scanning process generates a dense 3D point cloud representing the surface of the patient, ¶0079, ‘The 3D scanner 110 may be moved around the target portion of the patient 118 to obtain a precise 3D surface image of it through surface reconstruction 112. For example, computer means 100 may receive a dense 3D point cloud provided by the 3D scanning process, that represents the surface of the target portion of the patient 118 by a point cloud construction algorithm,’. This 3D point cloud (i.e., external depth data of the external body wall), which the computer converts into a 3D surface model, ¶0080, ‘computer means 100 make use of techniques of surface reconstruction 112 (e.g. Delaunay triangulation, alpha shapes, ball pivoting, etc.) for converting the point cloud to a 3D surface model (e.g. polygon mesh models, surface models, or solid computer-aided design models).’.
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the obtaining of the external body wall of Coste-Maniere to be obtained by directing an imaging device to scan an external body wall of a patient while the imaging device is external to the patient, external body wall data representative of a three-dimensional model of the external body wall of the patient and based on external depth data for the external body wall as taught by Casas. The motivation to do this yield predictable results such as to provide more accurate, quicker and real-time registration within image-guided navigation systems, as suggested by Casas, ¶0193.
Coste-Maniere in view of Casas fails to disclose:
obtain, by directing the imaging device to scan an internal space of the patient, internal depth data representative of a depth map for the internal space of the patient; and
wherein the imaging device includes a depth sensor and a visible light camera, and wherein the depth sensor is implemented by a time-of-flight sensor.
However, Refai in the context of three-dimensional scanning and mapping systems’ discloses,
obtain, by directing the imaging device to scan an internal space of the patient, internal depth data representative of a depth map for the internal space of the patient; and
-Refai teaches during a procedure, the endoscopic moves through the surgical environment to scan the area and capture images of the internal space of the patient, ¶0123-0124. As the imaging device scans the internal tissue, cameras (i.e., the time-of-flight camera) capture light reflecting off the tissue, ¶0049, ¶0117. The system’s image reconstruction software then processes this data to extract depth information and 3D position data for each point, ¶0050, ¶0118. Finally, the software uses this 3D position information to construct a 3D image depth map of the entire image, ¶0050, ¶0118.
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the obtained internal depth data of modified Coste-Maniere to be obtained, by directing the imaging device to scan an internal space of the patient, internal depth data representative of a depth map for the internal space of the patient as taught by Refai. The motivation to do this yield predictable results such as to “provide additional information to increase accuracy, improve ability to detect objects of interest that one configuration may have difficulty in detecting but another configuration detects easily, and/or provide supplemental data to inform and/or determine control signals for autonomous or semi-autonomous operation of surgical and endoscopic instruments.”, as suggested by Refai, ¶0119.
wherein the imaging device includes a depth sensor and a visible light camera, and wherein the depth sensor is implemented by a time-of-flight sensor (¶0143, ¶0113).
-Refai describes at ¶0143 the integrated system (130e and 130f) include a visible light source and one or more cameras with visible RBG light sensitivity. Refai explicitly highlights that these integrated systems can include a “forward facing medical scanning and mapping system 10, 10 a, 10 b, 10 c, and/or 10 d”. “10c” is defined by Refai as utilizing the time-of-flight camera and modulated infrared optical source to extract 3D depth information, ¶0113. Therefore, by the explicit teaching that the integrated system 130e/130f can simultaneously house the visible light camera components alongside the forward-facing system 10c, Refai teaches the imaging device includes a depth sensor and a visible light camera, and wherein the depth sensor is implemented by a time-of-flight sensor, “as claimed”.
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the imaging device of modified Coste-Maniere to include a depth sensor and a visible light camera, and wherein the depth sensor is implemented by a time-of-flight sensor as taught by Rafei for the advantage of providing an improved system with such a system being able to obtain the depth of tissue for which the target resolution is achieved, the field of view over which the system can make measurements, and the data made available to the surgeon or control systems for guiding the surgical or endoscopic procedure, as suggested by Refai, ¶0113.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Coste-Maniere et al (US 8170716 B2) in view of Casas (US 20160191887 A1) in view of Refai et al (US 2019/0231220 A1), as applied to claim 1, in further view of Nakaguchi (US20160113484A1).
Claim 2: Modified Coste-Maniere discloses all the elements above in claim 1, Coste-Maniere fails to disclose: wherein the obtaining of the internal depth data comprises either: i) inserting a distal end of the imaging device into the internal space through a camera port formed through the external body wall of the patient, or ii) aiming the imaging device at the internal space through the camera port.
However, Nakaguchi in the context of camera and trocar surgery assistance disclose: wherein the obtaining of the internal depth data comprises either: i) inserting a distal end of the imaging device into the internal space through a camera port formed through the external body wall of the patient, or ii) aiming the imaging device at the internal space through the camera port (¶Abstract, ¶0079-0080, ¶0075-0077, ¶0091-0092).
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the obtaining of the internal depth data of modified Coste-Maniere such that it comprises either: i) inserting a distal end of the imaging device into the internal space through a camera port formed through the external body wall of the patient, or ii) aiming the imaging device at the internal space through the camera port as taught by Nakaguchi. The motivation to do this yield predictable results such as to provide a wider surgical field and reduce the burden on the surgeon, as suggested by Nakaguchi, ¶0085, ¶0092.
Claim 38 is rejected under 35 U.S.C. 103 as being unpatentable over Coste-Maniere et al (US 8170716 B2) in view of Casas (US 20160191887 A1) in view of Refai et al (US 2019/0231220 A1), in further view of Clavin et al (US 2013/0296682).
Claim 38: Modified Coste-Maniere discloses all the elements above in claim 1, Coste-Maniere fails to disclose, wherein the external depth data and the internal depth data are both generated during a surgical procedure during which the computer-assisted surgical system is configured to perform the procedure with respect to the patient.
However, Clavin in the context of integrated internal and external images by an imaging device discloses, wherein the external depth data and the internal depth data are both generated during a surgical procedure during which the computer-assisted surgical system is configured to perform the procedure with respect to the patient (¶0009, ¶0011, ¶0013, ¶0015, ¶0017-0019, ¶0020, ¶0022, ¶0041).
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify wherein the external and internal depth data of modified Coste-Maniere such that both are generated during a surgical procedure during which the computer-assisted surgical system is configured to perform the procedure with respect to the patient as taught by Clavin. The motivation to do this yield predictable results such as to accurately identify internal structures, mapping internal images to the patients physical body, and improving surgical safety.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Nicholas Robinson whose telephone number is (571)272-9019. The examiner can normally be reached M-F 9:00AM-5:00PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pascal Bui-Pho can be reached at (571) 272-2714. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.A.R./Examiner, Art Unit 3798
/PASCAL M BUI PHO/Supervisory Patent Examiner, Art Unit 3798