DETAILED ACTION
This office action is in response to the communication received on 05/12/2026 concerning application no. 16/304,405 filed on 11/26/2018.
Claims 21-23 and 25-33 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 05/12/2026 have been fully considered but they are not persuasive.
Regarding the 112(a) and 112(b) rejections, Applicant alleges the amendments address the rejections.
Examiner disagrees. While some rejections may have been adequately addressed (see below), the amendments fail to remedy all the rejections established in the prior action, filed 09/10/2025. Additionally, the amendments introduce new 112 issues (see below). Mere allegation that the amended language addresses the rejections is insufficient to traverse the rejections.
Examiner maintains the 112 rejections as established below.
Applicant's arguments filed 05/12/2026 have been fully considered but they are not persuasive.
Regarding the 101 rejection, Applicant argues “Claims 21-33 stand rejected under 35 USC § 101 as being allegedly directed to an abstract idea without significantly more. Applicant asserts that claim 21, as amended, claims an apparatus comprising of an image fusion controller and a display controller. Applicant further asserts that a human mind cannot reasonably “compute a spatial transformation of the at least one ultrasound image onto an endoscopic image space of the at least one endoscopic image” or “fuse the spatial transformation of the at least one ultrasound image with the at least one endoscopic image into a fused image,” as recited by Applicant’s claim.”
Examiner disagree. Mere allegation that the human mind cannot compute a spatial transformation or perform fusion is conclusory and without support. MPEP 716.01(c) establishes “Arguments presented by the applicant cannot take the place of evidence in the record. In re Schulze, 346 F.2d 600, 602, 145 USPQ 716, 718 (CCPA 1965) and In re De Blauwe, 736 F.2d 699, 705, 222 USPQ 191, 196 (Fed. Cir. 1984).” MPEP 2106.04(a)(2).III. establishes that the “courts do not distinguish between mental processes that are performed entirely in the human mind and mental processes that require a human to use a physical aid (e.g., pen and paper or a slide rule) to perform the claim limitation. See, e.g., Benson, 409 U.S. at 67, 65, 175 USPQ at 674-75, 674 (noting that the claimed "conversion of [binary-coded decimal] numerals to pure binary numerals can be done mentally," i.e., "as a person would do it by head and hand."); Synopsys, Inc. v. Mentor Graphics Corp., 839 F.3d 1138, 1139, 120 USPQ2d 1473, 1474 (Fed. Cir. 2016) (holding that claims to a mental process of "translating a functional description of a logic circuit into a hardware component description of the logic circuit" are directed to an abstract idea, because the claims "read on an individual performing the claimed steps mentally or with pencil and paper").” MPEP 2106.04(a)(2).III.B establishes that “The use of a physical aid (e.g., pencil and paper or a slide rule) to help perform a mental step (e.g., a mathematical calculation) does not negate the mental nature of the limitation, but simply accounts for variations in memory capacity from one person to another.”
Examiner maintains the rejection.
Applicant’s arguments with respect to claim 21 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 21-23 and 25-33 rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 21 is rejected under 112(a) for the following reasons:
Recites “based on the detection of the laparoscopic ultrasound probe, retrieve at least one ultrasound image from the ultrasound images, wherein the at least one ultrasound image corresponds to the at least one endoscopic image: compute a spatial transformation of the at least one ultrasound image onto an endoscopic image space of the at least one endoscopic image: and fuse the spatial transformation of the at least one ultrasound image with the at least one endoscopic image into a fused image” (emphasis added). This is necessarily a computer-implemented functional limitation, consistent with those described in MPEP § 2161.01(I). For computer-implemented functional limitations, applicant must disclose how the result is achieved, and it is “not enough that one skilled in the art could write a program to achieve the claimed function because the specification [itself] must explain how the inventor intends to achieve the claimed function in order to satisfy the written description requirement,” as from the cited section of the manual, referencing Vasudevan Software, Inc. v. MicroStrategy, Inc., 782 F.3d 671, 681-683, 114 USPQ2d 1349, 1356, 1357 (Fed. Cir. 2015). The specification merely repeats this result without delineating the necessary “steps/procedure” (or flowcharts of sufficient specificity) as to how these results are achieved. See [0011], [0013] (paragraphs as numbered in applicant’s pre-grant publication). While the specification also describes what is targeted for transformation (2D planar US image or 3D scan US image, as in [0063]), the details for transforming either of these targets from the US probe being detected within the FOV of the endoscope are not delineated. In view of the evidence cited, applicant has not demonstrated sufficiency of disclosure for these computer-implemented functions.
Recites “process ultrasound images of the anatomical region received from a laparoscopic ultrasound probe”. This is understood to be a computer-implemented functional limitation which requires disclosure of the underlying algorithm(s) for obtaining the result in order to comply with the written description requirement. See MPEP § 2161.01(I). While paragraphs 0052-54 disclose the processing of the endoscopic images and the processing of the fusion, the specification fails to disclose the processing of the ultrasound ultrasound images of the anatomical region received from a laparoscopic ultrasound probe. Rather, the processing is for the operative generation of the fused images. Therefore, the claim contains subject matter which is not described in the specification in such a way as to reasonably convey to one with ordinary skill in the art that the inventor had possession of the claim invention at the time of filing.
Recites “compute a spatial transformation of the at least one ultrasound image onto an endoscopic image space of the at least one endoscopic image: and fuse the spatial transformation of the at least one ultrasound image with the at least one endoscopic image into a fused image”. While the specification discloses the performance of an image transformation (at least paragraphs 0011-13, 0053-54, 0061-66), the specification does not disclose a spatial transformation. Rather, the specification is disclosing transformation in the context of images rather than in the spatial context. There is no recitation or disclose of a “spatial transformation”. Therefore, the claim contains subject matter which is not described in the specification in such a way as to reasonably convey to one with ordinary skill in the art that the inventor had possession of the claim invention at the time of filing.
Recites “compute a spatial transformation of the at least one ultrasound image onto an endoscopic image space of the at least one endoscopic image: and fuse the spatial transformation of the at least one ultrasound image with the at least one endoscopic image into a fused image”. This is understood to be a computer-implemented functional limitation which requires disclosure of the underlying algorithm(s) for obtaining the result in order to comply with the written description requirement. See MPEP § 2161.01(I). Given that there is no disclosure of a spatial transformations, the specification fails to disclose the computation required for spatial transformations. Therefore, the claim contains subject matter which is not described in the specification in such a way as to reasonably convey to one with ordinary skill in the art that the inventor had possession of the claim invention at the time of filing.
Claim 23 is rejected under 112(a) for the following reasons:
Recites “compute a second spatial transformation of the at least one ultrasound image, wherein the second spatial transformation is of the spatial transformation, and wherein the second spatial transformation is onto an image space of an instrument inserted into the anatomical region; and control a display of the second spatial transformation at the instrument inserted into the anatomical region”. While the specification discloses the performance of an image transformation (at least paragraphs 0011-13, 0053-54, 0061-66), the specification does not disclose a spatial transformation. Rather, the specification is disclosing transformation in the context of images rather than in the spatial context. There is no recitation or disclose of a “spatial transformation”. Additionally, the specification does not disclose the transformation in the context of an instrument and the ultrasound image. Therefore, the claim contains subject matter which is not described in the specification in such a way as to reasonably convey to one with ordinary skill in the art that the inventor had possession of the claim invention at the time of filing.
Recites “compute a second spatial transformation of the at least one ultrasound image, wherein the second spatial transformation is of the spatial transformation, and wherein the second spatial transformation is onto an image space of an instrument inserted into the anatomical region”. This is understood to be a computer-implemented functional limitation which requires disclosure of the underlying algorithm(s) for obtaining the result in order to comply with the written description requirement. See MPEP § 2161.01(I). Given that there is no disclosure of a spatial transformations, the specification fails to disclose the computation required for spatial transformations. Therefore, the claim contains subject matter which is not described in the specification in such a way as to reasonably convey to one with ordinary skill in the art that the inventor had possession of the claim invention at the time of filing.
Claim 25 is rejected under 112(a) for the following reasons:
Recites “compute a second spatial transformation of the at least one ultrasound image, wherein the second spatial transformation is of the spatial transformation, and wherein the second spatial transformation is onto an image space of an instrument inserted into the anatomical region: and generate for display the second spatial transformation”. While the specification discloses the performance of an image transformation (at least paragraphs 0011-13, 0053-54, 0061-66), the specification does not disclose a spatial transformation. Rather, the specification is disclosing transformation in the context of images rather than in the spatial context. There is no recitation or disclose of a “spatial transformation”. Additionally, the specification does not disclose the transformation in the context of an instrument and the ultrasound image. Therefore, the claim contains subject matter which is not described in the specification in such a way as to reasonably convey to one with ordinary skill in the art that the inventor had possession of the claim invention at the time of filing.
Recites “compute a second spatial transformation of the at least one ultrasound image, wherein the second spatial transformation is of the spatial transformation, and wherein the second spatial transformation is onto an image space of an instrument inserted into the anatomical region”. This is understood to be a computer-implemented functional limitation which requires disclosure of the underlying algorithm(s) for obtaining the result in order to comply with the written description requirement. See MPEP § 2161.01(I). Given that there is no disclosure of a spatial transformations, the specification fails to disclose the computation required for spatial transformations. Therefore, the claim contains subject matter which is not described in the specification in such a way as to reasonably convey to one with ordinary skill in the art that the inventor had possession of the claim invention at the time of filing.
Claims that are not discussed above but are cited to be rejected under 35 U.S.C. 112(a) are also rejected because they inherit the deficiencies of the claims they respectively depend upon.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 21-23 and 25-33 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 is indefinite for the following reasons:
Recites “endoscopic images”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art if the endoscopic images comprise the endoscopic image established in the preceding claim element or are separate and distinct from the singular endoscopic image. If it is the later, it is further unclear what the relationship of the singular endoscopic image is with the remaining claim language.
Applicant is encouraged to provide consistent and clear language.
Recites “an endoscope”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art if the “endoscope” is the same as the “endoscope” established in the preceding claim element or is a separate and distinct feature.
Applicant is encouraged to provide consistent and clear language.
Recites “ultrasound images”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art if the ultrasound images comprise the ultrasound image established in the preceding claim element or are separate and distinct from the singular ultrasound image. If it is the later, it is further unclear what the relationship of the singular ultrasound image is with the remaining claim language.
Applicant is encouraged to provide consistent and clear language.
Recites “a laparoscopic ultrasound probe”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art if the “laparoscopic ultrasound probe” is the same as the “laparoscopic probe” established in the preceding claim element or is a separate and distinct feature.
Applicant is encouraged to provide consistent and clear language.
Recites “at least one endoscopic image”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art if the “at least one endoscopic image” is the same as the “endoscopic image” established in the preceding claim element or is a separate and distinct feature.
Applicant is encouraged to provide consistent and clear language.
Recites “at least one ultrasound image”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art if the “at least one ultrasound image” is the same as the “ultrasound image” established in the preceding claim element or is a separate and distinct feature.
Applicant is encouraged to provide consistent and clear language.
Recites “an endoscopic image space of the at least one endoscopic image”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art what an endoscopic image space is. One interpretation is that it is the field of view. Another interpretation that it is a data set associated to the endoscopic image.
Applicant is encouraged to provide consistent and clear language.
Recites “based on the detection of the laparoscopic ultrasound probe, retrieve at least one ultrasound image from the ultrasound images, wherein the at least one ultrasound image corresponds to the at least one endoscopic image; compute a spatial transformation of the at least one ultrasound image onto an endoscopic image space of the at least one endoscopic image: and fuse the spatial transformation of the at least one ultrasound image with the at least one endoscopic image into a fused image” (emphasis added). This claim element is indefinite. While it is clear how the transformation between each of the endoscopic image space and the ultrasound image space is derived from detection of the laparoscopic US probe within a FOV of the endoscope, it is unclear how the image fusion itself is specifically derived from detecting the ultrasound probe within the FOV of the endoscope, and the specification does not clarify this feature. For example, if the ultrasound probe is within the endoscopic FOV, those skilled would understand that the images acquired from each modality at that instant will not be of the same anatomical region (i.e., respective image space for each will be different at that time, and fusion of disparate anatomical regions at that instant would be impractical, if not impossible). If applicant intends that the image fusion is controlled between the two images based on the transformation between the two image spaces alone and it is only the transformation that is a precursor to fusion, rather than the fusion itself, which is specifically “derived from” the ultrasound probe appearing within the endoscope FOV and being thus detected, this is not effectively conveyed by the claim language and clarification of the claim is required by delineating these as separate steps/functions, for example.
Applicant is encouraged to provide consistent and clear language.
Claim 23 is indefinite for the following reasons:
Recites “wherein the second spatial transformation is of the spatial transformation”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art what the claim is attempting to establish. It is unclear if this is a subsequent transformation of the transformation in claim 21 or attempting to establish it as the same transformation. In the case of the former, it is further unclear in what manner the transformation is associated to the instrument as the initial transformation is between the ultrasound and endoscopy. If it is the later, it is further unclear in what manner this is a second transformation.
Applicant is encouraged to provide consistent and clear language.
Recites “wherein the second spatial transformation is onto an image space of an instrument inserted into the anatomical region”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art what instrument is being referred to. It is unclear if the claim is attempting to introduce a third modality as it is referring to an image space of the instrument. It is further unclear if the image space is a field of view or some associated data set.
Applicant is encouraged to provide consistent and clear language.
Recites “compute a second spatial transformation of the at least one ultrasound image, wherein the second spatial transformation is of the spatial transformation, and wherein the second spatial transformation is onto an image space of an instrument inserted into the anatomical region”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art if the second spatial transformation is being applied to the instrument or the ultrasound image.
Applicant is encouraged to provide consistent and clear language.
Recites “control a display of the second spatial transformation at the instrument”. This claim element is indefinite. It would be unclear in what manner a transformation can be targeted at an instrument. A transformation is a mathematical concept that is applied in a spatial or imaging context.
Applicant is encouraged to provide consistent and clear language.
The claim is generally narrative and indefinite, failing to conform with current U.S. practice. They appear to be a literal translation into English from a foreign document and are replete with grammatical and idiomatic errors.
Claim 25 is indefinite for the following reasons:
Recites “wherein the second spatial transformation is of the spatial transformation”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art what the claim is attempting to establish. It is unclear if this is a subsequent transformation of the transformation in claim 21 or attempting to establish it as the same transformation. In the case of the former, it is further unclear in what manner the transformation is associated to the instrument as the initial transformation is between the ultrasound and endoscopy. If it is the later, it is further unclear in what manner this is a second transformation.
Applicant is encouraged to provide consistent and clear language.
Recites “wherein the second spatial transformation is onto an image space of an instrument inserted into the anatomical region”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art what instrument is being referred to. It is unclear if the claim is attempting to introduce a third modality as it is referring to an image space of the instrument. It is further unclear if the image space is a field of view or some associated data set.
Applicant is encouraged to provide consistent and clear language.
Recites “compute a second spatial transformation of the at least one ultrasound image, wherein the second spatial transformation is of the spatial transformation, and wherein the second spatial transformation is onto an image space of an instrument inserted into the anatomical region”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art if the second spatial transformation is being applied to the instrument or the ultrasound image.
Applicant is encouraged to provide consistent and clear language.
The claim is generally narrative and indefinite, failing to conform with current U.S. practice. They appear to be a literal translation into English from a foreign document and are replete with grammatical and idiomatic errors.
Claim 28 is indefinite for the following reasons:
Recites “at least one reference planar view of the anatomical model”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art if the “at least one reference planar view of the anatomical model” is the same as the “reference planes of an anatomical model” established in claim 21 or is a separate and distinct feature.
Applicant is encouraged to provide consistent and clear language.
Recites “a display of at least one of the endoscopic image, the ultrasound image, and the fused image”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art if the “a display of at least one of the endoscopic image, the ultrasound image, and the fused image” is the same as the “display an overlay of the fused image and reference planes of the anatomical model on the at least one endoscopic image” established in claim 21 or is a separate and distinct feature.
Applicant is encouraged to provide consistent and clear language.
Claim 30 is indefinite for the following reasons:
Recites “display of a virtual laparoscopic probe”. This claim element is indefinite. An image is a virtual representation. Given that the claim 21 appears to establish that the laparoscopic probe is in the field of view of the endoscope as it is detected in the endoscopic image, it would be unclear if this is establishing the probe in the field of view or some other feature. If it is a different feature, it is further unclear as the probe in the field of view is a virtual representation as captured by the endoscope.
Applicant is encouraged to provide consistent and clear language.
Claim 32 is indefinite for the following reasons:
Recites “at least one reference planar view of the anatomical model”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art if the “at least one reference planar view of the anatomical model” is the same as the “reference planes of an anatomical model” established in claim 21 or is a separate and distinct feature.
Applicant is encouraged to provide consistent and clear language.
Claims that are not discussed above but are cited to be rejected under 35 U.S.C. 112(b) are also rejected because they inherit the indefiniteness of the claims they respectively depend upon.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 21-23 and 25-33 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 21 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Statutory Category: Yes - The claims recite an image fusion workstation for an image-based fusion of an endoscopic image of an anatomical region generated by an endoscope and an ultrasound image of the anatomical region generated by a laparoscopic probe and therefore, is an apparatus.
Step 2A, Prong 1, Judicial Exception: Yes - The claim recites the limitation “detect the laparoscopic ultrasound probe in at least one endoscopic image from the endoscopic images: based on the detection of the laparoscopic ultrasound probe, retrieve at least one ultrasound image from the ultrasound images, wherein the at least one ultrasound image corresponds to the at least one endoscopic image: compute a spatial transformation of the at least one ultrasound image onto an endoscopic image space of the at least one endoscopic image: and fuse the spatial transformation of the at least one ultrasound image with the at least one endoscopic image into a fused image: and retrieve reference planes of an anatomical model of the anatomical region, wherein the reference planes correspond to the endoscopic image space of the at least one endoscopic image”.
This limitation, as drafted, is a process step that, under its broadest reasonable interpretation, covers the performance of the limitation in the mind as it is regarding a concept relating to the detection of an ultrasound probe in the view of an endoscopic image and using that as the basis to perform a image fusion based on an image transformation between the image spaces and the retrieval of corresponding reference planes. Additionally, the performance of an spatial transformation is the implementation of a mathematical concept based on the associated spatial information. That is, nothing in the claim element precludes the step from practically being performed in the mind and/or being performed with the aid of a pen and paper. Accordingly, the claim recites a mental process-type abstract idea.
Step 2A, Prong 2, Integrated into Practical Application: No - The claim recites the following additional elements: “an image fusion controller configured to: process endoscopic images of the anatomical region received from an endoscope: process ultrasound images of the anatomical region received from a laparoscopic ultrasound probe: a display controller configured to: and generate for display an overlay of the fused image and reference planes of the anatomical model on the at least one endoscopic image”. Processing the images is a data gathering step that is a form of a pre-solution insignificant activity. Displaying the overlay is a display step merely amounts to a post-solution insignificant activity. The use of a processor does not integrate the judicial exception into a practical application as it is merely used to perform the judicial exception.
These additional elements, taken individually or in combination, merely amount to insignificant pre/post-solution activities and do not integrate the judicial exception into a practical application. This claim is therefore directed to an abstract idea.
Step 2B, Inventive Concept: No - Similarly to Step 2A Prong 2, the additional claim elements merely recite insignificant extra-solution activities, which do not amount to significantly more than the judicial exception. For these reasons, there is no inventive concept in the claim. In light of the above, claim 21 is ineligible.
Claim 22 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 and Step 2A, Prong 1, Judicial Exception are discussed above in the claim 21 rejection.
Claim 22 recites the following elements: “wherein the image fusion controller and the display controller are further configured to receive a user interactive selection of a depth of the ultrasound images received from the laparoscopic ultrasound probe”. This claim element is a mere data gathering step which amounts to a pre-solution insignificant activity. The use of processors does not integrate the judicial exception into a practical application as it is merely used to perform the judicial exception. This pre-solution insignificant activity does not integrate the judicial exception into a practical application nor does it contain an inventive step. In light of above, claim 22 is ineligible.
Claim 23 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Statutory Category: Yes - The claims recite an image fusion workstation and therefore, is an apparatus.
Step 2A, Prong 1, Judicial Exception: Yes - The claim recites the limitation “compute a second spatial transformation of the at least one ultrasound image, wherein the second spatial transformation is of the spatial transformation, and wherein the second spatial transformation is onto an image space of an instrument inserted into the anatomical region”.
This limitation, as drafted, is a process step that, under its broadest reasonable interpretation, covers the performance of the limitation in the mind as it is regarding a concept relating to the computation of a spatial transform with respect to the ultrasound image and the tool. Additionally, the performance of an spatial transformation is the implementation of a mathematical concept based on the associated spatial information. That is, nothing in the claim element precludes the step from practically being performed in the mind and/or being performed with the aid of a pen and paper. Accordingly, the claim recites a mental process-type abstract idea.
Step 2A, Prong 2, Integrated into Practical Application: No - The claim recites the following additional elements: “wherein the image fusion controller and the display controller are further configured to; control a display of the second spatial transformation at the instrument inserted into the anatomical region”. Display of the transformation is a display step that is a form of a pre-solution insignificant activity. The use of a processor does not integrate the judicial exception into a practical application as it is merely used to perform the judicial exception.
These additional elements, taken individually or in combination, merely amount to insignificant pre/post-solution activities and do not integrate the judicial exception into a practical application. This claim is therefore directed to an abstract idea.
Step 2B, Inventive Concept: No - Similarly to Step 2A Prong 2, the additional claim elements merely recite insignificant extra-solution activities, which do not amount to significantly more than the judicial exception. For these reasons, there is no inventive concept in the claim. In light of the above, claim 23 is ineligible.
Claim 25 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Statutory Category: Yes - The claims recite an image fusion workstation and therefore, is an apparatus.
Step 2A, Prong 1, Judicial Exception: Yes - The claim recites the limitation “compute a second spatial transformation of the at least one ultrasound image, wherein the second spatial transformation is of the spatial transformation, and wherein the second spatial transformation is onto an image space of an instrument inserted into the anatomical region”.
This limitation, as drafted, is a process step that, under its broadest reasonable interpretation, covers the performance of the limitation in the mind as it is regarding a concept relating to the computation of a spatial transform with respect to the ultrasound image and the tool. Additionally, the performance of an spatial transformation is the implementation of a mathematical concept based on the associated spatial information. That is, nothing in the claim element precludes the step from practically being performed in the mind and/or being performed with the aid of a pen and paper. Accordingly, the claim recites a mental process-type abstract idea.
Step 2A, Prong 2, Integrated into Practical Application: No - The claim recites the following additional elements: “wherein the image fusion controller and the display controller are further configured to: and generate for display the second spatial transformation”. Display of the transformation is a display step that is a form of a pre-solution insignificant activity. The use of a processor does not integrate the judicial exception into a practical application as it is merely used to perform the judicial exception.
These additional elements, taken individually or in combination, merely amount to insignificant pre/post-solution activities and do not integrate the judicial exception into a practical application. This claim is therefore directed to an abstract idea.
Step 2B, Inventive Concept: No - Similarly to Step 2A Prong 2, the additional claim elements merely recite insignificant extra-solution activities, which do not amount to significantly more than the judicial exception. For these reasons, there is no inventive concept in the claim. In light of the above, claim 25 is ineligible.
Claim 26 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Statutory Category: Yes - The claims recite an image fusion workstation and therefore, is an apparatus.
Step 2A, Prong 1, Judicial Exception: Yes - The claim recites the limitation “control a registration of the anatomical model of the anatomical region and the fused image based on an image transformation between a volumetric image space of the anatomical model and the endoscopic image space of the endoscope”.
This limitation, as drafted, is a process step that, under its broadest reasonable interpretation, covers the performance of the limitation in the mind as it is regarding a concept relating to registration of an anatomical model and fusion according to an image transformation. That is, nothing in the claim element precludes the step from practically being performed in the mind and/or being performed with the aid of a pen and paper. Accordingly, the claim recites a mental process-type abstract idea.
Step 2A, Prong 2, Integrated into Practical Application: No - The claim recites the following additional elements: “wherein, the image fusion controller is further configured to”. The use of a processor does not integrate the judicial exception into a practical application as it is merely used to perform the judicial exception.
These additional elements, taken individually or in combination, merely amount to insignificant pre/post-solution activities and do not integrate the judicial exception into a practical application. This claim is therefore directed to an abstract idea.
Step 2B, Inventive Concept: No - Similarly to Step 2A Prong 2, the additional claim elements merely recite insignificant extra-solution activities, which do not amount to significantly more than the judicial exception. For these reasons, there is no inventive concept in the claim. In light of the above, claim 26 is ineligible.
Claim 27 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Statutory Category: Yes - The claims recite an image fusion workstation and therefore, is an apparatus.
Step 2A, Prong 1, Judicial Exception: Yes - The claim recites the limitation “wherein the anatomical model is an anatomical atlas of an anatomical structure within the anatomical region”.
This limitation, as drafted, is a process step that, under its broadest reasonable interpretation, covers the performance of the limitation in the mind as it is regarding a concept relating to registration according to an anatomical atlas. That is, nothing in the claim element precludes the step from practically being performed in the mind and/or being performed with the aid of a pen and paper. Accordingly, the claim recites a mental process-type abstract idea.
Step 2A, Prong 2, Integrated into Practical Application: No - The claim does not contain additional elements. Therefore, the claim does not integrate the judicial exception into a practical application.
Step 2B, Inventive Concept: No - Similar to Step 2A Prong 2, the claim does not contain additional elements. For these reasons, there is no inventive concept in the claim. In light of the above, claim 27 is ineligible.
Claim 28 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 and Step 2A, Prong 1, Judicial Exception are discussed above in the claim 26 rejection.
Claim 28 recites the following elements: “wherein the image fusion controller and the display controller are further configured to control a display of at least one reference planar view of the anatomical model overlaid on a display of at least one of the endoscopic image, the ultrasound image, and the fused image”. This claim element is a mere display step which amounts to a post-solution insignificant activity. The use of processors does not integrate the judicial exception into a practical application as it is merely used to perform the judicial exception. This post-solution insignificant activity does not integrate the judicial exception into a practical application nor does it contain an inventive step. In light of above, claim 28 is ineligible.
Claim 29 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 and Step 2A, Prong 1, Judicial Exception are discussed above in the claim 26 rejection.
Claim 29 recites the following elements: “wherein the image fusion controller and the display controller are further configured to control a display a targeted ultrasound image among the ultrasound images of the anatomical region received by from the laparoscopic ultrasound probe based on the registration between the anatomical model of the anatomical region and the fused image”. This claim element is a mere display step which amounts to a post-solution insignificant activity. The use of processors does not integrate the judicial exception into a practical application as it is merely used to perform the judicial exception. This post-solution insignificant activity does not integrate the judicial exception into a practical application nor does it contain an inventive step. In light of above, claim 29 is ineligible.
Claim 30 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 and Step 2A, Prong 1, Judicial Exception are discussed above in the claim 26 rejection.
Claim 30 recites the following elements: “wherein the image fusion controller and the display controller are further configured to control a display of a virtual laparoscopic probe and the ultrasound image in conjunction with a display of the anatomical model based on the registered anatomical model of the anatomical region and the fused image”. This claim element is a mere display step which amounts to a post-solution insignificant activity. The use of processors does not integrate the judicial exception into a practical application as it is merely used to perform the judicial exception. This post-solution insignificant activity does not integrate the judicial exception into a practical application nor does it contain an inventive step. In light of above, claim 30 is ineligible.
Claim 31 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 and Step 2A, Prong 1, Judicial Exception are discussed above in the claim 21 rejection.
Claim 31 recites the following elements: “wherein the image fusion controller and the display controller are further configured to control a display of the ultrasound image indicative of a status of a surgical procedure performed within the anatomical region”. This claim element is a mere display step which amounts to a post-solution insignificant activity. The use of processors does not integrate the judicial exception into a practical application as it is merely used to perform the judicial exception. This post-solution insignificant activity does not integrate the judicial exception into a practical application nor does it contain an inventive step. In light of above, claim 31 is ineligible.
Claim 32 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 and Step 2A, Prong 1, Judicial Exception are discussed above in the claim 25 rejection.
Claim 32 recites the following elements: “wherein the image fusion controller and the display controller are further configured to display at least one reference planar view of the anatomical model indicative of a status of a surgical procedure performed within the anatomical region”. This claim element is a mere display step which amounts to a post-solution insignificant activity. The use of processors does not integrate the judicial exception into a practical application as it is merely used to perform the judicial exception. This post-solution insignificant activity does not integrate the judicial exception into a practical application nor does it contain an inventive step. In light of above, claim 32 is ineligible.
Claim 33 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 and Step 2A, Prong 1, Judicial Exception are discussed above in the claim 25 rejection.
Claim 33 recites the following elements: “wherein the image fusion controller and the display controller are further configured to provide a virtual display of at least one desired position of the laparoscopic ultrasound probe relative to the anatomical model based on the registered anatomical model of the anatomical region and fused image”. This claim element is a mere display step which amounts to a post-solution insignificant activity. The use of processors does not integrate the judicial exception into a practical application as it is merely used to perform the judicial exception. This post-solution insignificant activity does not integrate the judicial exception into a practical application nor does it contain an inventive step. In light of above, claim 33 is ineligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 21-22, 26, and 28-31 are rejected under 35 U.S.C. 103 as being unpatentable over Shekhar et al. (PGPUB No. US 2014/0303491) in view of DiMaio et al. (PGPUB No. US 2009/0036902).
Regarding claim 21, Shekhar teaches a image fusion workstation for an image-based fusion of an endoscopic image of an anatomical region generated by an endoscope and an ultrasound image of the anatomical region generated by a laparoscopic probe, the image fusion workstation comprising:
an image fusion controller (AR fusion module. Paragraph 0060 teaches it has major circuitry components) configured to:
process endoscopic images of the anatomical region received from an endoscope: process ultrasound images of the anatomical region received from a laparoscopic ultrasound probe: detect the laparoscopic ultrasound probe in at least one endoscopic image from the endoscopic images: based on the detection of the laparoscopic ultrasound probe, retrieve at least one ultrasound image from the ultrasound images, wherein the at least one ultrasound image corresponds to the at least one endoscopic image; compute a spatial transformation of the at least one ultrasound image onto an endoscopic image space of the at least one endoscopic image: and fuse the spatial transformation of the at least one ultrasound image with the at least one endoscopic image into a fused image (Paragraph 0061 teaches that the tracking device 120 determines the pose of the LUS and the stereoscopy device. The images are overlaid onto each other and allow for stereoscopic visualization. Fig. 11 shows that overlay of the LUS in the camera view. Paragraph 0071 teaches that the AR system presents the LUS image in correct reference to the laparoscopic camera view in real time. Fig. 11 shows that the LUS is present in the laparoscopic camera view. Paragraph 0124 teaches that the laparoscope and LUS images can be overlayed based on the transform of the LUS in relation to the laparoscope and the calculated MIPs are composited with the stereoscopic video images. Paragraph 0124 teaches that the imaging devices are spatially tracked and that the data of both are overlaid. The tracked position of the laparoscope is transformed with respect to the coordinates of the ultrasound data. Paragraph 0100 teaches the registration based on the relationship of the LUS with the stereoscopic view. Their coordinate and field of view is considered and shown in Fig. 10. Paragraphs 0080-81 teaches that the AR fusion module receives the endoscopic and the ultrasound image data from both instruments. Paragraph 01274 teaches the overlay based on the coordinate information. Fig. 11 shows that overlay of the LUS in the camera view).
However, Shekhar is silent regarding an image fusion workstation, a display controller configured to;
retrieve reference planes of an anatomical model of the anatomical region, wherein the reference planes correspond to the endoscopic image space of the at least one endoscopic image: and generate for display an overlay of the fused image and reference planes of the anatomical model on the at least one endoscopic image.
In an analogous imaging field of endeavor, regarding image fusion, DiMaio teaches an image fusion workstation, detect the laparoscopic ultrasound probe in at least one endoscopic image from the endoscopic images; a display controller configured to; retrieve reference planes of an anatomical model of the anatomical region, wherein the reference planes correspond to the endoscopic image space of the at least one endoscopic image: and generate for display an overlay of the fused image and reference planes of the anatomical model on the at least one endoscopic image (Paragraphs 0156 teaches that preoperative images from the MRI, CT, ultrasound, PET or other modalities is stored and displayed. The images are volumetric images via an overlay with the endoscopic image data. Paragraph 0170-72 teaches that slices for the volumetric image can be display and integrated. See Figs. 24-26. Paragraph 0081 teaches that the LUS and the endoscopic views allow for different structures of the anatomy to be observed with the tool and the tool can be observed and implemented with respect to a 3D model. Paragraphs 0053-54 teaches that the LapUS probe is tracked and the probe is calibrated and integrated with respect to the endoscopic system based on its position and registration. Paragraphs 0136-39 teaches that the LapUS is observed in the visual field of the endoscope’s field of view. This allows for the observation in three dimensions. The LapUS is shown in the flashlight image that shows the ultrasound image that is electronically attached to the surgical instrument image and this is performed in real time. See Fig. 21. Paragraphs 0145-46 teaches that the graphical tool icons can be used to indicate the respective instrument in the view of the endoscope. See Fig. 22. Paragraphs 0118-19 teaches that the overlay is selectively done with the medical image volume onto the fused ultrasound and endoscopic images. This combined image is rendered for display. The LapUS is captured and adapted for the monitoring of the implementation of the tool. Paragraph 0115 teaches the manipulation of the instruments and the tool arms according to the medical image volume. Paragraph 0130 teaches that the medical volume data can be overlayed with the video and live images along with the LapUS feed).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Shekhar with DiMaio’s teaching of overlay of an anatomical model reference plane with the endoscopic and ultrasound images. This modified apparatus would allow the user to move the LUS to see different views of the anatomical structure and manipulate a tool via input and the generation of a 3D model (Paragraph 0081 of DiMaio). Furthermore, the modification allows for precise placement of surgical tools prior to actual cutting and shearing of tissue (Paragraph 0190 of DiMaio).
Regarding claim 22, modified Shekhar teaches the image fusion workstation in claim 21, as discussed above.
Shekhar further teaches an image fusion workstation, wherein the image fusion controller and the display controller are further configured to receive a user interactive selection of a depth of the ultrasound images received from the laparoscopic ultrasound probe (Paragraph 0125 teaches that the user can select the ultrasound thickness and adjust the location in the volume prior to the volume rendering. Paragraph 0124 teaches that the MIP is used in the registration of the ultrasound onto the laparoscopic video).
Alternatively, in an analogous imaging field of endeavor, regarding image fusion, DiMaio teaches an image fusion workstation, wherein the image fusion controller and the display controller are further configured to receive a user interactive selection of a depth of the ultrasound images received from the laparoscopic ultrasound probe (Paragraphs 0087-88 teaches that the selected ultrasound image can be displayed at the proper depth in the master display as an overlay to the camera view. Paragraph 0148 teaches that the surgeon can control the depth variation based on the input devices).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Shekhar with DiMaio’s teaching of depth selection of the ultrasound image. This modified apparatus would allow the user to move the LUS to see different views of the anatomical structure and manipulate a tool via input and the generation of a 3D model (Paragraph 0081 of DiMaio). Furthermore, the modification allows for precise placement of surgical tools prior to actual cutting and shearing of tissue (Paragraph 0190 of DiMaio).
Regarding claim 26, modified Shekhar teaches the image fusion workstation in claim 21, as discussed above.
Shekhar further teaches an image fusion workstation, wherein, the image fusion controller is further configured to control a registration of the anatomical model of the anatomical region and the fused image based on an image transformation between a volumetric image space of the anatomical model and the endoscopic image space of the endoscope (Paragraphs 0088-89 teach that the composite stereoscopic images can be created through registration form a surface model that is derived from pre-operative tomography images and a surface model that is reconstructed from the endoscope. The organ deformation model can also be used and processed by the AR fusion module. Paragraph 0082 teaches that the preoperative volumes undergo deformable image registration algorithm as well as the intra operative tomographic images).
Regarding claim 28, modified Shekhar teaches the image fusion workstation in claim 26, as discussed above.
However, Shekhar is silent regarding an image fusion workstation, wherein the image fusion controller and the display controller are further configured to control a display of at least one reference planar view of the anatomical model overlaid on a display of at least one of the endoscopic image, the ultrasound image, and the fused image.
In an analogous imaging field of endeavor, regarding image fusion, DiMaio teaches an image fusion workstation, wherein the image fusion controller and the display controller are further configured to control a display of at least one reference planar view of the anatomical model overlaid on a display of at least one of the endoscopic image, the ultrasound image, and the fused image (Paragraphs 0156 teaches that preoperative images from the MRI, CT, ultrasound, PET or other modalities is stored and displayed. The images are volumetric images via an overlay with the endoscopic image data. Paragraph 0170-72 teaches that slices for the volumetric image can be display and integrated. See Figs. 24-26. Paragraph 0081 teaches that the LUS and the endoscopic views allow for different structures of the anatomy to be observed with the tool and the tool can be observed and implemented with respect to a 3D model. Paragraphs 0053-54 teaches that the LapUS probe is tracked and the probe is calibrated and integrated with respect to the endoscopic system based on its position and registration. Paragraphs 0136-39 teaches that the LapUS is observed in the visual field of the endoscope’s field of view. This allows for the observation in three dimensions. The LapUS is shown in the flashlight image that shows the ultrasound image that is electronically attached to the surgical instrument image and this is performed in real time. See Fig. 21. Paragraphs 0145-46 teaches that the graphical tool icons can be used to indicate the respective instrument in the view of the endoscope. See Fig. 22. Paragraphs 0118-19 teaches that the overlay is selectively done with the medical image volume onto the fused ultrasound and endoscopic images. This combined image is rendered for display. The LapUS is captured and adapted for the monitoring of the implementation of the tool. Paragraph 0115 teaches the manipulation of the instruments and the tool arms according to the medical image volume. Paragraph 0130 teaches that the medical volume data can be overlayed with the video and live images along with the LapUS feed).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Shekhar with DiMaio’s teaching of overlay of an anatomical model reference plane with the endoscopic and ultrasound images. This modified apparatus would allow the user to move the LUS to see different views of the anatomical structure and manipulate a tool via input and the generation of a 3D model (Paragraph 0081 of DiMaio). Furthermore, the modification allows for precise placement of surgical tools prior to actual cutting and shearing of tissue (Paragraph 0190 of DiMaio).
Regarding claim 29, modified Shekhar teaches the image fusion workstation in claim 26, as discussed above.
However, Shekhar is silent regarding an image fusion workstation, wherein the image fusion controller and the display controller are further configured to control a display a targeted ultrasound image among the ultrasound images of the anatomical region received by from the laparoscopic ultrasound probe based on the registration between the anatomical model of the anatomical region and the fused image.
In an analogous imaging field of endeavor, regarding image fusion, DiMaio teaches an image fusion workstation, wherein the image fusion controller and the display controller are further configured to control a display a targeted ultrasound image among the ultrasound images of the anatomical region received by from the laparoscopic ultrasound probe based on the registration between the anatomical model of the anatomical region and the fused image (Paragraphs 0156 teaches that preoperative images from the MRI, CT, ultrasound, PET or other modalities is stored and displayed. The images are volumetric images via an overlay with the endoscopic image data. Paragraph 0170-72 teaches that slices for the volumetric image can be display and integrated. See Figs. 24-26. Paragraph 0081 teaches that the LUS and the endoscopic views allow for different structures of the anatomy to be observed with the tool and the tool can be observed and implemented with respect to a 3D model. Paragraphs 0053-54 teaches that the LapUS probe is tracked and the probe is calibrated and integrated with respect to the endoscopic system based on its position and registration. Paragraphs 0136-39 teaches that the LapUS is observed in the visual field of the endoscope’s field of view. This allows for the observation in three dimensions. The LapUS is shown in the flashlight image that shows the ultrasound image that is electronically attached to the surgical instrument image and this is performed in real time. See Fig. 21. Paragraphs 0145-46 teaches that the graphical tool icons can be used to indicate the respective instrument in the view of the endoscope. See Fig. 22. Paragraphs 0118-19 teaches that the overlay is selectively done with the medical image volume onto the fused ultrasound and endoscopic images. This combined image is rendered for display. The LapUS is captured and adapted for the monitoring of the implementation of the tool. Paragraph 0115 teaches the manipulation of the instruments and the tool arms according to the medical image volume. Paragraph 0130 teaches that the medical volume data can be overlayed with the video and live images along with the LapUS feed).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Shekhar with DiMaio’s teaching of the display of a targeted ultrasound image of the anatomical region. This modified apparatus would allow the user to move the LUS to see different views of the anatomical structure and manipulate a tool via input and the generation of a 3D model (Paragraph 0081 of DiMaio). Furthermore, the modification allows for precise placement of surgical tools prior to actual cutting and shearing of tissue (Paragraph 0190 of DiMaio).
Regarding claim 30, modified Shekhar teaches the image fusion workstation in claim 26, as discussed above.
However, Shekhar is silent regarding an image fusion workstation, wherein the image fusion controller and the display controller are further configured to control a display of a virtual laparoscopic probe and the ultrasound image in conjunction with a display of the anatomical model based on the registered anatomical model of the anatomical region and the fused image.
In an analogous imaging field of endeavor, regarding image fusion, DiMaio teaches an image fusion workstation, wherein the image fusion controller and the display controller are further configured to control a display of a virtual laparoscopic probe and the ultrasound image in conjunction with a display of the anatomical model based on the registered anatomical model of the anatomical region and the fused image (Paragraphs 0156 teaches that preoperative images from the MRI, CT, ultrasound, PET or other modalities is stored and displayed. The images are volumetric images via an overlay with the endoscopic image data. Paragraph 0170-72 teaches that slices for the volumetric image can be display and integrated. See Figs. 24-26. Paragraph 0081 teaches that the LUS and the endoscopic views allow for different structures of the anatomy to be observed with the tool and the tool can be observed and implemented with respect to a 3D model. Paragraphs 0053-54 teaches that the LapUS probe is tracked and the probe is calibrated and integrated with respect to the endoscopic system based on its position and registration. Paragraphs 0136-39 teaches that the LapUS is observed in the visual field of the endoscope’s field of view. This allows for the observation in three dimensions. The LapUS is shown in the flashlight image that shows the ultrasound image that is electronically attached to the surgical instrument image and this is performed in real time. See Fig. 21. Paragraphs 0145-46 teaches that the graphical tool icons can be used to indicate the respective instrument in the view of the endoscope. See Fig. 22. Paragraphs 0118-19 teaches that the overlay is selectively done with the medical image volume onto the fused ultrasound and endoscopic images. This combined image is rendered for display. The LapUS is captured and adapted for the monitoring of the implementation of the tool. Paragraph 0115 teaches the manipulation of the instruments and the tool arms according to the medical image volume. Paragraph 0130 teaches that the medical volume data can be overlayed with the video and live images along with the LapUS feed).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Shekhar with DiMaio’s teaching of displaying the model and the fused image with the laparoscopic image. This modified apparatus would allow the user to move the LUS to see different views of the anatomical structure and manipulate a tool via input and the generation of a 3D model (Paragraph 0081 of DiMaio). Furthermore, the modification allows for precise placement of surgical tools prior to actual cutting and shearing of tissue (Paragraph 0190 of DiMaio).
Regarding claim 31, Shekhar teaches the image fusion workstation in claim 21, as discussed above.
Shekhar further teaches image fusion workstation, wherein the image fusion controller and the display controller are further configured to control a display of the ultrasound image indicative of a status of a surgical procedure performed within the anatomical region (Paragraph 0127 teaches that the marker 2000 and 2005 can show the entry and target position of the needle. This is done with the overlaid composite image data to aid the surgeon performing the procedure. The procedures can be precision of laparoscopic ablations, biopsies, drainages, aspirations, therapeutic delivery, vascular access, and anesthesia delivery. Fig. 20 shows the tracking to the needle and the needle path between the markers).
Claims 23, 25, and 32 are rejected under 35 U.S.C. 103 as being unpatentable over Shekhar et al. (PGPUB No. US 2014/0303491) in view of DiMaio et al. (PGPUB No. US 2009/0036902) further in view of Razzaque et al. (PGPUB No. US 2010/0268067).
Regarding claim 23, modified Shekhar teaches the image fusion workstation in claim 21, as discussed above.
However, the combination of Shekhar and DiMaio is silent regarding an image fusion workstation, wherein the image fusion controller and the display controller are further configured to; compute a second spatial transformation of the at least one ultrasound image, wherein the second spatial transformation is of the spatial transformation, and wherein the second spatial transformation is onto an image space of an instrument inserted into the anatomical region: and control a display of the second spatial transformation at the instrument inserted into the anatomical region.
In an analogous imaging field of endeavor, regarding ultrasound imaging and operative treatment, Razzaque teaches an image fusion workstation, wherein the image fusion controller and the display controller are further configured to; compute a second spatial transformation of the at least one ultrasound image, wherein the second spatial transformation is of the spatial transformation, and wherein the second spatial transformation is onto an image space of an instrument inserted into the anatomical region: and control a display of the second spatial transformation at the instrument inserted into the anatomical region (Paragraphs 0133-39 teaches that the body transformations can be performed with consideration of the ultrasound transducer’s position and orientation and the needle’s position and orientation. Paragraphs 0075-86 teaches the integration of the needle onto the ultrasound slice and defining the spatial relationship to each other. See Figs. 7-19).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the combination of Shekhar and DiMaio with Razzaque’s teaching of spatial transformation of the tool with respect to the ultrasound. This modified apparatus would allow the user to improve patient care and guide procedures (Paragraph 0038 of Razzaque). Furthermore, the modification proves improved perception and needle guidance to the physician (Paragraph 0089 of Razzaque).
Regarding claim 25, modified Shekhar teaches the image fusion workstation in claim 21, as discussed above.
However, the combination of Shekhar and DiMaio is silent regarding an image fusion workstation, wherein the image fusion controller and the display controller are further configured to: compute a second spatial transformation of the at least one ultrasound image, wherein the second spatial transformation is of the spatial transformation, and wherein the second spatial transformation is onto an image space of an instrument inserted into the anatomical region: and generate for display the second spatial transformation.
In an analogous imaging field of endeavor, regarding ultrasound imaging and operative treatment, Razzaque teaches an image fusion workstation, wherein the image fusion controller and the display controller are further configured to: compute a second spatial transformation of the at least one ultrasound image, wherein the second spatial transformation is of the spatial transformation, and wherein the second spatial transformation is onto an image space of an instrument inserted into the anatomical region: and generate for display the second spatial transformation (Paragraphs 0133-39 teaches that the body transformations can be performed with consideration of the ultrasound transducer’s position and orientation and the needle’s position and orientation. Paragraphs 0075-86 teaches the integration of the needle onto the ultrasound slice and defining the spatial relationship to each other. See Figs. 7-19).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the combination of Shekhar and DiMaio with Razzaque’s teaching of spatial transformation of the tool with respect to the ultrasound. This modified apparatus would allow the user to improve patient care and guide procedures (Paragraph 0038 of Razzaque). Furthermore, the modification proves improved perception and needle guidance to the physician (Paragraph 0089 of Razzaque).
Regarding claim 32, modified Shekhar teaches the image fusion workstation in claim 25, as discussed above.
However, Shekhar is silent regarding an image fusion workstation, wherein image fusion controller and the display controller are further configured to display at least one reference planar view of the anatomical model indicative of a status of a surgical procedure performed within the anatomical region.
In an analogous imaging field of endeavor, regarding image fusion, DiMaio teaches an image fusion workstation, wherein image fusion controller and the display controller are further configured to display at least one reference planar view of the anatomical model indicative of a status of a surgical procedure performed within the anatomical region (Paragraphs 0156 teaches that preoperative images from the MRI, CT, ultrasound, PET or other modalities is stored and displayed. The images are volumetric images via an overlay with the endoscopic image data. Paragraph 0170-72 teaches that slices for the volumetric image can be display and integrated. See Figs. 24-26. Paragraph 0081 teaches that the LUS and the endoscopic views allow for different structures of the anatomy to be observed with the tool and the tool can be observed and implemented with respect to a 3D model. Paragraphs 0053-54 teaches that the LapUS probe is tracked and the probe is calibrated and integrated with respect to the endoscopic system based on its position and registration. Paragraphs 0136-39 teaches that the LapUS is observed in the visual field of the endoscope’s field of view. This allows for the observation in three dimensions. The LapUS is shown in the flashlight image that shows the ultrasound image that is electronically attached to the surgical instrument image and this is performed in real time. See Fig. 21. Paragraphs 0145-46 teaches that the graphical tool icons can be used to indicate the respective instrument in the view of the endoscope. See Fig. 22. Paragraphs 0118-19 teaches that the overlay is selectively done with the medical image volume onto the fused ultrasound and endoscopic images. This combined image is rendered for display. The LapUS is captured and adapted for the monitoring of the implementation of the tool. Paragraph 0115 teaches the manipulation of the instruments and the tool arms according to the medical image volume. Paragraph 0130 teaches that the medical volume data can be overlayed with the video and live images along with the LapUS feed).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Shekhar with DiMaio’s teaching of indicating the status of a surgical procedure with an anatomical model. This modified apparatus would allow the user to move the LUS to see different views of the anatomical structure and manipulate a tool via input and the generation of a 3D model (Paragraph 0081 of DiMaio). Furthermore, the modification allows for precise placement of surgical tools prior to actual cutting and shearing of tissue (Paragraph 0190 of DiMaio).
Claim 27 is rejected under 35 U.S.C. 103 as being unpatentable over Shekhar et al. (PGPUB No. US 2014/0303491) in view of DiMaio et al. (PGPUB No. US 2009/0036902) further in view of Jain et al. (PGPUB No. US 2014/0147027).
Regarding claim 27, modified Shekhar teaches the image fusion workstation in claim 26, as discussed above.
However, the combination of Shekhar and DiMaio is silent regarding an image fusion workstation, wherein the anatomical model is an anatomical atlas of an anatomical structure within the anatomical region.
In an analogous imaging field of endeavor, regarding ultrasound imaging and endoscopy in relation to anatomical models, Jain teaches an image fusion workstation, wherein the anatomical model is an anatomical atlas of an anatomical structure within the anatomical region (Paragraph 0034 teaches that the registration of current images can be to a patient model that is an atlas. Paragraphs 0035, 0043-44, and 49 teaches that the models are representative of values associated to the tissue).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the combination of Shekhar and DiMaio with Jain’s teaching of an atlas model. This modified apparatus would allow the user to optimize the phase aberration (Paragraph 0034 of Jain). Furthermore, the modification allows for the incorporation of live corrections of the distortions of ultrasound images (Paragraph 0035 of Jain).
Claim 33 is rejected under 35 U.S.C. 103 as being unpatentable over Shekhar et al. (PGPUB No. US 2014/0303491) in view of DiMaio et al. (PGPUB No. US 2009/0036902) further in view of Razzaque et al. (PGPUB No. US 2010/0268067) further in view of Chen et al. ("Surgical planning and manual image fusion based on 3D model facilitate laparoscopic partial nephrectomy for intrarenal tumors", 2014).
Regarding claim 33, modified Shekhar teaches the image fusion workstation in claim 25, as discussed above.
However, the combination of Shekhar, DiMaio, and Razzaque is silent regarding an image fusion workstation, wherein the image fusion controller and the display controller are further configured to provide a virtual display of at least one desired position of the laparoscopic ultrasound probe relative to the anatomical model based on the registered anatomical model of the anatomical region and fused image.
In an analogous imaging field of endeavor, regarding endoscopic image fusion, Chens teaches an image fusion workstation, wherein the image fusion controller and the display controller are further configured to provide a virtual display of at least one desired position of the laparoscopic ultrasound probe relative to the anatomical model based on the registered anatomical model of the anatomical region and fused image (Fig. 2 teaches that the 3D model is superimposed on the 2D laparoscope image. The surgical planning markers are placed on the kidney and are superimposed. Fig. 3 teaches the intraoperative procedure and it shows the guidance of the ultrasound image in relation to the stereotactic image and the model).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the combination of Shekhar, DiMaio, and Razzaque with Chen’s teaching of desired position for the probe based on the model. As Shekhar teaches the fusion of the laparoscopic and ultrasound imaging modalities and Chen also teaches the implementation of both modalities, one with ordinary skill in the art would be able to implement the fused image in the selection of the desired position. This modified apparatus would allow the user to perform treatment of intrarenal tumors (Abstract of Chen). Furthermore, the modification improves allows for image guided surgery developments and the modeling of renal tumors and surgical planning (Conclusion of Chen).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Kawashima et al. (PGPUB No. US 2007/0239009): Teaches integration of a reference image in an endoscopic procedure.
Nihei et al. (PGPUB No. US 2011/0144500): Teaches integration of a reference image in an endoscopic procedure.
Sauer et al. (PGPUB No. US 2006/0281971): Teaches integration of preoperative modeling into an endoscopic procedure.
Kawashima et al. (PGPUB No. US 2005/0256402): Teaches integration of a reference image in an endoscopic procedure.
Uchikubo (PGPUB No. US 2004/0024288): Teaches detection of LUS in an endoscopic field of view.
Ikuma et al. (PGPUB No. US 2009/0036775): Teaches integration of a reference image in an endoscopic procedure.
De Almeida et al. (US Patent No. 10,499,996): Teaches spatial transformation for integration of the LUS view into the endoscopic view for surgical operations.
Yang et al. ("Ultrasound Image-guided Mapping of Endoscopic Views on a 3D Placenta Model: A Tracker-less Approach", 2013): Teaches spatial transformation for integration of the LUS view into the endoscopic view for surgical operations.
Buchs et al. ("Augmented environments for the targeting of hepatic lesions during image-guided robotic liver surgery", 2013): Teaches spatial transformation for integration of the LUS view into the endoscopic view for surgical operations.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ADIL PARTAP S VIRK whose telephone number is (571)272-8569. The examiner can normally be reached Mon-Fri 8-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pascal Bui-Pho can be reached on 571-272-2714. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ADIL PARTAP S VIRK/Primary Examiner, Art Unit 3798
