Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAIL OFFICE ACTIONS
The United States Patent & Trademark Office appreciates the response filed for the current application that is submitted on 06/24/2026. The United States Patent & Trademark Office reviewed the following documents submitted and has made the following comments below.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. DE 10 2021 125 412.4, filed on 09/30/2021.
Information Disclosure Statement
The IDS dated 03/28/2024 has been considered and placed in the application file.
Amendment
Applicant submitted amendments on 06/24/2026. The Examiner acknowledges the amendment and has reviewed the claims accordingly.
Status of Claims
Claims 1-24 and 27 are pending.
Claims 25-26 are cancelled.
Applicant Arguments:
Regarding Argument 1, see remarks (page 10), filed 06/24/2026, Applicant/s state/s “The Examiner objected to the specification and claims 23 and 24 due to various informalities (Office Action pp 2-3) that Applicant has corrected as described in the amendments above. Accordingly, withdrawal of objections to the specification and claims is respectfully requested.”
Regarding Argument 2, see remarks (page 10), filed 06/24/2026, Applicant/s state/s “the claims are no longer drafted as means-plus-function limitations, and the § 112(f) interpretation is no longer warranted. Withdrawal of the § 112(f) interpretation is respectfully requested.”
Regarding Argument 3, see remarks (pages 10-13), filed 06/24/2026, Applicant/s state/s “Claims 1 to 24 and 27 were rejected under 35 U.S.C. § 101 for allegedly being directed to an abstract idea, specifically, "mental process" of selecting an intra-op image and determining a matching image having a matching field of view, without significantly more (Office Action pp 6- 9). … For at least the foregoing reasons, Applicant respectfully submits that amended Claims 1 to 24 and 27 are directed to patent-eligible subject matter under 35 U.S.C. § 101. Withdrawal of the § 101 rejection of all pending claims is respectfully requested.”
Regarding Argument 4, see remarks (page 16), filed 06/24/2026, Applicant/s state/s “Because the combination of Piron and Liu does not teach or suggest the cross-section- construction limitation now recited in independent claims 1 and 16, the § 103 rejection is overcome. Accordingly, withdrawal of the § 103 rejection to these claims is respectfully requested.”
Regarding Argument 5, see remarks (page 18), filed 06/24/2026, Applicant/s state/s “Because neither Liu nor Piron, alone or in combination, teaches or suggests the cross-section- construction limitation now recited in claim 13, the § 103 rejection of claim 13 is overcome. Accordingly, withdrawal of the § 103 rejection of claim 13 is respectfully requested.”
Regarding Argument 6, see remarks (page 19), filed 06/24/2026, Applicant/s state/s “All remaining claims depend, directly or indirectly, from claims 1, 13, and 16. These claims are rejected only over Piron, or over Piron in combination with Siewerdsen, Liu, or Ben-Yishai, or over Liu in combination with Piron, and none has been rejected on independent grounds…none supplies the cross-section-construction limitation absent from Piron and Liu…Applicant respectfully submits that the cited art does not anticipate, or render, obvious, any of the pending dependent claims. Therefore, withdrawal of all § 102 and § 103 rejections is respectfully requested.”
Examiner’s Responses:
In response to Argument 1, see remarks (page 10), filed 06/24/2026, with respect to the objections to the specification and objections to claims 23 and 24 have been fully considered and are persuasive. The objections to the specification and objections to claims 23 and 24 have been withdrawn.
In response to Argument 2, see remarks (page 10), filed 06/24/2026, with respect to the 112(f) interpretation of claims 1 to 3, 6, 8, 10 to 12, and 13 to 15 has been fully considered and is persuasive. The 112(f) interpretation of claims 1 to 3, 6, 8, 10 to 12, and 13 to 15 has been withdrawn.
In response to Argument 3, see remarks (pages 10-13), filed 06/24/2026, with respect to the 35 U.S.C. § 101 rejection for Claims 1 to 24 and 27 have been fully considered and are persuasive, therefore the 35 U.S.C. § 101 rejection is withdrawn.
In response to Argument 4, see remarks (page 16), filed 06/24/2026 with respect to the 35 U.S.C. § 103 rejection (Piron and Liu) to claims 1 and 16 have been fully considered but they are not persuasive for the following reasons. Claims 1 and 16 were rejected under 35 U.S.C. § 102 in the Office Action (pages 10-12 and 17-18). Although the contents of prior claim 8 has been integrated into independent claims 1 and 16, additional limitations have been added to amended claims 1 and 16 such as, “different from the intra-op image” and “is a two-dimensional, cross-sectional image” that did not appear in original claims 1 or 8. Neither of original claims 1 or 8 recited that the matching image was constructed from three-dimensional image data different from the intra-op image or that the matching image is specifically two-dimensional. However, in view of the amendments to the claims which added limitations, a new ground(s) of rejection is made for Claims 1 and 16 under 35 U.S.C. § 103 in view of Piron (U.S. Patent Pub. No. 20170245761 A1, hereafter referred to as Piron) in view of Themelis (European Patent Application No. EP 3527123 A1, hereafter referred to as Themelis).
The Examiner finds that Piron in view of Themelis discloses the amended claim language “to provide images for comparison, the image processing device comprising a memory and one or more processors”. Specifically, Piron teaches a computing device which compares the intraoperative image with the guide image to identify at least one region of the guide image matching the intraoperative image (Paragraph [0005]). Piron also teaches that the computing device has a memory and a processor (Paragraph [0005]). In addition, the Examiner finds Themelis teaches the amended claim language “constructing a matching image from three-dimensional image data different from the intra-op image”. Specifically, Themelis teaches elastically matching the pre-operative three-dimensional image data to the interoperative data. As a result of the elastic matching, the pre-operative three-dimensional image data are shifted, rotated, and/or distorted so that the identified vascular plexus structure in both data (interoperative and preoperative) coincides geometrically. The elastic matching may also define a section through the pre-operative 3D image data which results in the field of view represented in the interoperative image data (Paragraph [0041]). The Examiner interprets the matching image (Themelis’ defined section with same field of view as interoperative image or the transformed pre-operative image data (shifted, rotated and/or distorted pre-op image data with coinciding plexus structure)) is constructed (shifted, rotated and/or distorted) from three-dimensional image data (Themelis’ “pre-operative three-dimensional image data”) which is different from the intraoperative data (Themelis’ pre-operative image data is not interoperative image data). Further, the Examiner finds that Themelis teaches the amended claim language “wherein the matching image is constructed by determining, based on a plurality of stored parameters associated with the intra-op image, a cross-section plane of the three-dimensional image data”. Specifically, Themelis teaches the mapping of the pre-operative three-dimensional data to the inter-operative data may be facilitated, in addition to the mere image data, using information about the position of the inter-operative image data. For this, a position sensor may be provided for generating position data representative of the position of a field of view of the camera assembly. The position data may comprise at least one of a focal length, field depth, and distance setting of the camera assembly. The positioning data may be used to elastically match the pre-operative three-dimensional image data to the inter-operative image data (Paragraph [0026]). Further, Themelis teaches the elastic matching may define a section through the pre-operative three-dimensional image data which results in the field of view represented in the interoperative image data. Also, see Fig. 1 (right corner), see the “section” (cross-section) identified. The “section” represents a cross-section plane of the brain represented by the pre-operative 3D image data. Lastly, the Examiner finds Themelis in view of Piron teaches the amended claim language “is a two-dimensional cross-sectional image”. Themelis teaches defining a section through pre-operative 3D image data which results in the field of view represented in the interoperative image. See Fig. 1 (bottom right corner), depicts a “cross-section” image. The Examiner will maintain prior art Piron, and details of the rejection are below.
In response to Argument 5, see remarks (page 18), filed 06/24/2026 with respect to the 35 U.S.C. § 103 rejection (Liu and Piron) to claim 13 has been fully considered and is persuasive. The 35 U.S.C. § 103 rejection (Liu in view of Piron) of claim 13 has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Themelis (European Patent Application No. EP 3527123 A1, hereafter referred to as Themelis) in view of Piron (U.S. Patent Pub. No. 20170245761 A1, hereafter referred to as Piron) in further view of Piron et al. (U.S. Patent Pub. No. 20170249737 A1, hereafter referred to as Piron 2).
Specifically, the Examiner finds that Themelis teaches the claim language “a surgical imaging device comprising one or more processors, one or more detectors, and a memory”. Themelis teaches a medical observation device which includes a medical processor assembly (one or more processors), a camera assembly (one or more detectors), and a memory assembly (a memory) (Paragraph [0038], Fig. 1). Further, Themelis teaches “the one or more processors configured for: capturing an intra-op image with the one or more detectors”. Themelis teaches the image processor assembly may be adapted to acquire interoperative image data from the camera assembly (Paragraphs [0032], [0038]). In addition, Themelis teaches the claim language, “determining a plurality of acquisition parameters at which the intra-op image was captured”, Themelis teaches collecting position data representative of the position of the field of view of the camera assembly for the intraoperative image data. Position data may include one or focal length, field depth, and distance setting of the camera assembly (Paragraph [0026]). The Examiner interprets focal length, field depth, and distance setting to each be examples of acquisition parameters at which the interoperative image data was captured. Themelis teaches “constructing a matching image from three-dimensional image data different from the intra-op image”, Themelis discloses an elastic-matching module for elastically matching the pre-op 3D image data to the inter-op image data based on at least one identified vascular plexus structure. The pre-op image data may be shifted, rotated, and/or distorted so that the identified vascular plexus structure in both data coincides geometrically (Paragraph [0041]). The Examiner interprets applying transformations to the pre-op 3D image data such as rotations, shifts, and distortions is “constructing” the matching image. Further, the three-dimensional pre-op image data is “different from” the intra-op image since it is captured prior to surgery rather than during surgery. Themelis also teaches “wherein the matching image is constructed by determining, based on the plurality of acquisition parameters, a cross-section plane of the three-dimensional image data”. Specifically, Themelis discloses the mapping of the pre-op 3D image data to the intra-op image data may be facilitated, in addition to the mere image data, using additional information about the position of the inter-op data. Position data may comprise at least one of focal length, field depth and distance setting of the camera assembly (Paragraph [0026]). Further, Themelis teaches the elastic matching may define a section through the pre-operative 3D image data (Paragraph [0041], see Fig. 1, bottom right corner, “cross-section”). Finally, Themelis teaches the limitations “wherein the matching image is a ” and “storing the intra-op image … in the memory.” Themelis teaches defining the section through the pre-op 3D image data which results in the field of view represented in the inter-op image data (Paragraph [0041]). Themelis also teaches storage where the inter-op image data are stored (Paragraph [0038]).
Themelis does not explicitly disclose wherein the matching image is a two-dimensional, two-dimensional and can be an image captured preoperatively. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to determine the two-dimensional cross-section that matches the intraoperative image data to avoid the need for the surgeon to manually align the pre-operative three-dimensional data with the surgeon’s actual view (2D view). This consumes time and time is critical for all surgery. Further, the surgeon may become distracted during surgery (Themelis, Paragraph [0006]). Further, since the intraoperative image data (actual view) may be two-dimensional, representing a plane in the field of view (Themelis, Paragraph [0032]), the matching pre-operative image section may also be two-dimensional. Finally, Themelis in view of Piron does not explicitly disclose “storing … the matching image in the memory”. Piron 2 is in the same field of art of performing surgical image registration to align two images with each other such that both are placed in a common coordinate system. Further, Piron 2 teaches storing the preoperative image and the preoperative measurements. A transformation for registering the preoperative image and the intraoperative image is stored in association with one of the intraoperative image and the preoperative image (Abstract, Paragraph [0050], Figs. 7A & 7B). Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to store the matching image alongside the intraoperative image in memory for retrieval by the surgeon during surgery to facilitate simultaneous visualization and direct comparison of images obtained preoperatively, alongside images obtained intraoperatively (Paragraph [0072]). The Examiner will maintain prior art Piron, and details of the rejection are below.
In response to Argument 6, see remarks (page 19), filed 06/24/2026, with respect to the dependent claims rejected under 35 U.S.C. 102 and 35 U.S.C 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn.
However, upon further consideration, a new ground(s) of rejection is made under 35 U.S.C. 103 for dependent claims 2, 4-6, 9-10, 20-21 and 23 in view of Piron (U.S. Patent Pub. No. 20170245761 A1, hereafter referred to as Piron) in view of Themelis (European Patent Application No. EP 3527123 A1, hereafter referred to as Themelis).
A new ground(s) of rejection is made under 35 U.S.C. 103 for dependent claims 3 and 22 in view of Piron (U.S. Patent Pub. No. 20170245761 A1, hereafter referred to as Piron) in view of Themelis (European Patent Application No. EP 3527123 A1, hereafter referred to as Themelis) in further view of Siewerdsen et al. (U.S. Patent Pub. No. 20140334709 A1, hereafter referred to as Siewerdsen).
A new ground(s) of rejection is made under 35 U.S.C. 103 for dependent claims 7 and 12 in view of Piron (U.S. Patent Pub. No. 20170245761 A1, hereafter referred to as Piron) in view of Themelis (European Patent Application No. EP 3527123 A1, hereafter referred to as Themelis) in further view of Liu (U.S. Patent Pub. No. 20170202633 A1, hereafter referred to as Liu).
A new ground(s) of rejection is made under 35 U.S.C. 103 for dependent claims 11, 17-19 and 24 in view of Piron (U.S. Patent Pub. No. 20170245761 A1, hereafter referred to as Piron) in view of Themelis (European Patent Application No. EP 3527123 A1, hereafter referred to as Themelis) in further view of Ben-Yishai (U.S. Patent Pub. No. 20240180629 A1, hereafter referred to as Ben-Yishai).
A new ground(s) of rejection is made under 35 U.S.C. 103 for dependent claims 14-15 in view of Themelis (European Patent Application No. EP 3527123 A1, hereafter referred to as Themelis) in view of Piron (U.S. Patent Pub. No. 20170245761 A1, hereafter referred to as Piron) in further view of Piron et al. (U.S. Patent Pub. No. 20170358083 A1, hereafter referred to as Piron 2).
Lastly, a new ground(s) of rejection is made under 35 U.S.C. 103 for dependent claim 8 in view of Piron (U.S. Patent Pub. No. 20170245761 A1, hereafter referred to as Piron) in view of Themelis (European Patent Application No. EP 3527123 A1, hereafter referred to as Themelis) in further view of Wosnitzer et al. (NPL “The truncation artifact”, 2011, hereafter referred to as Wosnitzer).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-2, 4-6, 9-10, 16, 20-21, 23, and 27 are rejected under 35 U.S.C. 103 as being unpatentable over in view of Piron (U.S. Patent Pub. No. 20170245761 A1, hereafter referred to as Piron) in view of Themelis (European Patent Application No. EP 3527123 A1, hereafter referred to as Themelis).
Regarding Claim 1, Piron teaches an image processing device to provide images for comparison (Paragraph [0005], Piron teaches a computing device for adaptive image acquisition and compares the intraoperative image with the guide image to identify at least one region of the guide image matching the intraoperative image.), the image processing device comprising a memory (Paragraph [0005], Piron teaches the computing device comprises a memory.) and one or more processors (Paragraph [0005], Piron teaches the computing device comprises a processor.) configured for: selecting an intra-op image which is stored in the memory (Abstract, Paragraphs [0004], [0005], Piron teaches receiving an intraoperative image of a portion of a patient tissue from an imaging instrument and storing the intraoperative image in a memory of a computing device.), wherein the intra-op image has a field of view (Figs. 6 & 7, Piron teaches intraoperative images (600) and (700), each having a field of view.); (Paragraph [0058], Figs. 6 & 7, Piron teaches enlarging regions 704, 708, and 712 (of guide image) in comparison with regions 604, 608, and 612 due to the greater size of composite image 702 relative to intraoperative image 600. Region 704 remains an accurate match with image (702), and thus determination of a match is affirmative (exceeds specified confidence criteria) from three-dimensional image data different from the intra-op image (Paragraph [0042], Piron teaches the guide image may be three-dimensional and can be an image captured preoperatively.), (Paragraphs [0058], [0042], Fig. 7, Piron teaches enlarged region 704 of guide image remains an accurate match with composite image 702. The guide image may be two-dimensional. Further, see Fig. 7, composite intraoperative image (702) and region of two-dimensional guide image (704) have matching fields of view.).
Although Piron discloses matching regions of guide images from three-dimensional image data different from the intra-op image (the guide image may be three-dimensional and pre-operative) and the matching image is a two dimensional image that matches the field of view of the intra-op image, Piron does not explicitly disclose constructing a matching image from three-dimensional image data different from the intra-op image, wherein the matching image is constructed by determining, based on a plurality of stored parameters associated with the intra-op image, a cross-section plane of the three-dimensional image data, wherein the matching image is a two-dimensional, cross-sectional image that matches the field of view of the intra-op image
Themelis is in the same field of art of matching images collected pre-operatively with intraoperative images acquired during surgery to identify identical structures in each of the images. Further, Themelis teaches constructing a matching image (Paragraph [0041], Themelis teaches an elastic-matching module for elastically matching the pre-operative three-dimensional image data (6) to the interoperative image data (14) based on the at least one identified vascular plexus structure (48). As a result of the elastic matching, the pre-operative three-dimensional image data (6) are shifted, rotated, and/or distorted so that the identified vascular plexus structure (48) in both data coincides geometrically.) from three-dimensional image data (Paragraph [0008], Themelis teaches providing pre-operative three-dimensional image data of soft biological tissue.) different from the intra-op image (Paragraph [0009], Themelis teaches matching the pre-operative three-dimensional image data to the intraoperative data. The Examiner interprets the “pre-operative three-dimensional image data” is “different from” the intra/inter-operative image data.) wherein the matching image is constructed (Paragraphs [0009], [0041], Themelis teaches a matching module for elastically matching the pre-operative three-dimensional image data to the intraoperative image data. As a result of the elastic matching, the pre-operative three-dimensional image data (6) are shifted, rotated and/or distorted so that the identified vascular plexus structure (48) in both data coincides geometrically. The step of elastic matching may also define a section through the pre-operative three-dimensional image data (6) which results in the field of view (10) represented in the intraoperative image data (14).) by determining, based on a plurality of stored parameters associated with the intra-op image (Paragraphs [0026], [0044-45], Themelis teaches the mapping of the pre-operative three-dimensional image data to the intraoperative image data may be facilitated if in addition to the mere image data, additional information of the intraoperative image data, is available. For this, a position sensor may be provided for generating position data representative of the position of a field of view of the camera assembly. The position data may comprise at least one focal length, field depth and distance setting of the camera assembly. The positioning data may be used to elastically match the pre-operative three-dimensional image data to the interoperative image data.), a cross-section plane of the three-dimensional image data (Paragraph [0041], Fig. 1, Themelis teaches the elastic matching may define a section through the pre-operative three-dimensional image data (6) which results in the field of view (10) represented in the interoperative image data (14).)
PNG
media_image1.png
292
346
media_image1.png
Greyscale
wherein the matching image is a cross-sectional image that matches the field of view of the intra-op image (Paragraph [0041], Fig. 1, Themelis teaches the step of elastic matching may also define a section through the pre-operative three-dimensional image data (6) which results in the field of view represented in the intraoperative image data (14). See Fig. 1, cross-section of the three-dimensional pre-operative image data (6). The cross-section “section” has the same field of view represented in the intraoperative data (14).).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Piron by “constructing” the matching image data by performing shifts, rotations, and/or distortions to the preoperative three-dimensional image data to generate the corresponding section (“cross-section”) of the three-dimensional pre-operative image data matching the interoperative image data field of view as well as using the collected position data associated with the intraoperative image in order to identify the corresponding matching section that is taught by Themelis, to make the invention that eliminates the need for the surgeon to align the pre-operative image data manually with the actual (inter-operative) view; thus, one of ordinary skilled in the art would be motivated to combine the references since automating the matching/aligning between pre-operative and interoperative image data during surgery can reduce time consumption and help the surgeon to avoid distractions during surgical procedures (Themelis, Paragraph [0006]). Further, the computational burden for locating and/or identifying the vascular plexus structure of the interoperative data in the pre-operative 3D image data may be high. The computational burden may be reduced if position information is provided as to where the field of view, the vascular plexus structure, and/or the interoperative image data are located within the soft biological tissue (Themelis, Paragraph [0044]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 2, Piron in view of Themelis teaches the image processing device of claim 1, wherein the intra-op image is one of a plurality of intra-op images stored in the memory (Paragraph [0005], Piron teaches storing an intraoperative image in memory and storing a “further” intraoperative image. The Examiner interprets storing an intraoperative image in memory and subsequently storing a “further” intraoperative image in memory means that at least two intraoperative images are stored in the memory. Under BRI, “a plurality” means two or more.).
In regards to Claim 4, Piron in view of Themelis teaches the image processing device of claim 1, wherein the intra-op image is an image captured by a surgical imaging apparatus (Paragraphs [0026-27], Piron teaches collecting images during the procedure, referred to as intraoperative images, using any of a variety of imaging modalities, such as Magnetic Resonance Imaging (MRI), Optical Coherence Tomography (OCT) ultrasound, Computer Tomography (CT), optical spectroscopy and the like.).
In regards to Claim 5, Piron in view of Themelis teaches the image processing device of claim 4, wherein the surgical imaging apparatus is a microscope, infrared microscope, optical coherence tomography device, photoacoustic imaging device, ultrasound imaging device, magnetic resonance imaging device, or computed tomography device (Paragraphs [0026-27], Piron teaches collecting images during the procedure, referred to as intraoperative images using any of a variety of imaging modalities, such as Magnetic Resonance Imaging (MRI), Optical Coherence Tomography (OCT) ultrasound, Computer Tomography (CT), optical spectroscopy and the like. Under BRI, the Examiner interprets “or” to indicate only one of the types of surgical imaging apparatus is required.).
In regards to Claim 6, Piron in view of Themelis teaches the image processing device of claim 1, wherein the one or more processors (Paragraph [0005], Piron teaches a processor.) are further configured for: selecting a second intra-op image which is stored in the memory (Paragraph [0057], Fig. 7, Piron teaches a computing device configured to receive and store a further intraoperative image. Fig. 7 depicts intraoperative image (600) and a further intraoperative image (700).), wherein a field of view of the second intra-op image matches the field of view of the intra-op image (Fig. 7, Piron teaches a first (600) and second (700) intraoperative image. Fig. 7 shows that both the first intraoperative image and second intraoperative image have the same field of view. The Examiner interprets that the two intraoperative images, (600) and (700), have the same field of view since they both show the same angular or spatial area that can be captured at a particular moment and the claim is silent to the definition of field of view.), and wherein the second intra-op image is automatically selected when the intra-op image is selected (Paragraphs [0058-60], Fig. 3, Piron teaches after an affirmative determination is achieved (a region in the guide image exceeds required confidence level to match with intraoperative image) and computing device renders the guide images with an indication of the matching region, the computing device can return to block 310 to receive further intraoperative images. The method can be performed to continuously track the locations of within the patient as depicted in the guide image from which the intraoperative images are being captured. The Examiner interprets the continuous tracking of locations in the guide images by receiving and storing new intraoperative images and performing method 300 to mean that the second or next intraoperative image is selected automatically once the matching intraoperative image is selected.).
In regards to Claim 9, Piron in view of Themelis teaches the image processing device of claim 1, wherein the plurality of stored parameters includes at least one of: a user input, a position of a detector, an orientation of the detector, a plurality of reference positions, a magnification, a focal length, or a working distance (Paragraph [0026], Themelis discloses the position data may comprise at least one of focal length, field depth and distance setting of the camera assembly. Under BRI, the Examiner interprets “at least one of” and “or” to mean only one of the parameters are required to meet the claim limitation.).
In regards to Claim 10, Piron in view of Themelis teaches the image processing device of claim 1, wherein determining the cross-section plane comprises applying an algorithm that includes edge recognition (Paragraph [0050], Piron teaches a computing device configured to identify features, such as edges, lines, points, a histogram of pixel intensities, etc. of the intraoperative image and to identify regions of the guide image depicting the same area or volume of tissue as the intraoperative image that have similar features.).
In regards to Claim 16, Piron teaches a method of image processing to provide images for comparison (Abstract, Piron teaches a method for comparing the intraoperative image with the guide image to identify at least one region of the guide image matching the intraoperative image.), comprising selecting an intra-op image which is stored in a memory (Abstract, Paragraphs [0004], [0005], Piron teaches receiving an intraoperative image of a portion of a patient tissue from an imaging instrument and storing the intraoperative image in a memory of a computing device.), wherein the intra-op image has a field of view (Figs. 6 & 7, Piron teaches intraoperative images (600) and (700), each having a field of view.); (Paragraph [0058], Figs. 6 & 7, Piron teaches enlarging regions 704, 708, and 712 (of guide image) in comparison with regions 604, 608, and 612 due to the greater size of composite image 702 relative to intraoperative image 600. Region 704 remains an accurate match with image (702), and thus determination of a match is affirmative (exceeds specified confidence criteria).) from three- dimensional image data different from the intra-op image (Paragraph [0042], Piron teaches the guide image may be three-dimensional and can be an image captured preoperatively.), (Paragraphs [0058], [0042], Fig. 7, Piron teaches enlarged region 704 of guide image remains an accurate match with composite image 702. The guide image may be two-dimensional. Further, see Fig. 7, composite intraoperative image (702) and region of two-dimensional guide image (704) have matching fields of view.).
Although Piron discloses matching regions of guide images from three-dimensional image data different from the intra-op image (the guide image may be three-dimensional and pre-operative) and the matching image is a two dimensional image that matches the field of view of the intra-op image,
Piron does not explicitly disclose constructing, with one or more processors, a matching image from three-dimensional image data different from the intra-op image, wherein the matching image is constructed by determining, based on a plurality of stored parameters associated with the intra-op image, a cross-section plane of the three-dimensional image data, wherein the matching image is a two-dimensional, cross-sectional image that matches the field of view of the intra-op image.
Themelis is in the same field of art of matching images collected pre-operatively with intraoperative images acquired during surgery to identify identical structures in each of the images. Further, Themelis teaches constructing, with one or more processors, a matching image from three-dimensional image data different from the intra-op image (Paragraph [0041], Themelis teaches the image processor assembly (40) may further comprise an elastic-matching module for elastically matching the pre-operative three-dimensional image data (6) to the interoperative image data (14) based on the at least one identified vascular plexus structure (48). As a result of the elastic matching, the pre-operative three-dimensional image data (6) are shifted, rotated, and/or distorted so that the identified vascular plexus structure (48) in both data coincides geometrically.), wherein the matching image is constructed by determining, based on a plurality of stored parameters associated with the intra-op image (Paragraphs [0026], [0044-45], Themelis teaches the mapping of the pre-operative three-dimensional image data to the intraoperative data may be facilitated, in addition to the mere image data, additional information about the position of the interoperative image data. For this, a position sensor may be provided for generating position data representative of the position of a field of view of the camera assembly. The position data may comprise at least one of focal length, field depth and distance setting of the camera assembly. The positioning data may be used for elastically matching the pre-operative three-dimensional image data to the interoperative image date.), a cross-section plane of the three-dimensional image data (Paragraph [0041], Fig. 1, Themelis teaches the elastic matching may define a section through the pre-operative three-dimensional image data (6) which results in the field of view (10) represented in the interoperative image data (14). See Fig. 1, bottom right corner, cross-section through pre-operative 3D image data.), wherein the matching image is a cross-sectional image that matches the field of view of the intra-op image (Paragraph [0041], Fig. 1, Themelis teaches the step of elastic matching may also define a section through the pre-operative three-dimensional image data (6) which results in the field of view (10) represented in the intraoperative image data (14). See Fig. 1, cross section.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Piron by “constructing” the matching image data by performing shifts, rotations, and/or distortions to the image data to generate the corresponding cross-section of the three-dimensional pre-operative image data matching the interoperative image data field of view as well as using the collected position data associated with capturing the intraoperative image in order to identify the corresponding matching section that is taught by Themelis, to make the invention that eliminates the need for the surgeon to align the pre-operative image data manually with the actual (inter-operative) view; thus, one of ordinary skilled in the art would be motivated to combine the references since automating the matching/aligning between pre-operative and interoperative image data during surgery can reduce time consumption and help the surgeon to avoid distractions during surgical procedures (Themelis, Paragraph [0006]). Further, the computational burden for locating and/or identifying the vascular plexus structure of the interoperative data in the pre-operative 3D image data may be high. The computational burden may be reduced if position information is provided as to where the field of view, the vascular plexus structure, and/or the interoperative image data are located within the soft biological tissue (Themelis, Paragraph [0044]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 20, Piron in view of Themelis discloses the method of claim 16, wherein determining the cross-section plane includes recognizing edge features (Paragraph [0050], Piron teaches a computing device configured to identify features, such as edges, lines, points, a histogram of pixel intensities, etc. of the intraoperative image and to identify regions of the guide image depicting the same area or volume of tissue as the intraoperative image that have similar features.).
In regards to Claim 21, Piron in view of Themelis discloses the method of claim 16, wherein the matching image is constructed from pre-op, three-dimensional image data or post-op, three-dimensional image data (Paragraphs [0042], [0004], Piron teaches the guide image may be three-dimensional and can be captured preoperatively. The intraoperative image and guide image are compared to identify at least one region of the guide image matching the intraoperative image. Under BRI, the Examiner interprets “or” to mean only one of either the pre-op 3D image data or the post-op 3D image data is required to meet the claim limitation.).
In regards to Claim 23, Piron in view of Themelis discloses the method of claim 16, further comprising: automatically (Paragraph [0008], Themelis teaches automatically identifying at least one vascular plexus structure in the intraoperative image data and identifying at least one identified vascular plexus structure in the pre-operative three-dimensional image data.), with the one or more processors (Paragraph [0005], Piron teaches the processor receives an intraoperative image of a portion of the patient tissue from an imaging instrument.), selecting a second intra-op image (Paragraph [0057], Piron teaches receiving and storing a further intraoperative image (700). Intraoperative image 700 is combined with intraoperative image 600 to produce a composite image 702. The Examiner interprets the composite image as the “second intra-op image”.) when the intra-op image is selected (Paragraphs [0056-57], Piron teaches intraoperative image 600 contains too little information to accurately locate intraoperative image within the guide image. The method therefore returns to block 310 and a further intraoperative image is received and stored.), and determining a second matching image (Paragraph [0058], Piron teaches region 704 remains an accurate match with image 702, and thus the determination is affirmative because a single region identified exceeds a required level of confidence specified by the criteria.) based on: selecting the second matching image from the memory, or constructing the second matching image (Paragraph [0058], Fig. 7, Piron teaches enlarging regions (704, 708 and 712) due to the greater size of composite image (702) relative to intraoperative image (600). Region 704 of the guide image remains an accurate match with image (702), and thus the determination is affirmative (exceeds required confidence value). The Examiner interprets “enlarging” the regions of the guide image to match the intraoperative image to be “constructing” since it is performing a transformation on the image to match it.) from three-dimensional image data (Paragraph [0042], Piron teaches the guide image may be three-dimensional.); wherein the second matching image matches the field of view of the second intra-op image (Fig. 7, Piron teaches region 704 of the guide image remains an accurate match with image 702 (the intraoperative composite image). As shown in Fig. 7, region 704 and image 702 have a matching field of view and depict the same structure.).
In regards to Claim 27, Piron discloses a non-transitory, computer-readable medium comprising a program code that, when the program code is executed on a processor, a computer, or a programmable hardware component, causes the processor, computer, or programmable hardware component (Paragraphs [0037], [0031], Piron teaches a computing device which stores, in memory (a non-transitory computer readable medium), an adaptive image application (application) comprising a plurality of computer-readable instructions executable by a processor. When the processor executes the instructions of the application, the processor performs various functions implemented by those instructions.).
Piron does not explicitly disclose to perform the method of claim 16.
(Examiner states see rejection for Claim 16, method for combining references.)
Claim(s) 3 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over in view of Piron (U.S. Patent Pub. No. 20170245761 A1, hereafter referred to as Piron) in view of Themelis (European Patent Application No. EP 3527123 A1, hereafter referred to as Themelis) in further view of Siewerdsen et al. (U.S. Patent Pub. No. 20140334709 A1, hereafter referred to as Siewerdsen).
Regarding Claim 3, Piron in view of Themelis teaches the image processing device of claim 1, (Paragraph [0041], Themelis teaches elastically matching the pre-operative three-dimensional image data (6) to the intraoperative image data (14). Further, the step of elastic matching may also define a section through the pre-operative three-dimensional image data (6) which results in the field of view (10) represented in the intraoperative image data (14).)
Piron in view of Themelis does not explicitly disclose wherein the one or more processors are further configured for determining a second matching image, wherein the second matching image matches the field of view of the intra-op image, and the second matching image is a post-op image.
Siewerdsen is in the same field of art of obtaining matching surgical images (intra-, pre-, or post-operative images) for the purpose of surgical navigation and surgical guidance. Further, Siewerdsen teaches wherein the one or more processors (Paragraph [0036], Siewerdsen teaches a processor executing one of the methods.) are further configured for determining a second matching image (Paragraphs [0094], [0096], Siewerdsen teaches a 3D-2D registration method for detecting changes and determining a similarity metric between an intra-operative 3D image and a 2D image, such as a post-operative 2D radiograph. The 3D-2D registration method allows quantitative comparison and analysis of the pre- and post-operative images which can elucidate the changes occurring between them, such as intraoperatively introduced tools or objects that were unintentionally left in the body. The quantitative comparison may assist in scrutinizing the surgical result.), wherein the second matching image matches the field of view of the intra-op image (Paragraphs [0059], [0005], Siewerdsen teaches an optimizer which uses a similarity metric to compute the transformation yielding the best match between an acquired 2D image and a reconstructed projecting of the 3D data (image). The 3D-2D registration method allows structures to be geometrically registered to the mobile radiograph and overlaid in a manner that enhances visualization and verification of surgical delivery. The 3D-2D registration method can compute alignment between two or more coordinate systems such as reference frames of an intra-operative image.), and the second matching image is a post-op image (Paragraphs [0079], [0094], [0096], Siewerdsen teaches using the projecting imaging system for acquisition of a radiograph after closure of a surgical field (postoperatively). A post-operative image provides useful documentation and/or verification of a surgical result and can be used to help detect retained foreign bodies (RFBs) from the procedure. A simulated 2D projection of the registered 3D image with exactly the same geometry as the actual radiograph may be computed from the result of the 3D-2D registration.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Piron in view of Themelis by determining a second matching image, which is a post-operative image, having the same field of view as the intra-operative image that is taught by Siewerdsen, to make the invention that allows for quantitative comparison and analysis between pre-, intra-, and post-operative images; thus, one of ordinary skilled in the art would be motivated to combine the references to increase precision and accuracy of surgeries by bringing 3D pre-operative images and planning structures into the context of 2D images acquired both during an operation (Siewerdsen, Paragraph [0007]) and post-operatively to provide useful documentation and verification of the surgical result and to help detect any retained foreign bodies (Siewerdsen, Paragraph [0079]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 22, Piron in view of Themelis teaches the method of claim 16, further comprising: (Paragraph [0041], Themelis teaches elastically matching the pre-operative three-dimensional image data (6) to the intraoperative image data (14). Further, the step of elastic matching may also define a section through the pre-operative three-dimensional image data (6) which results in the field of view (10) represented in the intraoperative image data (14).).
Piron in view of Themelis does not explicitly disclose determining a second matching image which is a post-op image.
Siewerdsen is in the same field of art of obtaining matching surgical images (intra-, pre-, or post-operative images) for the purpose of surgical navigation and surgical guidance. Further, Siewerdsen teaches determining a second matching image which is a post-op image (Paragraphs [0079], [0096], [0111], Siewerdsen teaches acquisition of a radiograph after closure of a surgical field (post-operatively). Post-operative images provide useful documentation and/or verification of the surgical result and may be used to help detect retained foreign bodies (RFBs). The simulated 2D projection of the registered 3D image with exactly the same geometry as the actual radiograph may be computed from the result of the 3D-2D registration. It allows quantitative comparison/analysis of the pre- and post-operative images which can elucidate changes occurring between them, such as intra-operatively introduced tools or objects that were unintentionally retained in the body.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Piron in view of Themelis by determining a second matching image, which is a post-operative image, having the same field of view as the intra-operative image that is taught by Siewerdsen, to make the invention that allows for quantitative comparison and analysis between pre-, intra-, and post-operative images; thus, one of ordinary skilled in the art would be motivated to combine the references to increase precision and accuracy of surgeries by bringing 3D pre-operative images and planning structures into the context of 2D images acquired both during an operation (Siewerdsen, Paragraph [0007]) and post-operatively to provide useful documentation and verification of the surgical result and to help detect any retained foreign bodies (Siewerdsen, Paragraph [0079]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim(s) 7 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over in view of Piron (U.S. Patent Pub. No. 20170245761 A1, hereafter referred to as Piron) in view of Themelis (European Patent Application No. EP 3527123 A1, hereafter referred to as Themelis) in further view of Liu (U.S. Patent Pub. No. 20170202633 A1, hereafter referred to as Liu).
Regarding Claim 7, Piron in view of Themelis discloses the image processing device of claim 6, (Paragraph [0026], Themelis teaches additional information about the position of the intraoperative image data is available. A position sensor may be provided for generating position data representative of the position of a field of view of the camera assembly. The position data may comprise at least one of focal length, field depth, and distance setting of the camera assembly.), wherein the plurality of acquisition parameters corresponds to at least a portion of the plurality of stored parameters associated with the intra-op image (Paragraph [0026], Themelis teaches the mapping of the pre-operative three-dimensional image data to the intraoperative image data may be facilitated if in addition to the mere image data, additional information of the intraoperative image data, is available. For this, a position sensor may be provided for generating position data representative of the position of a field of view of the camera assembly. The position data may comprise at least one focal length, field depth and distance setting of the camera assembly. The positioning data may be used to elastically match the pre-operative three-dimensional image data to the interoperative image data.), and the first intra-op image is captured at the plurality of acquisition parameters (Paragraph [0026], Themelis teaches additional information about the position of the intraoperative image data, and thus the vascular plexus structure within the tissue, is available. For this, a position sensor may be provided for generating position data representative of the position of a field of view of the camera assembly. The position data may comprise at least one of focal length, field depth and distance setting of the camera assembly.).
Piron in view of Themelis does not explicitly disclose wherein the second intra-op image
Liu is in the same field of art of guiding surgical interventions by matching pre-operative and intra-operative images to account for issues such as tissue deformation that occur during surgery to provide real-time imaging updates. Further, Liu teaches wherein the second intra-op image are (is) (captured using a plurality of acquisition parameters) (Paragraphs [0059-61], Liu teaches that the detector includes a field of view that captures image data of the target of interest that is within the field of view. The detector may be used in conjunction with any suitable optical lens or optical assembly to provide any desired field of view, working distance, resolution, and zoom level. Additionally, the detector may comprise a stereoscopic detector, which includes multiple imaging sensors or cameras designated respectively as 120A and 120B in Figs. 1 and 3A, with each individual camera having multiple sensor elements. For example, the first sensor element provides for full-color imaging and the second sensor element provides selective or switchable florescence imaging.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Piron in view of Themelis by capturing a second intraoperative image at a field of view, working distance, resolution, and zoom level that is taught by Liu, to make the invention that captures intraoperative images at various acquisition settings; thus, one of ordinary skilled in the art would be motivated to combine the references since the detector can capture any desired statistic or moving image data from a target of interest, which may be any desired object, such as a wound that is being treated in a patient enabling the detector to capture the desired object within the field of view (Liu, Paragraph [0059]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 12, Piron in view of Themelis teach the image processing device of claim 1, (Claim 7, Themelis teaches acquiring positioning data from the optical lens system.), one or more of the plurality of stored parameters (Claim 7, Themelis teaches the positioning data being representative of at least one of the focal length of an optical lens system used for acquiring the intraoperative image data, the field depth of the optical lens system, the distance setting of the optical lens system, a size, a dimension and an orientation of a field of view, at the time of acquiring the interoperative image data.) used for constructing the matching image (Claim 7, Themelis teaches the positioning data are used in identifying the vascular plexus structure in the pre-operative 3D image data and elastically matching the pre-operative 3D image data to the intraoperative image data and displaying the output image data.).
Piron in view of Themelis does not explicitly disclose wherein the one or more processors are further configured for: bidirectionally communicating with an image-guided surgery system.
Liu is in the same field of art of guiding surgical interventions by correlating surgical imaging data. Further, Liu teaches wherein the one or more processors are further configured for: bidirectionally communicating with an image-guided surgery system (Paragraph [0119], Liu teaches each imaging and display system having a detector and a communication interface, which allows a plurality of systems (100A-X) to communicate various data with one another and/or with one or more remote computing devices. Each system (100) has the ability to be a sender of data and a recipient of data. The system (100) can send data to any type of display unit, including non-wearable display and wearable display units.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Piron in view of Themelis by adding a bi-directional communication interface for communicating with the plurality of connected systems or other remote computing devices that is taught by Liu, to make the invention that facilitates medical training, diagnosis, and treatment by transferring images or image data over the network to one or more other systems; thus, one of ordinary skilled in the art would be motivated to combine the references to enable recipients of the image data to analyze and provide medical guidance based on the transferred images.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over in view of Piron (U.S. Patent Pub. No. 20170245761 A1, hereafter referred to as Piron) in view of Themelis (European Patent Application No. EP 3527123 A1, hereafter referred to as Themelis) in further view of Wosnitzer et al. (NPL “The truncation artifact”, Sept. 2020, hereafter referred to as Wosnitzer.
Regarding Claim 8, Piron in view of Themelis discloses the image processing device of claim 1.
Piron in view of Themelis does not explicitly disclose excluding orientations that are from a perspective outside of an expected range of perspectives for the intra-op image, wherein the expected range of perspectives is based on an input related to a type of surgery to be performed.
Wosnitzer is in the same field of art of preventing truncation artifacts (cropping/excluding part of the imaging target), which occur when the subject/patient is at least partially outside of the field of view of the imager. Further, Wosnitzer discloses excluding orientations that are from a perspective outside of an expected range of perspectives for the intra-op image (Discussion, Wosnitzer teaches the truncation artifact occurs when the patient is large relative to the field of view of the camera. Such truncation can occur in the X-axis or the Y-axis. The Examiner interprets the orientations (field of view) excluded (truncated) if they are outside the camera’s field of view (expected range of perspectives).), wherein the expected range of perspectives is based on an input related to a type of surgery to be performed (Discussion, Case report, Wosnitzer teaches the equipment used can contribute to the truncation artifact. The size of the field of view of the camera is a key factor in determining whether a truncation artifact will be created. Using a camera with a larger field of view can minimize the occurrence of truncation artifacts. For the myocardial perfusion imaging performed on the patient, the patient was repositioned to ensure the body and heart are well within the field of view.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Piron in view of Themelis by preventing truncation artifacts in imaging during surgery that is taught by Wosnitzer, to make the invention that ensures the entirely of the patient’s body and/or the area of interest (specific organ, bone, etc.) is within the field of view of the imager; thus, one of ordinary skilled in the art would be motivated to combine the references since truncation artifacts can cause misinterpretations in medical images, potentially leading to improper diagnoses and/or procedures (Wosnitzer, Discussion).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim(s) 11, 17-19 and 24 are rejected under 35 U.S.C. 103 as being unpatentable over in view of Piron (U.S. Patent Pub. No. 20170245761 A1, hereafter referred to as Piron) in view of Themelis (European Patent Application No. EP 3527123 A1, hereafter referred to as Themelis) in further view of Ben-Yishai (U.S. Patent Pub. No. 20240180629 A1, hereafter referred to as Ben-Yishai).
Regarding Claim 11, Piron in view of Themelis discloses the image processing device of claim 1, wherein the one or more processors are further configured for: (Claim 7, Themelis teaches acquiring positioning data from the optical lens system.), one or more of the plurality of stored parameters (Claim 7, Themelis teaches the positioning data being representative of at least one of the focal length of an optical lens system used for acquiring the intraoperative image data, the field depth of the optical lens system, the distance setting of the optical lens system, a size, a dimension and an orientation of a field of view, at the time of acquiring the interoperative image data.) used to determine the cross-section plane (Claim 7, Paragraph [0041], Fig. 1, Themelis teaches the positioning data are used in identifying the vascular plexus structure in the pre-operative 3D image data and elastically matching the pre-operative 3D image data to the intraoperative image data and displaying the output image data. The elastic matching may define a section through the pre-operative three-dimensional image data (6) which results in the field of view (10) represented in the interoperative image data (14).).
Piron in view of Themelis does not explicitly disclose sending a ping or timestamp to an image-guided surgery system.
Ben-Yishai is in the same field of art of matching a preoperative image to an intraoperative image to determine identical features in an image and accommodate distortions between the images. Further, Ben-Yishai discloses sending a ping or timestamp to an image-guided surgery system (Paragraphs [0086-87], Ben-Yishai teaches once an alignment score threshold between an intraoperative and preoperative image is met, providing an auditory indicator such as a beeping sound having a frequency increasing with the alignment score.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Piron in view of Themelis by providing an auditory indication to the surgeon, such as a beeping sound or “ping” when an alignment score between two images is met that is taught by Ben-Yishai, to make the invention that generates an auditory signal when the matching image is determined; thus, one of ordinary skilled in the art would be motivated to combine the references to indicate to the surgeon how close the actual location and/or orientation of the surgical object or patient tissue is to the desired location and/or orientation of the surgical object or patient tissue (Ben-Yishai, Paragraph [0086]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 17, Piron in view of Themelis discloses the method of claim 16.
Piron in view of Themelis does not explicitly disclose wherein the intra-op image is selected by a user.
Ben-Yishai is in the same field of art of matching a preoperative image to an intraoperative image to determine identical features in an image and accommodate for distortions between images. Further, Ben-Yishai discloses wherein the intra-op image is selected by a user (Paragraph [0087], Ben-Yishai teaches automatically displaying a “picture-in-picture” (PIP) when a surgeon presses a footswitch. The surgeon can momentarily freeze a live image to see the desired indicator overlaid with the intraoperative snapshot image.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Piron in view of Themelis by allowing the surgeon to select a live intraoperative image by pressing a footswitch that is taught by Ben-Yishai, to make the invention that determines a matching image for an intraoperative image selected by the surgeon during a surgery; thus, one of ordinary skilled in the art would be motivated to combine the references to enable comparison between the live image snapshot and its matching/identical preoperative image to improve alignment and carefully determine a desired location for an object with respect to the surgery, for example, during an intraocular lens (IOL) placement surgery (Ben-Yishai, Paragraph [0003]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 18, Piron in view of Themelis in further view of Ben-Yishai discloses the method of claim 17, further comprising prior to selecting the intra-op image (Paragraph [0051], Piron teaches detecting vessels and generating one or more metrics describing the detected vessels. The metrics can be combined into dendrograms. One or more dendrograms can be constructed by computing device for guide image, and the registration process can include comparison of the dendrograms. The Examiner interprets since the metrics are generated prior to creating the dendrograms (used for registration/mapping) the parameters (metrics) are determined prior to selecting the intra-op image.): determining a plurality of parameters during a surgery (Paragraphs [0050-51], Piron teaches a computing device configured to identify features such as edges, lines, points, or histograms of pixel intensities of an intraoperative image. In addition to these image features used in registration between intraoperative image and guide image, the computing device can be configured to generate a network model of any vessels (e.g. neurons) depicted in the intraoperative image. For example, the computing device can be configured to detect such vessels and generate metrics describing the vessels such as vessel dimensions (length, diameter), number of branches connected to each vessel, and fractal dimension. These metrics can be computed into dendrograms, which may also be compared in the registration process.), capturing the intra-op image during the surgery (Paragraphs [0027], [0044], Piron teaches during the procedure, collecting intraoperative images, such as, for example, images of the patient’s brain. The intraoperative image is received from an imaging instrument, such as a scope.), and storing the plurality of parameters in association with the intra-op image (Paragraph [0051], Piron teaches generating a network model of any vessels (neurons) depicted in the intraoperative image. The computing device may detect the vessels and generate one or more metrics describing the detected vessels. These metrics can be used to register the intraoperative images to the guide images. The Examiner interprets “metrics” describing the vessels in the intraoperative tissue to be parameters since the claim is silent to the definition of parameters.).
In regards to Claim 19, Piron in view of Themelis in further view of Ben-Yishai discloses the method of claim 18, wherein the plurality of parameters include a position and an orientation for providing the field of view of the intra-op image (Paragraph [0058], Ben-Yishai teaches data associated with the first image that can be represented as an overlay that is displayed on the second image. The overlay data can be guidance information determined during the capture of the preoperative image. The guidance information (reference data) can include a desired location and/or orientation of a medical object within the first image. The guidance information can be a desired location for insertion, placement, and/or positioning of any kind of medical object with respect to a patient. For example, the guidance information could be xy coordinates of the center of a visual axis and an orientation, and it can be represented as a line centered and oriented per the data and is displayed as an overlay with the second image (intraoperative image).).
In regards to Claim 24, Piron in view of Themelis discloses the method of claim 23.
Piron in view of Themelis does not explicitly disclose further comprising: determining a label for the intra-op image by user input, and storing the label; and determining a second label for the second intra-op image by user input, and storing the second label.
Ben-Yishai is in the same field of art of matching a preoperative image to an intraoperative image to determine identical features in an image and accommodate for distortions between images. Further, Ben-Yishai discloses further comprising: determining a label for the intra-op image by user input (Paragraphs [0109], [0111], [0008], Fig. 13, Ben-Yishai teaches that a user (surgeon, surgeon’s assistant, or another user) can select multiple points in a first image. Next, a symbol can be overlaid on and/or around each point in the first image upon the selection, to indicate the point, the symbol including a circle and a unique label (e.g. a number 1-7) that identifies each point. A symbol can comprise an identifying label such as, for example, a number, a letter, or any combination thereof. The “first image” are selected from at least one of: a preoperative image and an intraoperative image of the patient.), and storing the label (Paragraphs [0060], [0062] Ben-Yishai teaches a software tool for overlaying or drawing on the preoperative image (or a first image). The software tool may store the overlay as textual data that is used during the procedure to generate an overlay on the intraoperative image.); and determining a second label for the second intra-op image by user input (Paragraphs [0101-102], Ben-Yishai teaches designating points on the second image by overlaying a symbol on the image, such as a circle, square, arrow, or any desired shape. The symbol designating the point in the second image can have the same color, shape, or label as the corresponding point in the first image. The user can verify that the designated points in the second image designate the same image elements as the points in the first image.), and storing the second label (Paragraphs [0060], [0062] Ben-Yishai teaches a software tool for overlaying or drawing on the preoperative image (or a first image). The software tool may store the overlay as textual data that is used during the procedure to generate an overlay on the intraoperative image.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Piron in view of Themelis by determining labels for intraoperative images and storing the labels/symbols associated with the intraoperative images that is taught by Ben-Yishai, to make the invention that displays guidance information overlaid on the image; thus, one of ordinary skilled in the art would be motivated to combine the references since the addition of symbols/labels may enable the surgeon to quickly identify corresponding points between the pre-operative and intraoperative images during surgery (Ben-Yishai, Paragraph [0109]). Further, displaying symbols and/or regions of interest on the images can facilitate verification (Ben-Yishai, Paragraph [0110]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim(s) 13-15 are rejected under 35 U.S.C. 103 as being unpatentable over Themelis (European Patent Application No. EP 3527123 A1, hereafter referred to as Themelis) in view of Piron (U.S. Patent Pub. No. 20170245761 A1, hereafter referred to as Piron) in further view of Piron et al. (U.S. Patent Pub. No. 20170249737 A1, hereafter referred to as Piron 2).
Regarding Claim 13, Themelis teaches a surgical imaging device (Paragraph [0038], Fig. 1, Themelis teaches a medical observation device (1).) comprising one or more processors (Paragraph [0038], Fig. 1, Themelis teaches the medical observation device further includes a medical processor assembly (40).), one or more detectors (Paragraph [0038], Fig. 1, Themelis teaches the image processor assembly is connected to the camera assembly (8) by a data connection.), and a memory (Paragraph [0038], Themelis teaches the memory assembly (4) may be part of the image processor (40).), the one or more processors configured for (Paragraph [0038], Themelis teaches the image processor assembly (40) may be adapted to acquire intraoperative image data (14) from at least one camera assembly (8).): capturing an intra-op image with the one or more detectors (Paragraph [0032], Themelis teaches during surgery, the camera assembly (8) acquires intraoperative image data (14).); determining a plurality of acquisition parameters at which the intra-op image was captured (Paragraph [0026], Themelis teaches additional information about the position of the intraoperative image data, and thus the vascular plexus structure within the tissue, is available. For this, a position sensor may be provided for generating position data representative of the position of a field of view of the camera assembly. The position data may comprise at least one of focal length, field depth and distance setting of the camera assembly.); constructing a matching image from three-dimensional image data different from the intra-op image (Paragraph [0041], Themelis teaches the image processor assembly (40) may further comprise an elastic-matching module for elastically matching the pre-operative three-dimensional image data (6) to the interoperative image data (14) based on the at least one identified vascular plexus structure (48). As a result of the elastic matching, the pre-operative three-dimensional image data (6) are shifted, rotated, and/or distorted so that the identified vascular plexus structure (48) in both data coincides geometrically.), wherein the matching image is constructed by determining, based on the plurality of acquisition parameters (Paragraph [0026], Themelis teaches the mapping of the pre-operative three-dimensional image data to the intraoperative data may be facilitated, in addition to the mere image data, additional information about the position of the interoperative image data. For this, a position sensor may be provided for generating position data representative of the position of a field of view of the camera assembly. The position data may comprise at least one of focal length, field depth and distance setting of the camera assembly. The positioning data may be used for elastically matching the pre-operative three-dimensional image data to the interoperative image data.), a cross-section plane of the three-dimensional image data (Paragraph [0041], Fig. 1, Themelis teaches the elastic matching may define a section through the pre-operative three-dimensional image data (6) which results in the field of view (10) represented in the interoperative image data (14).), wherein the matching image is a (Paragraph [0041], Fig. 1, Themelis teaches the step of elastic matching may also define a section through the pre-operative three-dimensional image data (6) which results in the field of view (10) represented in the intraoperative image data (14).); storing the intra-op image (Paragraph [0038], Themelis teaches a storage where the intraoperative image data are stored.)
Although Themelis discloses a section (cross-section, see Fig. 1, cross section) through the pre-operative three-dimensional image data which results in a field of view represented in the intraoperative image data.
Themelis does not explicitly disclose a two-dimensional,
Piron is in the same field of art of performing registration between preoperative and intraoperative images to improve tracking inaccuracies due to tissue deformation and other errors such as patient movement, shifting in tracking equipment, etc. Further, Piron teaches a two-dimensional, (Paragraphs [0042], [0005], [0013], [0058], Fig. 7, Piron teaches the guide image may be two-dimensional and can be an image captured preoperatively. The intraoperative image is compared with the guide image to identify at least one region of the guide image matching the intraoperative image. Fig. 7 depicts the intraoperative image of Fig. 6 in combination with a further intraoperative image, and example matching regions. As shown in Fig. 7, region 704 of guide image remains an accurate match with image 702 (intra-op image), and thus determination is affirmative (match). Both region 704 and intra-op image 702 have matching fields of view. See Fig. 7.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Themelis by obtaining a matching two-dimensional image that is taught by Piron, to make the invention that identifies the matching two-dimensional cross-section from the pre-operative three-dimensional image data; thus, one of ordinary skilled in the art would be motivated to combine the references to avoid the need for the surgeon to manually align the pre-operative three-dimensional data with the surgeons actual view. This consumes time and time is critical for all surgery. Further, the surgeon may become distracted during surgery (Themelis, Paragraph [0006]). Further, since the intraoperative image data (actual view) may be two-dimensional, representing a plane in the field of view (Themelis, Paragraph [0032]), the matching pre-operative image section should also be two-dimensional.
Themelis in view of Piron does not explicitly disclose (storing)
Piron 2 is in the same field of art of performing surgical image registration to align two images with each other such that both are placed in a common coordinate system. Further, Piron 2 teaches (storing) the matching image in the memory (Abstract, Paragraph [0050], Figs. 7A & 7B, Piron 2 teaches storing the preoperative image and the preoperative measurements. A transformation for registering the preoperative image and the intraoperative image is stored in association with one of the intraoperative image and the preoperative image. The Examiner interprets the preoperative image to be “the matching image” since as shown in Figs, 7A & 7B, there is a portion of preoperative image (400) that matches intraoperative image (600), see Fig. 7B, matched up image.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Themelis in view of Piron by also storing the matching image in memory that is taught by Piron 2, to make the invention that stores both the intra-op image and the corresponding matching image in memory/storage for retrieval by the surgeon during surgery and to facilitate simultaneous visualization of images obtained preoperatively, alongside images obtained intraoperatively (Piron 2, Paragraph [0072]); thus, one of ordinary skilled in the art would be motivated to combine the references to enable direct comparison between preoperative and intraoperative images (Piron 2, Paragraph [0072), therefore allowing a surgeon to quickly verify that the intra-operative images match a predetermined pre-operative surgical plan.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 14, Themelis in view of Piron in further view of Piron 2 teaches the surgical imaging device of claim 13, wherein the acquisition parameters are at least one of: working distance, magnification, focal length, position of the one or more detectors of the intra-op image, orientation of the one or more detectors, coordinates of a cross-section of the image data that includes the matching field of view, or user input (Paragraph [0026], Themelis teaches additional information about the position of the intraoperative image data is available. A position sensor may be provided for generating position data representative of the position of a field of view of the camera assembly. The position data may comprise at least one of focal length, field depth, and distance setting of the camera assembly. Under BRI, the Examiner interprets “at least one of” and “or” to mean only one of the types of acquisition parameters is required to meet the claim limitation.).
In regards to Claim 15, Themelis in view of Piron in further view of Piron 2 teaches the surgical imaging device of claim 13, wherein the one or more processors are further configured for: obtaining, from an image-guided surgery system (Claim 7, Themelis teaches acquiring positioning data from the optical lens system.), one or more of the plurality of acquisition parameters used for constructing the matching image (Claim 7, Paragraph [0026], Themelis teaches the positioning data being representative of at least one of the focal length of an optical lens system used for acquiring the intraoperative image data, the field depth of the optical lens system, the distance setting of the optical lens system, a size, a dimension and an orientation of a field of view, at the time of acquiring the interoperative image data. The mapping of the pre-op 3D image data to the interoperative image data may be facilitated if in addition to the mere image data, additional information about the position of the intraoperative image data, and thus the vascular plexus structure within the tissue, is available), and determining the cross-section plane based on the one or more of the plurality of acquisition parameters of the image-guided surgery system (Claim 7, Paragraph [0041], Fig. 1, Themelis teaches the positioning data are used in identifying the vascular plexus structure in the pre-operative 3D image data and elastically matching the pre-operative 3D image data to the intraoperative image data and displaying the output image data. The elastic matching may define a section through the pre-operative three-dimensional image data (6) which results in the field of view (10) represented in the interoperative image data (14).).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Wittenberg et al. (U.S. Patent Pub. No. 20120188352 A1) teaches a device for matching an intraoperative live image of an operating field, i.e., an image obtained during a surgical intervention, with an image of the operating field, e.g., the heart, has been preoperatively obtained. A transformation is performed which adapts the preoperative image to the intraoperative/live image in the best possible manner. See Paragraph [0012].
Lafitte et al. (NPL “Patch-based field-of-view matching in multi-modal images for electroporation-based ablations”, Sept. 2020) teaches “a new fast method to align the FOV in multi-modal 3D medical images. To this end, a patch-based approach is introduced and combined with a state-of-the-art multi-modal image similarity metric in order to cope with multi-modal medical images. The occurrence of estimated patch shifts is computed for each spatial direction and the shift value with maximum occurrence is selected and used to adjust the image field-of-view. (Abstract)”
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SYDNEY L BLACKSTEN whose telephone number is (571)272-7120. The examiner can normally be reached 8:30am-4:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Oneal Mistry can be reached at 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SYDNEY L BLACKSTEN/Examiner, Art Unit 2674
/ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674
