DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDSs) submitted on April 4, 2023, October 16, 2023, December 11, 2023, March 26, 2024 and January 23, 2025 have been considered and by the examiner and placed in the file.
Response to Arguments
Applicant’s arguments filed on January 12, 2026 have been considered and are persuasive in part. Applicant argues that new limitations that have been added to the independent claims by the present amendment are not taught or suggested by the prior art of record. Specifically, Applicant argues that the prior art of record does not teach or suggest the following limitations that have been added to the independent claims: “recognizing, from the image data, an optical indicia on the tool indicating the known size of the tool tip," and "fitting a shape of the tool onto the pixel mask by utilizing the geometry of the tool tip and the known size of the tool tip indicated by the optical indicia." In the new rejection set forth below, Applicant does not rely on the previously-relied on prior art of record as explicitly teaching this new limitation. A new prior art reference is relied on for its explicit teaching of “recognizing, from the image data, an optical indicia on the tool indicating the known size of the tool tip".
Claim Interpretation
The claims in this application are given their broadest reasonable interpretation (BRI) using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The BRI of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification.
The BRIs for certain terms in the claims are provided below. Should applicant wish different definitions, Applicant should point to the portions of the specification that clearly support a different interpretation.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-3, 5, 8-10,12, 14, 18-20, 26-29, 32-33, 37 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Publ. Appl. No. 2021/0059758 A1 to Avendi et al. (hereinafter referred to as “Avendi”) in view of U.S. Publ. Appl. No. 2006/0258938 A1 to Hoffman et al. (hereinafter referred to as “Hoffman”) and U.S. Publ. Appl. No. 2021/0338149 A1 to Angelo (hereinafter referred to as “Angelo”) and further in view of U.S. Publ. Appl. No. 2015/0161802 A1 to Christiansen (hereinafter referred to as “Christiansen”).
Regarding claim 1, Avendi discloses a system (Fig. 1, imaging system 10, para. [0046]) for performing tissue landmarking during a medical procedure in a patient (para. 0097]), the landmarking performed with a tool having a tool tip with a known geometry (needle guide assembly 82, Fig. 3; paras. [0047] and [0051], the controller/processor 16 locates, identifies, labels and tracks the medical instrument 145 of the assembly 82 or other medical instrument and places a landmark at the location of the tip of the assembly 82 on the tip and/or on the anatomical object), the system comprising:
one or more processors (controller/processor 16, Figs. 2 and 3, para. [0052]);
a memory (Fig. 2, memory device(s) 18, para. [0047]) to store instructions operable on the one or more processors; and
an interface to receive a video stream (the controller/processor 16 receives the video feed 60, paras. [0047]-[0048], Figs. 2 and 3; Figs. 7 and 8) encoding image data from an imaging device (probe 28, Fig. 1B) positioned to image a scene of the tool positioned at a selected tissue location of the patient (para. [0048], controller/processor 16 tracks the tool and or anatomical objects contained in images of a scene from the video feed), wherein the one or more processors execute instructions stored in the memory (para. [0048]) to process the image data received from the interface and to perform operations comprising:
recognizing the tool from the image data and generating a segmented tool outline depicting pixels where the tool is detected, wherein the segmented tool outline is represented as a pixel mask (paras. [0080]-[0082], the tip of the medical instrument is identified using a machine-learned model 110a that segments the medical instrument 145; paras. [0094]-[0096] and Figs. 14-15, the segmented tool outline depicts pixels where the tool is detected represented as a pixel mask);
recognizing, from the image data, an optical indicia on the tool indicating the known size of the tool tip (Avendi does not explicitly disclose this limitation);
fitting a shape of the tool onto the pixel mask by utilizing the geometry of the tool tip and the known size of the tool tip indicated by the optical indicia, wherein the fitted shape of the tool minimizes error or inaccuracy of a shape of the pixel mask (Avendi does not explicitly disclose using shape fitting to fit the shape of the tool onto the pixel mask);
recognizing a predetermined gesture of the tool (Avendi does not explicitly disclose this limitation);
determining a location to be landmarked based on the geometry and size of the tool tip and the recognized predetermined gesture of the tool (the BRI for the phrase “determining a location to be landmarked” is based on para. [0105] of the present disclosure, which describes the process as recognizing the target anatomical structure and designating the location of the recognized anatomical structure as the location to be landmarked; the BRI for the phrase “based on the geometry and size of the tool tip and the recognized predetermined gesture of the tool” is based on paras. [0122]-[0127], which describe adjusting the location to be landmarked based on the geometry and size of the tool tip and based on the position and/or orientation of the tool tip to compensate for inaccuracies of the segmented outlines of the tool tip caused by camera viewing angle or lighting conditions; Avendi teaches determining a location to be landmarked as the location of the recognized anatomical object of interest, but does not explicitly teach that the location to be landmarked is based on the geometry and size of the tool tip and a recognized predetermined gesture of the tool);
utilizing the fitted shape of the tool and the determined location to generate a digital landmark at the selected tissue location (Avendi teaches generating a digital landmark at the selected tissue location based on the determined location, para. [0097], Fig. 14 landmark 42, for example; Avendi does not explicitly teach utilizing a fitted shape of the tool to generate the digital landmark); and
overlaying the digital landmark onto the video stream (para. [0081], the digital landmark is overlaid onto the video stream).
As indicated above, Avendi does not explicitly disclose using shape fitting to fit the shape of the tool onto the pixel mask representing the segmented tool outline. The BRI for shape fitting is based on para. [0023] of the specification, which describes the mapping process as recognizing the tip of the tool while tracking it, generating a segmented outline of the tip and fitting a refined or smoothed outline of the tip onto the segmented outline.
Hoffman, in the same field of endeavor, discloses this shape fitting limitation. In Hoffman, the tool tracking and identification algorithm segments the tool tip, i.e., the effector end of the tool, into segments corresponding to the outline of the tool tip (paras. [0174] and [0179]) and a computer model silhouette of the tool tip corresponding to a refined outline of the extracted edges and contours of the tool tip is fitted onto the segmented outline (paras. [0180]-[0181]). The fitted shape of the tool in Hoffman is based on the geometry and size of the tool because it is based on fitting of the refined computer model silhouette to the extracted edges and contours of the tool, which, in turn, define the geometry and size of the tool. In Hoffman, the fitted shape of the tool minimizes error or inaccuracy of the shape of the pixel mask representing the segmented tool outline because the computer model silhouette is refined until the difference (i.e., the error) between the silhouette and the detected edges and contour of the tool is minimized.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the landmarking algorithm performed by the imaging system 10 of Avendi based on the teachings of Hoffman to use shape fitting as taught by Hoffman to fit the shape of the tool tip onto the pixel mask based on the geometry and size of the tool and to determine the location to be landmarked based in part on the size and geometry of the tool tip such that the fitted shape and the determined location are utilized to generate the digital landmark. A person of skill in the art would have been, before the effective filing date of the present disclosure, motivated to make the modification to achieve greater accuracy in landmark placement and improve the quality of the medical procedure. The modification could have been made by one of ordinary skill in the art before the effective filing data of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results.
As indicated above, Avendi does not explicitly disclose recognizing, from the image data, an optical indicia on the tool indicating the known size of the tool tip. Angelo, in the same field of endeavor, discloses recognizing, from the image data, an optical indicia on the tool indicating the known size of the tool tip (para. [0019] discloses that the probe tip can include indicia indicative of the size of the probe tip that is recognized the probe tip can include ruler markings, which are indicia of size of the probe tip: “[i]n a preferred embodiment, the probe tip size is a known three millimeters (3 mm) reference size; and the probe tip may be provided with millimeter ruler markings. Other known probe tip sizes and scale of reference marking may be available depending on the anatomical feature application.”; see also para. [0022] discussing the probe tip having a known size that allows it to be used to determine the sizes of anatomical features based on the known size of the probe tip).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the landmarking algorithm performed by the imaging system 10 of Avendi as modified based on the teachings of Hoffman further to use the known tool size and indicia on the tip indicative of the known size (e.g., ruler markings) as taught by Angelo in combination with the shape fitting algorithm of Hoffman to fit the shape of the tool tip onto the pixel mask based on the geometry and known size of the tool. A person of skill in the art would have been, before the effective filing date of the present disclosure, motivated to make the modification to achieve greater accuracy in placing landmarks. The modification could have been made by one of ordinary skill in the art before the effective filing data of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (configuring the system to recognize indicia located on the tool tip indicative of tool size).
As indicated above, Avendi does not explicitly disclose recognizing a predetermined gesture of the tool and determining the location to be landmarked based on the geometry and size of the tool tip and the recognized predetermined gesture of the tool.
Christiansen, in the same field of endeavor, discloses recognizing gestures of a medical tool during a medical procedure and determining an action to be taken based on the recognized gesture (paras. [0049], [0103], [0107] and [0110]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to further modify the landmarking algorithm performed by the imaging system 10 of Avendi based on the teachings of Christiansen to determine the location to be landmarked based on the size and geometry of the tool tip and the recognized gesture as taught by Christiansen. A person of skill in the art would have been, before the effective filing date of the present disclosure, motivated to make the modification to achieve greater accuracy in landmark placement while also improving the quality of the medical procedure. The modification could have been made by one of ordinary skill in the art before the effective filing data of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (incorporating a gesture recognition algorithm into the landmarking algorithm of Avendi).
Regarding claim 2, Avendi discloses that the scene is that of a selected tissue site or intraoperative tissue site (para. [0081]).
Regarding claim 3, Avendi discloses that the segmented tool outline depicts only pixels where the tool is detected (para. [0096], the segmented tool outline is represented as a border of pixels corresponding to the outline of the detected tool; see also para. [0060], if outlines of objects of interest, e.g., medical tools, are to be identified by the machine learning model 110, then the model 110 will be trained with ground truth outlines to be used for identification).
Regarding claim 5, Hoffman discloses that the error or inaccuracy of the shape of the pixel mask is caused by a viewing angle of the imaging device relative to the tool or the tool tip or by suboptimal lighting of the tool or the tool tip (paras. [011], [0029], [0054] and [0174]-[0181], from the field of view (FOV) of the camera, the image of the tool in the video feed is captured and the tool’s position and orientation are estimated; the position and orientation of the camera model silhouette is then modified, or refined, to correct for the difference, i.e., the inaccuracies, between the position and orientation of the computer model silhouette and the position and orientation of the tool based on the tool edge pixels extracted from the image until the difference is at a minimum; that position and orientation of the computer model silhouette is then overlaid over the predicted orientation and position of the tool in the displayed image; this transforms the image being displayed into a perspective image simulating the view point of the operator that is manipulating the tool as opposed to the FOV of the camera that captures the image; the error can also be caused by suboptimal lighting conditions, as discussed in para. [0175]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to further modify the landmarking algorithm performed by the imaging system 10 of Avendi based on the teachings of Hoffman to use shape fitting as taught by Hoffman to fit the shape of the tool tip onto the pixel mask based on the geometry and size of the tool and to map the location to be landmarked based on the size and geometry of the tool tip to minimize the error or inaccuracy of the shape of the pixel mask caused by the viewing angle of the imaging device relative to the tool or the tool tip or by suboptimal lighting of the tool or the tool tip as taught by Hoffman A person of skill in the art would have been, before the effective filing date of the present disclosure, motivated to make the modification to achieve greater accuracy and robustness in landmark placement when confronted with the imaging device having a poor FOV of the tool or partial occlusion of the tool from the FOV of the imaging device. The modification could have been made by one of ordinary skill in the art before the effective filing data of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results.
Regarding claim 8, Avendi teaches that the mapping of the location to be landmarked is further based on a predetermined target point of the tool tip because Avendi teaches that the location to be landmarked is a predetermined target anatomical structure (e.g., a target nerve) and navigating the tool tip to the landmarked target (para. [0051]).
Regarding claim 9, Avendi teaches that the mapping of the predetermined target point is determined using a machine learning algorithm dataset, a training data set, or a combination thereof (paras. [0053]-[0059] and Figs. 4 and 5).
Regarding claim 10, Avendi teaches that the imaging device is an arthroscope, an endoscope or a laparoscope (para. [0051]).
Regarding claim 12, Avendi teaches that the recognition of the tool or the generation of the segmented tool outline is performed using a deep learning network or architecture (para. [0056]).
Regarding claim 14, Hoffman teaches that the fitting of the shape of the tool onto the pixel mask is performed using a computer vision algorithm or a shape fitting algorithm (paras. [0180]-[0181]), as discussed above in the rejection of claim 1. Avendi discloses using a computer vision algorithm to identify, track and landmark the tool tip and/or the target anatomical structure (para. [0001]), but does not explicitly disclosure using shape fitting.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to further modify the landmarking algorithm performed by the imaging system 10 of Avendi based on the teachings of Hoffman to use shape fitting as taught by Hoffman in conjunction with, or in lieu of, the computer vision algorithm of Avendi to fit the shape of the tool tip onto the pixel mask based on the geometry and size of the tool and to map the location to be landmarked based on the size and geometry of the tool tip such that the fitted shape and the mapped location are utilized to generate the digital landmark. A person of skill in the art would have been, before the effective filing date of the present disclosure, motivated to make the modification to achieve greater accuracy and robustness in landmark placement. The modification could have been made by one of ordinary skill in the art before the effective filing data of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results.
Regarding claim 18, Avendi discloses a variety of tools that can be considered part of the system of Avendi (para. [0051]).
Regarding claim 19, Avendi discloses that the tool can comprise a surgical probe such as a laparoscopic or anthroscopic probe (para. [0051]).
Regarding claim 20, Avendi does not explicitly disclose that the tip of the tool has a rounded or spherical geometry. Angelo discloses that the tip of the probe can have a rounded or spherical geometry (para. [0013]: “[i]n FIG. 1, a preferred embodiment of the medical device probe or endoscopic probe is shown with a tip of various shapes, such as a sphere, cube, triangle, ring, etc….”).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to further modify the landmarking algorithm performed by the imaging system 10 of Avendi based on the teachings of Angelo to use tool tips having rounded or spherical geometries as taught by Angelo. A person of skill in the art would have been, before the effective filing date of the present disclosure, motivated to make the modification to “optimize recognition by the artificial intelligence system and algorithms” as taught by Angelo. The modification could have been made by one of ordinary skill in the art before the effective filing data of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (designing the tool tip to have a suitable geometry).
Regarding claim 26, the limitation of the shaft of the tool having a conical shape is taught by Avendi A portion of Fig. 3 of Avendi is duplicated below, which shows the conical shape of the needle 45 as it tapers toward the distal tip of the needle 45.
PNG
media_image1.png
200
400
media_image1.png
Greyscale
Regarding claim 27, the rejection of claim 20 applies mutatis mutandis to claim 27.
Regarding claim 28, to the extent that claim 28 recites limitations that are recited in claim 1, the rejection of claim 1 applies mutatis mutandis to claim 28. The only limitation recited in claim 28 that is not also recited in claim 1 is the tool, is wherein the tool comprises a shaft and the tool tip is coupled to the shaft, wherein the tool tip is a patterned tool tip configured to enhance recognition of the tool tip by a deep learning network, a machine learning algorithm, or a computer vision algorithm executed on the one or more processors.
Avendi does not explicitly teach that the tool tip is a patterned tip configured to enhance recognition of the tool tip by deep learning. The BRI for the term “patterned” is based on the plain meaning of the term as it would be understood by one of ordinary skill in the art because the specification of the present disclosure does not define what is meant by the term and no example of a patterned tip is shown in the drawings of the present disclosure. To pattern a thing is known in the art to mean disposing a pattern in or on a surface. In the context of the present disclosure, the BRI of “patterned tip” is disposing a pattern on or in the tool tip that can be recognized by a deep learning network, a machine learning algorithm, or a computer vision algorithm executed on the one or more processors.
Hoffman discloses that markers are drawn or formed on the tool tip, i.e., on the effector end to assist the processor 101 performing the computer vision algorithm to recognize the tool and to determine the orientation and position of the tool. Hoffman describes an example of the patterned tool tip as an effector end having four stripes 801-804 and four line segments 811-814 drawn or formed thereon. The pattern is recognized by the computer vision algorithm and used to determine the rotational position of the tool about the axis of the tool (para. [0173]-[0181]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to further modify the landmarking algorithm performed by the machine learned model 110 of Avendi based on the teachings of Hoffman to use a patterned tool tip as taught by Hoffman A person of skill in the art would have been, before the effective filing date of the present disclosure, motivated to make the modification to achieve greater accuracy in determining the position and orientation of the tool tip, resulting in better accuracy in landmark placement. The modification could have been made by one of ordinary skill in the art before the effective filing data of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results.
Regarding claim 29, the rejection of claim 5 applies mutatis mutandis to claim 29.
Regarding claim 32, the rejection of claim 1 applies mutatis mutandis to claim 32.
Regarding claim 33, the rejection of claim 3 applies mutatis mutandis to claim 33.
Regarding claim 37, Avendi discloses displaying the overlaid video stream on one or more display devices (para. [0081], Fig. 7).
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over Avendi in view of Hoffman, Angelo and Christiansen and further in view of an article entitled “Real-time Visual Servoing For Laparoscopic Surgery”, by Wei et al., published in IEEE Engineering in Medicine and Biology, Vol. 16, No. 1, pp. 40-45, 1997 (hereinafter referred to as “Wei”)
Avendi does not explicitly disclose the tool tip having a pattern, a texture, or a contrast configured to enhance recognition. The BRI for the term “contrast” in claim 22 is based on para. [0012] of the specification, which indicates that the term means that the color of the tool tip contrasts with the color of the tissue at the selected tissue site.
Wei, in the same field of endeavor, discloses a method for tracking laparoscopes in which color information on the tool is used to track the tool (page 40) to facilitate recognition of the tool when it is partially occluded by anatomical features or blood. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to further modify the tool used in Avendi to color the tip of the tool with a color that contrasts with the color of tissue at the target tissue site as taught by Wei. A person of skill in the art would have been, before the effective filing date of the present disclosure, motivated to make the modification to assist the machine learning algorithm of Avendi in detecting the tip of the tool even in situations in which the tip may be partially occluded by anatomical features or blood as taught by Wei. The modification could have been made by one of ordinary skill in the art before the effective filing data of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL J SANTOS whose telephone number is (571)272-2867. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Bella can be reached at (571)272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL J. SANTOS/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667