DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims and Rejections
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
Claims 1-30 were pending prior to examination. Claims 4, 7, 14, 16-18, 21, 23, and 28-19 were cancelled in a preliminary amendment. Claim 30 was cancelled in the last amendment. Claims 1-3, 5-6, 8-13, 15, 19, 22, 24, and 27 were amended.
Claims 1-3, 5-6, 8-13, 15, 19-20, 22, and 24-27 are now pending.
In view of the claim amendments, Section 112(f) is no longer invoked for claim limitations that were previously recited in claim 19.
In view of the claim amendments, the Section 101 and Section 112(b) rejections have been withdrawn.
Claim Objections
Claim 1 is objected to because of the following informalities:
Claim 1 should be amended to read “wherein the implantation apparatus comprises an implantation needle….”
Claim 1 should be amended to read “capturing a first image by a first camera of a brain surface, and….”
Claim 19, at the end of the implantation apparatus paragraph, should be amended to read “wherein the neurosurgery robot is configured to:….”
Claim 19 should be amended to read “a first camera configured to capture a first image of a brain surface; a second camera configured to capture a second image of a brain surface; and….”
Claims 24 and 27 should be amended to read “the neurosurgery robot….” Alternatively, claim 19 could be amended to recite “a neurosurgery robotic system…” If Applicant chooses the latter, please be sure to make similar changes throughout.
Claim 27 has a parallelism issue and should be amended to change the formatting and recite:
the neurosurgery robotic system is further configured to:
determine the implantation sequence of the electrodes so that an electrode being implanted does not apply an action force on an implanted electrode; or
determine the implantation sequence of the electrodes so that an electrode being implanted does not apply an action force on an implanted electrode, and plan paths for implanting the electrodes based on the implantation sequence of the electrodes, wherein the paths are not crossed.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5, 6, 8, 9, 13, 15, 19, 20, 22, 24, 25, and 27 are rejected under 35 U.S.C. 103 as being unpatentable over by U.S. Patent Appl. Publ. No. 2020/0085508 A1 (hereinafter referred to as “O’HARA”) and Haouchine N, Nercessian M, Juvekar P, Golby A, Frisken S. Cortical vessel segmentation for neuronavigation using vesselness-enforced deep neural networks. IEEE Transactions on Medical Robotics and Bionics. 2021 Oct 22;4(2):327-30 (hereinafter referred to as “HAOUCHINE”) and Lu B, Chu HK, Huang K, Lai J. Surgical suture thread detection and 3-D reconstruction using a model-free approach in a calibrated stereo visual system. IEEE/ASME transactions on mechatronics. 2019 Sep 20;25(2):792-803 (hereinafter “BO LU”).
O’HARA teaches a machine vision based method for determining electrode implantation position.
With respect to claim 1, O’HARA teaches systems and methods that use computer vision techniques in connection with robotic surgery. (Abstract). “The system may use computer vision techniques to facilitate implanting a micro-manufactured bio-compatible electrode device in biological tissue (e.g., neurological tissue such as the brain) using robotic assemblies.” (Abstract). The method includes:
capturing a first image by a first camera for a brain surface, and capturing a second image by a second camera for the brain surface. See Figure 2 shown here. “The computing system may include computer software that provides a user interface configured to display the images obtained by cameras 204 and 205…Cameras 204 and 205 may be configured to image the surface of the biological tissue in the surgical field.” ([0052]).
selecting at least one implantation position in the implantable area, and calculating a distance between an implantation position of the at least one implantation position and a known electrode position of an electrode implanted earlier, so as to determine an implantation sequence of the electrodes corresponding to the at least one implantation
PNG
media_image1.png
1314
948
media_image1.png
Greyscale
position based on the distance. “The robot registers insertion sites to a common coordinate frame with landmarks on the skull, which, when combined with depth tracking, enables precise targeting of anatomically defined brain structures. Integrated custom computer instructions may allow pre-selection of all insertion sites, enabling planning of insertion paths optimized to minimize tangling and strain on the threads.” ([0086]). “In some embodiments, the computer software may automatically propose one or more target locations for implantation. The user interface may be configured such that the user of the computing system may provide approval of the automatically generated proposed target locations. Such automatically generated proposed target locations can be based on the obtained image, e.g., by applying computer vision, artificial intelligence, or machine learning heuristics to the image. The computing system may propose target locations that avoid vasculature, are geometrically advantageous for recording and/or stimulating sites of interest, and/or are a minimal distance apart.” ([0058]).
wherein the implantation apparatus comprises an implantation needle (see Figure 2, “needle 220”), an implantation feeding mechanism (“NPC 210” and “needle pincher 222”) and an implantation actuation mechanism (“needle actuator 214”), wherein the implantation needle is configured to engage a free end of an electrode with a needle tip portion thereof so as to drive motion of the electrode (see, e.g., Figure 8), the implantation feeding mechanism is configured to move the implantation needle along a longitudinal direction of the implantation apparatus, and the implantation actuation mechanism is configured to drive the implantation needle to insert the needle tip portion of the implantation needle into the brain (see, e.g., [0081]-[0087]).
While O’HARA does not explicitly teach performing arithmetic processing on the first image and the second image, wherein, a vascular area mask of the brain surface is obtained based on a vascular segmentation algorithm to determine an implantable area in an image of the brain surface. O’HARA teaches using image analysis to distinguish blood vessels along the brain surface. First, O’HARA teaches avoiding vasculature “to avoid damage to the blood-brain barrier and thereby reduce inflammatory response.” ([0086]; see also [0058]). Second, O’HARA teaches configuring different light sources “to apply light in a way that can differentiate biological tissue and features such as blood vessels,” ([0055]), and to use filters “to identify blood vessels.” ([0057]). O’HARA specifically teaches using wavelength 590 nm because this “amber light may be absorbed by hemoglobin such that the image obtained by the cameras can be used to differentiate between biological tissue and blood vessels.” ([0055]). Third, O’HARA teaches that the two-camera configuration can be used to create a stereo composite image to distinguish the biological structures and tissue and to determine a contour or surface map. ([0057]). Fourth, O’HARA teaches processing the images to identify the blood vessels and that the different implantation targets may be identified automatically. ([0118]). To summarize O’HARA, “[t]he computing system may propose target locations that avoid vasculature, are geometrically advantageous for recording and/or stimulating sites of interest, and/or are a minimal distance apart.” ([0058]; see also [0086]: “Integrated custom computer instructions may allow pre-selection of all insertion sites, enabling planning of insertion paths optimized to minimize tangling and strain on the threads. The planning feature highlights the ability to avoid vasculature during insertions, one of the key advantages of inserting electrodes individually.”).
HAOUCHINE teaches an efficient method to segment cortical vessels in craniotomy images acquired by the surgical microscope. (Abstract). HAOUCHINE “uses a vesselness-enforced convolutional neural network to classify each pixel of a craniotomy image as a vessel or surrounding tissue.” (Abstract). HAOUCHINE summarized various segmentation methods that can be used to identify the vessels. HAOUCHINE teaches a two-stage model that includes a segmenter network that estimates a probability map that outline the cortical vessels from an input image a vesselness-enforced network that, given the previously extracted probability map, corrects the geometrical continuities of the segmented images to produce a smooth and continuous vascular structure. (p.2, left column, II. Method, A. Overview). Accordingly, HAOUCHINE teaches performing arithmetic processing on the first image and the second image, wherein, a vascular area mask of the brain surface is obtained based on a vascular segmentation algorithm to determine an implantable area in a brain surface image.
It would have been obvious to one having ordinary skill in the art at the time of filing to generate a vascular area mask of the brain surface as taught in HAOUCHINE and use the vascular area mask to identify the brain vessels for the O’HARA system. One having ordinary skill in the art would have been motivated to use the HAOUCHINE method to avoid the vasculature during the implantation process as taught in O’HARA. There would have been a reasonable expectation of success because both O’HARA and HAOUCHINE use optical images of the brain surface.
With respect to the claim limitation parameters in the vascular segmentation algorithm are adjusted based on sites of implanted electrode, the vascular segmentation algorithm, as recited, provides a vascular area mask to determine an implantable area in the image of the brain surface. As such, the broadest reasonable interpretation of claim 1 includes the vascular segmentation algorithm determining the location and size of the implantable area. (MPEP 2111).
O’HARA is clearly concerned with the location and size of the implantable area as well as the sites of previously-implanted electrodes. O’HARA teaches “[t]he computing system may propose target locations that avoid vasculature, are geometrically advantageous for recording and/or stimulating sites of interest, and/or are a minimal distance apart.” ([0058]; see also [0086]: “Integrated custom computer instructions may allow pre-selection of all insertion sites, enabling planning of insertion paths optimized to minimize tangling and strain on the threads. The planning feature highlights the ability to avoid vasculature during insertions, one of the key advantages of inserting electrodes individually.”). Thus, the location and size of the vascular area mask would be based on the sites of previously-implanted electrodes.
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the vascular segmentation algorithm of HAOUCHINE such that the parameters in the vascular segmentation algorithm are adjusted based on sites of implanted electrode. One having ordinary skill in the art would have been motivated to modify the HAOUCHINE algorithm to provide a vascular area mask having a location and size based on sites of previously-implanted electrodes so that future sites are identified within the area determined by the vascular area mask. There would have been a reasonable expectation of success because both O’HARA and HAOUCHINE use optical images of the brain surface.
While O’HARA does not explicitly teach matching imaging of the first camera and the second camera to obtain a transformation matrix, projecting a first straight line where the implantation position is situated in the imaging of the first camera onto the imaging of the second camera, and determining an intersection point between the first straight line and a second straight line where the implantation position is situated in the imaging of the second camera as a predicted landing point of an implantation apparatus, O’HARA teaches using the cameras to track the needle and to analyze the brain surface. “Stereoscopic cameras, computer vision methods such as monocular extended depth of field calculations, and illumination with 525 nm light can allow for precise estimation of the location of the cortical surface.” ([0085]; see also [0079] describing how the cameras are used to track the needle tip as well).
In the field of robot-assisted surgery, BO LU teaches three-dimensional position estimation method that may be used for wound suturing. BO LU teaches matching imaging of the first camera and the second camera to obtain a transformation matrix. BO LU uses the “fundamental matrix of the stereo camera” when calculating the projection of the 3-D suture’s tip. (p.796, right column, B. Evaluations of Suture Thread’s Tip Correspondence). BO LU teaches projecting a first straight line to a point situated in the imaging of the first camera onto the imaging of the second camera. See Figure 5 here. “The projection of the 3-D suture thread’s tip W1S in the right camera projection W1S,r should lie on the epipolar line l2…[can be calculated].” (p.796, right column, B. Evaluations of Suture Thread’s Tip Correspondence). BO LU also teaches determining an intersection point between the first straight line and a
PNG
media_image2.png
200
400
media_image2.png
Greyscale
second straight line where the implantation position is situated in the imaging of the second camera. See Figure 5 showing the tip and the grasping point of the thread.
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the O’HARA system by projecting a straight line onto the implantation position, as recited in the claimed invention, and determine an intersection point between the first and second straight lines, as recited in the claimed invention, where the implantation position is situated in the imaging of the second camera as a predicted landing point. One having ordinary skill in the art would have been motivated to incorporate the BO LU process because the BO LU system’s stereo-camera configuration is similar to O’HARA’s and provides a robust process across different visual conditions, can improve accuracy, and can be used for more difficult thread/electrode configurations. (p.802, Conclusion). There would have been a reasonable expectation of success as O’HARA suggests and BO LU teaches that a stereo camera system can be used to robotic sew a thread to tissue.
With respect to claim 2, O’HARA teaches that the method may include full-supervision or semi-supervision control is performed on the implantation apparatus according to the predicted landing point. “In some embodiments, the computer software may automatically propose one or more target locations for implantation. The user interface may be configured such that the user of the computing system may provide approval of the automatically generated proposed target locations…In some embodiments, the computing system can apply such heuristics to determine and/or select a target site automatically, and does not require input from a user.” ([0058]).
With respect to claim 3, O’HARA teaches that the method may include the first camera and the second camera have a same imaging plane; or the first camera and the second camera have the same imaging plane, and the first camera and the second camera form an included angle of about 90 on a horizontal projection. (See Figure 5 of LU and compare to Figure 1 of Applicant’s disclosure in which “cameras 101 and 102” have “the same imaging plane.”
With respect to claim 5, O’HARA teaches that the method may include the first camera and the second camera are associated with an optical system of an electrode implantation system (see, e.g., Figure 2 in which each of cameras 204 and 205 has affixed thereto a light source 208, 209 (respectively), and the first camera and the second camera are coupled to a motion control system of an electrode implantation system. The cameras 204, 205 can be mounted to and move with the inserter head 200, which is movable on a stage. (see, e.g., [0070] and [0081]).
With respect to claim 6, O’HARA teaches that the method may include the optical system comprises an external light source which uniformly illuminates the brain surface; or the optical system comprises an external light source which uniformly illuminates the brain surface, and the external light source has a wavelength range of 495 nm to 570 nm. The system “contain light sources configured to illuminate the surgical field. The light sources illuminating the electrode device or an insertion needle can produce light of wavelengths selected based on a material associated with the electrode device or needle, while the light sources illuminating the surgical field can produce light of wavelengths chosen for imaging the target tissue.” ([0044]).
With respect to claim 8, O’HARA teaches that the method may include performing image processing by the first camera and the second camera respectively, and merging image-processed data from the first camera and the second camera. “In some embodiments, the computing system may process the obtained images such that biological structures and tissue are distinguishable within the image, or can determine a contour or surface map of biological tissue 208, such as the exterior contours of a brain, or a particular target site within biological tissue 208. In an embodiment, the computing system can form a composite image (e.g., a stereo composite image) based on target tissue images from multiple cameras (e.g., a left and a right camera), thereby providing Extended Depth of Field (EDF) information.” ([0057]).
determining coordinates of the implantation apparatus in the imaging of the first camera and the second camera according to the merged data. “In an embodiment, the system and/or a camera can image the needle. In an embodiment, when imaging the insertion needle, the system uses a light source to illuminate the needle with another wavelength of light that does not cause the electrode device to fluoresce, such as red light. Based on the images obtained and/or coordinates associated with motion of the robotic assembly determined in operation 1204, the system can configure one or more robotic assemblies to engage the needle.” ([0108]).
With respect to claim 9, O’HARA teaches that the method may include the motion control system controls movement of the implantation apparatus according to the coordinates. “In an embodiment, triangulating the location of the electrode further comprises determining, based on the first image and the second image, three-dimensional coordinates of the electrode associated with a motion of the robotic assembly to engage the needle with the implantable device.” ([0107]). “In an embodiment, the system and/or a camera can image the needle. In an embodiment, when imaging the insertion needle, the system uses a light source to illuminate the needle with another wavelength of light that does not cause the electrode device to fluoresce, such as red light. Based on the images obtained and/or coordinates associated with motion of the robotic assembly determined in operation 1204, the system can configure one or more robotic assemblies to engage the needle.” ([0108]).
With respect to claim 13, O’HARA teaches that the method may include the implantation sequence of the electrodes is determined so that an electrode being implanted does not apply an action force on an implanted electrode; or the implantation sequence of the electrodes is determined so that an electrode being implanted does not apply an action force on an implanted electrode, and paths for implanting the electrodes are planned based on the implantation sequence of the electrodes, wherein the paths are not crossed. “Integrated custom computer instructions may allow pre-selection of all insertion sites, enabling planning of insertion paths optimized to minimize tangling and strain on the threads.” ([0086]).
With respect to claim 15, O’HARA does not explicitly teach feature matching of data is performed on the first camera and the second camera for calibration; or feature matching of data is performed on the first camera and the second camera for calibration, and a square filter is used to realize an image processing effect of Gaussian blur.
However, BO LU teaches using a calibrated stereo camera. (Abstract). “For a fixed stereo camera system, calibration can be performed offline using a chessboard with known dimensions.” (p.796, bottom right column).
It would have been obvious to one having ordinary skill in the art at the time of filing to use feature matching of data, as taught in BO LU, in order to calibrate the stereo camera. In order to use the BO LU method, one would have been motivated to calibrate the stereo-camera as it is necessary to use a calibrated stereo-camera in BO LU. Moreover, calibration is a common, necessary step for surgical navigation systems.
With respect to claim 19, O’HARA teaches a machine vision based electrode implantation system. O’HARA teaches systems and methods that use computer vision techniques in connection with robotic surgery. (Abstract). “The system may use computer vision techniques to facilitate implanting a micro-manufactured bio-compatible electrode device in biological tissue (e.g., neurological tissue such as the brain) using robotic assemblies.” (Abstract). The system includes a neurosurgery robot (see, e.g., [0042] and [0132] and Figs. 1A, 1B, and 2) comprising:
a first camera configured to capture a first image for a brain surface, and a second camera configured to capture a second image for a brain surface. See Figure 2 shown here. “The computing system may include computer software that provides a user interface configured to display the images obtained by cameras 204 and 205…Cameras 204 and 205 may be configured to image the surface of the biological tissue in the surgical field.” ([0052]).
an implantation apparatus, comprising an implantation needle (see Figure 2, “needle 220”), an implantation feeding mechanism (“NPC 210” and “needle pincher 222”) and an implantation actuation mechanism (“needle actuator 214”), wherein the implantation needle is configured to engage a free end of an electrode with a needle tip portion thereof so as to drive motion of the electrode (see, e.g., Figure 8), the implantation feeding mechanism is configured to move the implantation needle along a longitudinal direction of the implantation apparatus, and the implantation actuation mechanism is configured to drive the implantation needle to insert the needle tip portion of the implantation needle into the brain (see, e.g., [0081]-[0087]).
wherein the neurosurgery robot is configured to:
select at least one implantation position in the implantable area, and calculate a distance between an implantation position of the at least one implantation position and a known electrode position of an electrode implanted earlier, so as to determine an implantation sequence of the electrodes corresponding to the at least one implantation position based on the distance. “The robot registers insertion sites to a common coordinate frame with landmarks on the skull, which, when combined with depth tracking, enables precise targeting of anatomically defined brain structures. Integrated custom computer instructions may allow pre-selection of all insertion sites, enabling planning of insertion paths optimized to minimize tangling and strain on the threads.” ([0086]). “In some embodiments, the computer software may automatically propose one or more target locations for implantation. The user interface may be configured such that the user of the computing system may provide approval of the automatically generated proposed target locations. Such automatically generated proposed target locations can be based on the obtained image, e.g., by applying computer vision, artificial intelligence, or machine learning heuristics to the image. The computing system may propose target locations that avoid vasculature, are geometrically advantageous for recording and/or stimulating sites of interest, and/or are a minimal distance apart.” ([0058]).
control the implantation apparatus in real time according to the predicted landing point...“The inserter head of the targeting sub-system 200 can include an imaging stack, such as cameras 204, 205, and insertion camera stack 206, used for guiding the needle into the thread loop, insertion targeting, live insertion viewing, and insertion verification.” ([0056]; see also [0085]). “In an embodiment, the robot can feature an auto-insertion mode, which can insert up to 6 threads (192 electrodes) per minute. While the entire insertion procedure can be automated, a surgeon can retain control, and can make manual micro-adjustments to the thread position before each insertion into the target tissue, such as a cortex.” ([0086]). “In some embodiments, the computer software may automatically propose one or more target locations for implantation. The user interface may be configured such that the user of the computing system may provide approval of the automatically generated proposed target locations. Such automatically generated proposed target locations can be based on the obtained image, e.g., by applying computer vision, artificial intelligence, or machine learning heuristics to the image. The computing system may propose target locations that avoid vasculature, are geometrically advantageous for recording and/or stimulating sites of interest, and/or are a minimal distance apart.” ([0058])… until an implantation point coincides with the predicted landing point. “Computing system 1008 can process images obtained by cameras 1002 according to a computer vision heuristic in order to determine implantable electrode device 1004 and/or an insertion needle are correctly implanted. Based on this determination, computing system 1008 can send further instructions to robotic assembly 1010 and robotic assembly 1012. For example, computing system 1008 can instruct robotic assemblies 1010 and 1012 to undertake further motions to correct the positioning or implantation of the electrode device. In a second example, computing system 1008 can determine that no further motions are needed, and robotic surgery can proceed to the next stage, e.g. implantation of a subsequent thread, as in the examples of FIGS. 8 and 9.” ([0089]). O’HARA also teaches correcting an electrode implantation. “In an embodiment, the system and/or a camera can obtain an image of the electrode and the target surgical tissue, and can verify implantation of the electrode based on the image. Based on this verification, the computing system can determine whether to end process 1200 or whether further correction is needed. In an embodiment, the computing system can instruct the robotic surgery system and/or the robotic assemblies to return to step 1208 in order to correct the positioning or implantation of the electrode device.” ([0116]).
While O’HARA does not explicitly teach perform arithmetic processing on the first image and the second image to determine an implantable area of an image of the brain surface by obtaining a vascular area mask of the brain surface based on a vascular segmentation algorithm. O’HARA teaches using image analysis to distinguish blood vessels along the brain surface. First, O’HARA teaches avoiding vasculature “to avoid damage to the blood-brain barrier and thereby reduce inflammatory response.” ([0086]; see also [0058]). Second, O’HARA teaches configuring different light sources “to apply light in a way that can differentiate biological tissue and features such as blood vessels,” ([0055]), and to use filters “to identify blood vessels.” ([0057]). O’HARA specifically teaches using wavelength 590 nm because this “amber light may be absorbed by hemoglobin such that the image obtained by the cameras can be used to differentiate between biological tissue and blood vessels.” ([0055]). Third, O’HARA teaches that the two-camera configuration can be used to create a stereo composite image to distinguish the biological structures and tissue and to determine a contour or surface map. ([0057]). Fourth, O’HARA teaches processing the images to identify the blood vessels and that the different implantation targets may be identified automatically. ([0118]). To summarize O’HARA, “[t]he computing system may propose target locations that avoid vasculature, are geometrically advantageous for recording and/or stimulating sites of interest, and/or are a minimal distance apart.” ([0058]; see also [0086]: “Integrated custom computer instructions may allow pre-selection of all insertion sites, enabling planning of insertion paths optimized to minimize tangling and strain on the threads. The planning feature highlights the ability to avoid vasculature during insertions, one of the key advantages of inserting electrodes individually.”).
HAOUCHINE teaches an efficient method to segment cortical vessels in craniotomy images acquired by the surgical microscope. (Abstract). HAOUCHINE “uses a vesselness-enforced convolutional neural network to classify each pixel of a craniotomy image as a vessel or surrounding tissue.” (Abstract). HAOUCHINE summarized various segmentation methods that can be used to identify the vessels. HAOUCHINE teaches a two-stage model that includes a segmenter network that estimates a probability map that outline the cortical vessels from an input image a vesselness-enforced network that, given the previously extracted probability map, corrects the geometrical continuities of the segmented images to produce a smooth and continuous vascular structure. (p.2, left column, II. Method, A. Overview). Accordingly, HAOUCHINE teaches perform arithmetic processing on the first image and the second image to determine an implantable area of an image of the brain surface by obtaining a vascular area mask of the brain surface based on a vascular segmentation algorithm.
It would have been obvious to one having ordinary skill in the art at the time of filing to perform arithmetic processing on the first image and the second image to determine an implantable area of an image of the brain surface by obtaining a vascular area mask of the brain surface based on a vascular segmentation algorithm. One having ordinary skill in the art would have been motivated to use the HAOUCHINE method (i.e., algorithm) to avoid the vasculature during the implantation process as taught in O’HARA. There would have been a reasonable expectation of success because both O’HARA and HAOUCHINE use optical images of the brain surface.
While O’HARA does not explicitly teach match imaging of the first camera and the second camera to obtain a transformation matrix, project a first straight line where the implantation position is situated in the imaging of the first camera onto the imaging of the second camera, and determine an intersection point between the first straight line and a second straight line where the implantation position is situated in the imaging of the second camera as a predicted landing point of an implantation apparatus, O’HARA teaches using the cameras to track the needle and to analyze the brain surface. “Stereoscopic cameras, computer vision methods such as monocular extended depth of field calculations, and illumination with 525 nm light can allow for precise estimation of the location of the cortical surface.” ([0085]; see also [0079] describing how the cameras are used to track the needle tip as well).
In the field of robot-assisted surgery, BO LU teaches three-dimensional position estimation method that may be used for wound suturing. BO LU teaches matching imaging of the first camera and the second camera to obtain a transformation matrix. BO LU uses the “fundamental matrix of the stereo camera” when calculating the projection of the 3-D suture’s tip. (p.796, right column, B. Evaluations of Suture Thread’s Tip Correspondence). BO LU teaches projecting a first straight line to a point situated in the imaging of the first camera onto the imaging of the second camera. See Figure 5 here. “The projection of the 3-D suture thread’s tip W1S in the right camera projection W1S,r should lie on the epipolar line l2…[can be calculated].” (p.796, right column, B. Evaluations of Suture Thread’s Tip Correspondence). BO LU also teaches determining an intersection point between the first straight line and a second straight line where the implantation position is situated in the imaging of the second camera. See Figure 5 showing the tip and the grasping point of the thread.
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the O’HARA system by projecting a straight line onto the implantation position, as recited in the claimed invention, and determine an intersection point between the first and second straight lines, as recited in the claimed invention, where the implantation position is situated in the imaging of the second camera as a predicted landing point. One having ordinary skill in the art would have been motivated to incorporate the BO LU process because the BO LU system’s stereo-camera configuration is similar to O’HARA’s and provides a robust process across different visual conditions, can improve accuracy, and can be used for more difficult thread/electrode configurations. (p.802, Conclusion). There would have been a reasonable expectation of success as O’HARA suggests and BO LU teaches that a stereo camera system can be used to robotic sew a thread to tissue.
With respect to claim 20, O’HARA teaches that the method may include full-supervision or semi-supervision control is performed on the implantation apparatus according to the predicted landing point; or the first camera and the second camera have a same imaging plane; or feature matching of data is performed on the first camera and the second camera for calibration. “In some embodiments, the computer software may automatically propose one or more target locations for implantation. The user interface may be configured such that the user of the computing system may provide approval of the automatically generated proposed target locations…In some embodiments, the computing system can apply such heuristics to determine and/or select a target site automatically, and does not require input from a user.” ([0058]).
With respect to claim 22, O’HARA teaches that the system may include the first camera and the second camera are associated with an optical system of the electrode implantation system, (see, e.g., Figure 2 in which each of cameras 204 and 205 has affixed thereto a light source 208, 209 (respectively), and the first camera and the second camera are coupled to a motion control system of the electrode implantation system respectively. The cameras 204, 205 can be mounted to and move with the inserter head 200, which is movable on a stage. (see, e.g., [0070] and [0081]).
With respect to claim 24, O’HARA teaches that the system may include the neurosurgery robotic system being further configured to
merge image-processed data from the first camera and the second camera. “In some embodiments, the computing system may process the obtained images such that biological structures and tissue are distinguishable within the image, or can determine a contour or surface map of biological tissue 208, such as the exterior contours of a brain, or a particular target site within biological tissue 208. In an embodiment, the computing system can form a composite image (e.g., a stereo composite image) based on target tissue images from multiple cameras (e.g., a left and a right camera), thereby providing Extended Depth of Field (EDF) information.” ([0057]).
determine coordinates of the implantation apparatus in the imaging of the first camera and the second camera according to the merged data. “In an embodiment, the system and/or a camera can image the needle. In an embodiment, when imaging the insertion needle, the system uses a light source to illuminate the needle with another wavelength of light that does not cause the electrode device to fluoresce, such as red light. Based on the images obtained and/or coordinates associated with motion of the robotic assembly determined in operation 1204, the system can configure one or more robotic assemblies to engage the needle.” ([0108]).
With respect to claim 25, O’HARA teaches that the system may include a motion control system controls movement of the implantation apparatus according to the coordinates. “In an embodiment, triangulating the location of the electrode further comprises determining, based on the first image and the second image, three-dimensional coordinates of the electrode associated with a motion of the robotic assembly to engage the needle with the implantable device.” ([0107]). “In an embodiment, the system and/or a camera can image the needle. In an embodiment, when imaging the insertion needle, the system uses a light source to illuminate the needle with another wavelength of light that does not cause the electrode device to fluoresce, such as red light. Based on the images obtained and/or coordinates associated with motion of the robotic assembly determined in operation 1204, the system can configure one or more robotic assemblies to engage the needle.” ([0108]).
With respect to claim 27, O’HARA teaches that the system may include the neurosurgery robotic system being further configured to:
determine the implantation sequence of the electrodes so that an electrode being implanted does not apply an action force on an implanted electrode. “Integrated custom computer instructions may allow pre-selection of all insertion sites, enabling planning of insertion paths optimized to minimize tangling and strain on the threads.” ([0086]).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over by U.S. Patent Appl. Publ. No. 2020/0085508 A1 (hereinafter referred to as “O’HARA”) and Haouchine N, Nercessian M, Juvekar P, Golby A, Frisken S. Cortical vessel segmentation for neuronavigation using vesselness-enforced deep neural networks. IEEE Transactions on Medical Robotics and Bionics. 2021 Oct 22;4(2):327-30 (hereinafter referred to as “HAOUCHINE”) and Lu B, Chu HK, Huang K, Lai J. Surgical suture thread detection and 3-D reconstruction using a model-free approach in a calibrated stereo visual system. IEEE/ASME transactions on mechatronics. 2019 Sep 20;25(2):792-803 (hereinafter “BO LU”) as applied to claim 5 above, and further in view of Ghanbari, Leila, et al. “Craniobot: A computer numerical controlled robot for cranial microsurgeries.” Scientific reports 9.1 (2019): 1023 (hereinafter “GHANBARI”).
O’HARA does not explicitly teach using the motion control system comprises three stepper motors configured to control the implantation apparatus to move in an area substantially parallel to the electrode implantation area. However, O’HARA system is capable of moving along the brain surface while also adjusting depth of the needle.
GHANBARI teaches “a cranial microsurgery platform that combines automated skull surface profiling with a computer numerical controlled (CNC) milling machine to perform a variety of cranial microsurgical procedures on mice.” Figure 1b shows different views of the “craniobot,” including three motors for X, Y, and Z axes. “A 3-axis hobby machining mill (Lukcase LC8110) was used as the base machine for the Craniobot…3D positioning was accomplished by three NEMA 17 stepper motors and M8 stainless steel lead-screw assemblies.”
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the O’HARA system to include three stepper motors that are configured to control the implantation apparatus to move in an area substantially parallel to the electrode implantation area. One would have been motivated to use three stepper motors for controlling movement of the inserter head along the surface so that the needle could be positioned at any position within the area, including adjustments for depth, as taught in O’HARA. There would have been a reasonable expectation of success as GHANBARI teaches that stepper motors can be incorporated into microsurgery platforms.
Claims 11, 12, and 26 are rejected under 35 U.S.C. 103 as being unpatentable over by U.S. Patent Appl. Publ. No. 2020/0085508 A1 (hereinafter referred to as “O’HARA”) and Haouchine N, Nercessian M, Juvekar P, Golby A, Frisken S. Cortical vessel segmentation for neuronavigation using vesselness-enforced deep neural networks. IEEE Transactions on Medical Robotics and Bionics. 2021 Oct 22;4(2):327-30 (hereinafter referred to as “HAOUCHINE”) and Lu B, Chu HK, Huang K, Lai J. Surgical suture thread detection and 3-D reconstruction using a model-free approach in a calibrated stereo visual system. IEEE/ASME transactions on mechatronics. 2019 Sep 20;25(2):792-803 (hereinafter “BO LU”) as applied to claims 1 and 19 above, and further in view of U.S. Patent Appl. Publ. No. 2010/0119113 A1 (hereinafter referred to as “KULESCHOW”) and Mart, Fernando, Fredy H. Mart, and Holman Montiel Ariza. “Genetic Algorithms Applied to the Searching of the Optimal Path in Image-based Robotic Navigation Environments.” International Journal of Advanced Computer Science and Applications 13.5 (2022) (hereinafter “SANTA”).
With respect to claims 11 and 26, neither O’HARA and HAOUCHINE teach the claim recitations of claims 11 and 26 pertaining to image segmentation. However, the recited steps of claims 11 and 26 are common steps used for image segmentation. For example, KULESCHOW teaches segmentation methods for identifying objects that have no solid appearance (e.g., scratches). While KULESCHOW focuses on industrial applications, KULESCHOW teaches that the processes are also applicable to blood vessels. (see, e.g., [0025]). KULESCHOW teaches using “pixel-oriented” methods that “decide for every individual picture element whether the same belongs to a certain segment or not. This decision can be influenced, for example, by the surroundings. The most widespread method is the thresholding method.” ([0007]). Notably, “[w]hen segmenting images with elongated objects” KULESCHOW teaches initially binarizing the image with a known method, such as global or adaptive thresholding. ([0009]). KULESCHOW also teaches identifying “blobs” (i.e., bubbles) and removing them. (see, e.g., [0056]).
Accordingly, KULESCHOW teaches transforming the brain surface image into a gray-scale map; segmenting the gray-scale map according to an adaptive threshold; removing small contour noise from the segmented gray-scale map; performing opening operation to remove a bubble noise pattern in a blood vessel.
However, KULESCHOW does not teach performing inverse operation; performing expansion processing to obtain a safe distance at a boundary of a vascular area; and performing inverse operation again.
Nonetheless, SANTA teaches an optimal path-finding strategy that includes a safety or buffer zone from obstacles or walls. To provide a safe distance from obstacles or walls, SANTA teaches using performing an inverse operation, performing expansion processing to obtain a safe distance at a boundary of a vascular area, and performing inverse operation again. More specifically, SANTA teaches “[i]n a binary image, the dilation process is achieved over the white objects, so it is necessary to invert the image to obtain the desired white obstacles. At the end, the obtained image is inverted again to obtain an image with the dilated black obstacles. After inverting the obstacles binary image, a morphology dilation operation is performed by means of using a 2D convolution between that image and a disk shape a radius equivalent to rd. After that, the image is inverted back, and the result is shown in Fig. 4, where the area of the obstacles or walls are expanded because of the dilation operation.” (p.804, left column; see also Figure 4 showing the obstacles or walls after dilation).
Accordingly, SANTA teaches performing inverse operation; performing expansion processing to obtain a safe distance at a boundary of a vascular area; and performing inverse operation again.
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the image processing of O’HARA to include process the image data as recited in claim 11. One of ordinary skill in the art would have been motivated to process the image data in this manner to identify the blood vessels but also provide a safe buffer zone to account for error in the data. There would have been a reasonable expectation of success as these processes are well-known in image processing.
With respect to claim 12, O’HARA teaches that parameters in the vascular segmentation algorithm are adjusted based on a number of sites to be detected, a site distance, and an imaging resolution. “The computing system may propose target locations that avoid vasculature, are geometrically advantageous for recording and/or stimulating sites of interest, and/or are a minimal distance apart. In some embodiments, the computing system can apply such heuristics to determine and/or select a target site automatically, and does not require input from a user.” ([0058]). “In a seventh step 1314, the computing system may generate a surgical plan based on the implantation targets and/or the obtained image.” ([0118]).
RESPONSE TO APPLICANT’S ARGUMENTS
Applicant’s arguments filed on August 6, 2026 have been considered but are not persuasive. Applicant argues that the prior art does not teach “wherein parameters in the vascular segmentation algorithm are adjusted based on sites of implanted electrode.” (see p. 14 of Applicant’s Remarks). Applicant specifically alleges that O’HARA does not teach “dynamic adjustment linked to implantation layout density or image clarity” and that HAOUCHINE “cannot adjust segmentation thresholds according to intraoperative implantation distribution” and that no other cited reference “connects segmentation parameter calibration with planned electrode implantation sites (such as the quantity and spacing thereof), a dedicated constraint customized for multi-electrode implantation planning.” (see pp. 14-15 of Applicant’s Remarks).
However, none of these specific examples are required by the claim because none of them are recited in the claim. Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
As explained above, the vascular segmentation algorithm, as recited in the claim, provides a vascular area mask to determine an implantable area in the image of the brain surface. The broadest reasonable interpretation of claim 1 includes the vascular segmentation algorithm determining the location and size of the implantable area based on sites of implanted electrode. (MPEP 2111).
O’HARA is clearly concerned with the location and size of the implantable area as well as the sites of previously-implanted electrodes. O’HARA teaches “[t]he computing system may propose target locations that avoid vasculature, are geometrically advantageous for recording and/or stimulating sites of interest, and/or are a minimal distance apart.” ([0058]; see also [0086]: “Integrated custom computer instructions may allow pre-selection of all insertion sites, enabling planning of insertion paths optimized to minimize tangling and strain on the threads. The planning feature highlights the ability to avoid vasculature during insertions, one of the key advantages of inserting electrodes individually.”). One having ordinary skill in the art would have been motivated to modify the HAOUCHINE algorithm to provide a vascular area mask having a location and size based on sites of previously-implanted electrodes, as taught in O’HARA, so that future sites are identified within the area determined by the vascular area mask.
Prior Art Made of Record
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US20210264623A1 teaches adjusting parameters of vascular segmentation algorithm. “The use of CSF boundary layers to mask and constrain the parameter space for blood vessel segmentation using up-scaled contrast MRIs is a novel approach for facilitating the segmentation of cerebral vasculature.” ([0031]).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASON P GROSS whose telephone number is (571)272-1386. The examiner can normally be reached Monday-Friday 9:00-5:00CT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anne M. Kozak can be reached at (571) 270-5284. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JASON P GROSS/Examiner, Art Unit 3797
/SERKAN AKAR/Primary Examiner, Art Unit 3797
Prosecution Insights