Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 6/2/26 have been fully considered.
Applicant argues that Zacharias and Fung do not teach the limitations incorporated into claims 1 and 12 from previous claims 9 and 20, namely the image enhancement information including information associated with one or more recognized anatomical features of a patient or one or more equipment of the ophthalmic surgical system visible within the received video content, and the overlay including at least one graphical element identifying the recognized anatomical features or equipment. These arguments are not persuasive.
First, applicant argues that Wolf only describes analyzing surgical video for indexing, timeline markers, summaries, reports and other post-processing analytics. This is not persuasive. Wolf is not limited to post-processing. Wolf teaches receiving surgical video and analyzing it in real time to provide decision support to the surgeon (pars. 21-22). Wolf also teaches using machine learning and object detection algorithms to recognize medical instruments and anatomical structures that are visible in the video (pars. 156, 202-203 and 205) and teaches overlaying graphical markers based on that analysis on the video that is output for display (pars. 108 and 144).
Next, applicant argues that Ren only addresses the placement of a surgical data overlay relative to a region of interest and does not identify recognized anatomical features or equipment. This is not persuasive. Ren recognizes the surgical tool within the image, including by image analysis that detects the boundaries of the tool (pars. 35-36), and displays graphical elements identifying what has been recognized. The region of interest and the scan line are marked on the displayed image along with the visible surgical instrument (par. 30 and figure 2), and anatomical features such as an epi-retinal membrane, its thickness and its contour are emphasized with highlighting (par. 44). It is also noted that the claims recite the recognized anatomical features or the equipment in the alternative, so a graphical element identifying either one meets the claim.
That is, in combination they teach arranging the graphical elements representing the ophthalmic surgical system parameters to not overlap with the graphical elements identifying the recognized anatomical features or equipment. Ren determines an optimal location for the surgical data overlay based on the region of interest and updates the position so the data does not obstruct the identified features (pars. 45, 50-52). Ren further shows an overlay position that partially obstructs the tool and teaches setting the position so that the overlay does not do so (pars. 56-57 and figures 5A-5D).
As such, this rejection is made final. See the rejection below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 9, 11-14, 20 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Zacharias (20030159141) in view of Fung (20210203889) in view of Wolf (20220301674) in further view of Ren (20170035287).
Regarding claim 1, Zacharias teaches a method for processing and streaming video content relating to a surgical procedure the method comprising: receiving video content captured from a video source during the surgical procedure (pars. 5-6);
receiving overlay information from a surgical system coupled with the video source (pars. 5-6 and 83);
providing at least a portion of the video content and the overlay information to a processor configured to provide video stream enhancement or to a processing device, via a data interface, that is configured to provide a video stream enhancement application (pars. 85-86, 97-98);
receiving image enhancement information from the processor or from the processing device via the data interface (par. 98);
processing, based on the received image enhancement information, the video content to produce an enhanced video stream including an overlay representing at least a portion of the overlay information (par. 6 and 86);
Zacharias teaches sending, using a communication interface, the enhanced video stream to one or more receive devices configured to display the enhanced video stream (see figures 2 and 3a and par. 86).
Zacharias does not teach ophthalmic surgery.
Fung teaches this in par. 2 and it also teaches ophthalmic parameters being displayed on a monitor in par. 21 and the abstract.
It would have been obvious prior to the effective filing date of the invention to one of ordinary skill in the art to include in Zacharias the ability to present the live surgical video and associated parameters in the integrated GUI format as taught by Fung. The reason is to allow the system to display data without having to look for the parameters in multiple areas.
Zacharias and Fung do not teach the image enhancement information including information associated with one or more recognized anatomical features of a patient or one or more equipment of the surgical system visible within the received video content.
Wolf teaches analyzing surgical video, including in real time (pars. 21-22), using machine learning and object detection algorithms to recognize medical instruments and anatomical structures visible in the video (pars. 156, 202-203 and 205), the locations of which may be represented as pixels, bounding boxes or bounding polygons (par. 153), and overlaying graphical markers based on that analysis on the video output for display (pars. 108 and 144).
It would have been obvious prior to the effective filing date of the invention to one of ordinary skill in the art to include in Zacharias and Fung the ability to use learning models to recognize the instruments and anatomy in the video as taught by Wolf. The reason is to allow the use of learning models to analyze video and give the surgeon information about what is in the video.
Zacharias, Fung and Wolf do not explicitly teach the overlay including at least one graphical element identifying the recognized anatomical features or equipment on the displayed video.
Ren teaches an overlay including at least one graphical element identifying the one or more recognized anatomical features of the patient or the one or more equipment of the ophthalmic surgical system, wherein the tool is recognized within the image (pars. 35-36), the region of interest and scan line are marked on the displayed image (par. 30 and figure 2), and anatomical features such as an epi-retinal membrane are emphasized with highlighting (par. 44).
It would have been obvious prior to the effective filing date of the invention to one of ordinary skill in the art to include in Zacharias, Fung and Wolf the ability to identify the recognized features on the displayed image as taught by Ren. The reason is to let the surgeon see the recognized anatomy and tools directly on the live image without searching for them.
Regarding claim 2, Wolf teaches AI/ML in par. 159 and 244.
Regarding claim 3, see pars. 2 and 21 and the abstract of Fung.
Regarding claim 9, the rejection of claim 1 teaches all the elements of claim 9 except for the specifics of displaying the information.
Ren teaches that, based on a presence of the graphical element identifying the recognized features, the data is arranged to not overlap with the graphical elements identifying the one or more recognized anatomical features of the patient or the one or more equipment of the ophthalmic surgical system (see pars. 45, 50, 52 and 56-57 and figures 5A-5D).
Regarding claim 11, see par. 24 and figure 7 of Fung, a dominant display and surgical control display.
Regarding claim 12, see the rejection of claim 1.
Regarding claim 13, see the rejection of claim 2.
Regarding claim 14, see the rejection of claim 3.
Regarding claim 20, see the rejection of claim 9.
Regarding claim 22, see the rejection of claim 11.
Claim(s) 4 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Zacharias (20030159141) in view of Fung (20210203889) in view of Wolf (20220301674) in view of Ren (20170035287) in further view of Hertweck (20080114387).
Regarding claim 4, Zacharias teaches a foot pedal and wand in figure 2 and overlaid data in pars. 97-98.
Zacharias, Fung, Wolf and Ren do not teach the specific control system.
Hertweck teaches this in the abstract and pars. 2 and 4.
It would have been obvious prior to the effective filing date of the invention to one of ordinary skill in the art to include in Zacharias and Fung the ability to have the surgical control means as taught by Hertweck. The reason is to allow the system a simple control means for surgery.
Regarding claim 15, see the rejection of claim 4.
Claims 5-7 and 16-18 are rejected under 35 U.S.C. 103 as being unpatentable over Zacharias (20030159141) in view of Fung (20210203889) in view of Wolf (20220301674) in view of Ren (20170035287) in further view of Braness (20130054958).
Regarding claim 5, Fung and Zacharias teach streaming in the rejection of claim 1.
They do not teach adaptive streaming.
Braness teaches receiving a request from at least one of the one or more receive devices to obtain the enhanced video stream; transmitting, using the communication interface, the enhanced video stream to the one of the one or more receive devices in response to the received request; and adjusting a resolution of the enhanced video stream in accordance with a bandwidth or image quality requirement associated with a link to at least one of the one or more receive devices (see the abstract, pars. 6, 60-61).
It would have been obvious prior to the effective filing date of the invention to one of ordinary skill in the art to include in Fung and Zacharias the ability to use adaptive streaming as taught by Braness. The reason is to adjust the resolution based on streaming capabilities.
Regarding claim 6, see par. 89 of Zacharias, templates are customized. Also see pars. 30-35 of Fung.
Regarding claim 7, see pars. 30-33 of Fung, rearranging.
Regarding claims 16-18, see the rejection of claims 5-7.
Claims 8 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Zacharias (20030159141) in view of Fung (20210203889) in view of Wolf (20220301674) in view of Ren (20170035287) in view of Braness (20130054958) in further view of Mentis (20160210411).
Regarding claim 8, Zacharias, Fung, Wolf, Ren and Braness do not teach an insert.
Mentis teaches wherein the change in the one or more characteristics of the graphical presentation of the overlay includes an indication to insert one or more graphical elements providing notational information associated with the ophthalmic surgical procedure (see pars. 4, 18-21 and 74, annotations).
It would have been obvious prior to the effective filing date of the invention to one of ordinary skill in the art to include in Zacharias, Fung, Wolf, Ren and Braness the ability to have insertion graphical elements as taught by Mentis. The reason is to show more information to the surgeon.
Regarding claim 19, see the rejection of claim 8.
Claims 10 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Zacharias (20030159141) in view of Fung (20210203889) in view of Wolf (20220301674) in view of Ren (20170035287) in further view of Charles (20080291015).
Regarding claim 10, Zacharias, Fung, Wolf and Ren do not teach ambient light sensor.
Charles teaches the enhanced video stream is produced with image quality parameters selected based on ambient lighting conditions of the ophthalmic surgical environment (see pars. 5-6).
It would have been obvious prior to the effective filing date of the invention to one of ordinary skill in the art to include in Zacharias and Fung the ability to take lighting into account when adjusting the display as taught by Charles. The reason is to improve visibility.
Regarding claim 21, see the rejection of claim 10.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HADI AKHAVANNIK whose telephone number is (571)272-8622. The examiner can normally be reached 9 AM - 5 PM Monday to Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HADI AKHAVANNIK/Primary Examiner, Art Unit 2676
Prosecution Insights