CTNF 19/047,522 CTNF 88163 DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Claim Rejections - 35 USC § 112 07-30-01 AIA The following is a quotation of the first paragraph of 35 U.S.C. 112(a): (a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention. The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112: The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention. Claims 1-11 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. If the neural network is not a generic but a specifical neural network, Applicant has failed to show any possession of an algorithm for this neural network that can classify at least one of anatomical objects, surgical objects, and tissue manipulations in the live feed of video images. 07-30-02 AIA The following is a quotation of 35 U.S.C. 112(b): (b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention. The following is a quotation of 35 U.S.C. 112 (pre-AIA), second paragraph: The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention. 07-34-01 Claims 1-11 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. In re claims 1 and 11, it is unclear what the metes and bounds of this neural network are and how it is implemented. It is unclear what algorithm is used for this neural network. Claims 1-11 essentially claim a magic black box can do everything but without explain what is in side this black box. If the neural network is not a generic but a specifical neural network, Applicant has failed to explain what this neural network is. Furthermore, it is unclear what the metes and bounds of “a surgical guidance generator for outputting, in real time, guidance based on … ‘are. What’s the scope of the guidance? Who or what is being guided? In re claim 2, it is unclear how the neural network is trained to do identify tissue .. including tumor. In re claim 3, it is unclear how the neural network is trained to identify changes in contour of specific tissue types. In re claim 4, it is unclear how the neural network is trained to identify and track changes over time and across … In re claim 5, it is unclear how the neural network is trained to identify a pointer instrument … In re claim 6, it is unclear how the surgical guidance generator is trained to interact with a surgeon using voice recognition. In re claim 7, it is unclear what is “surgeon- and surgery-specific” manners are. What is a surgery-specific manners? What is “surgeon-“? Is it supposed to be “surgeon-specific”? But even if it is “surgeon-specific,” what’s the scope of “surgeon-specific” manners? The Examiner will interpret as any movement done by the surgeon in a surgery. Furthermore, how is movement efficiency metrics being determined? Applicant describes “Surgeon-specific metrics could include movement efficiency metrics based on, for example, the amount of time a particular instrument was used for a given step in the procedure.” See para 0033. But an A based on a B else does not explain what the A is. It is unclear how either Surgeon-specific metrics or movement efficiency metrics is determined. What are these metrics? A number? The Examiner will interpret as any desired result. If the neural network is not a special neural network, but a generic one, then Claim Rejections - 35 USC § 101 07-04-01 AIA 07-04 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claims 1-11 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Step 1: Statutory Category: YES - The claim recites a method for generating and providing artificial intelligence assisted surgical guidance and, therefore, is a method. Step 2A, Prong 1, Judicial Exception: YES - The claim recites the limitation of identify and classify. This limitation, as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. The method does not even require any apparatus, or any computer components. That is, other than reciting “a neural network,” nothing in the claim element precludes the step from practically being performed in the mind. For example, but for the “a neural network” language, the claim encompasses a user simply looking at the video with his mind, to identify and classify objects. The mere nominal recitation of a processor with a neural network does not take the claim limitation out of the mental processes grouping. Thus, the claim recites a mental process. Step 2A, Prong 2, Integrated into Practical Application: No - The claim recites additional elements: additional organs or tumors, additional changes in contour, or changes of an object, tip of the instrument, voice recognition, displaying, overlaying different videos and images. All these steps recited at a high level of generality (i.e., identifying objects), and amounts to mere object viewing and drawing the outline, and adding two different images on top of each other, which is a form of insignificant extra-solution activity. Each of the additional limitations is no more than mere instructions identify an object (with a neural network). The combination of these additional elements is no more than mere instructions to apply the exception using a generic neural network. Accordingly, even in combination, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to the abstract idea. Step 2B, Inventive Concept: No - As discussed with respect to Step 2A Prong Two, the additional elements in the claim amount to no more than mere instructions to apply the exception using a generic neural network. The same analysis applies here in 2B, i.e., mere instructions to apply an exception on a generic computer cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B. Under the 2019 PEG, a conclusion that an additional element is insignificant extra-solution activity in Step 2A should be re-evaluated in Step 2B. Here, training and identification of types and changes were considered to be extra-solution activity in Step 2A, and thus it is re-evaluated in Step 2B to determine if it is more than what is well-understood, routine, conventional activity in the field. The background of the example does not provide any indication that the a neural network, and storage is anything other than a generic, off-the-shelf computer component, and the Symantec, TLI, and OIP Techs . court decisions cited in MPEP 2106.05(d)(II) indicate that mere collection or receipt of data over a network is a well ‐ understood, routine, and conventional function when it is claimed in a merely generic manner (as it is here). Accordingly, a conclusion that the collecting and comparing step is well-understood, routine, conventional activity is supported under Berkheimer Option 2. For these reasons, there is no inventive concept in the claim, and thus it is ineligible. Claim Rejections - 35 USC § 102 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-07-aia AIA 07-07 The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action: A person shall be entitled to a patent unless – 07-12-aia AIA (a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention. 07-15-03-aia AIA Claim(s) 1, 4-9, 11 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipate by Roh et al.(US 2019/0262084, hereinafter Roh ‘084) . In re claims 1 and 11, Roh ‘084 teaches a method for generating and providing artificial intelligence assisted surgical guidance (abstract), the method comprising: analyzing of video images (0028) from surgical procedures and training a neural network to identify at least one of anatomical objects, surgical objects, and tissue manipulations in the video images (0026-0029, 0031, 0033, 0034, 0044, 0049, 0053, 00550074, 0077, 0079 etc.); receiving, by the neural network (0050), a live feed (0107) of video images from a surgery (0028-0030); classifying, by the neural network, at least one of anatomical objects, surgical objects, and tissue manipulations in the live feed of video images (0026-0029, 0031, 0033, 0034, 0044, 0049, 0053, 0055, 0074, 0077-0079, 0097, 0100, 0101, 0107, 0118etc.); outputting, in real time (0078-0079, 0097, 0100, 0101, 0107, 0118), surgical guidance based on the classified at least one of anatomical objects, surgical objects, and tissue manipulations in the live feed of video images (0041-0046, 0050). In re claim 4, Roh ‘084 teaches wherein the neural network includes training the neural network to identify and track changes in at least one of anatomical objects, surgical objects, and tissue manipulations over the course of each of the surgical procedures and over time (0028, 0077-0080, note that real time include changes over time and multiple surgical procedures in at least one of anatomical objects). In re claim 5, Roh ‘084 teaches wherein the neural network to identify surgical objects include training the neural network to identify a pointer instrument having a pointer end (0026, 0029, 0030, 0048, 0049, 0073, 0076, 0097, 0103) that when brought in close proximity to an anatomical object in a live feed of video (0028) from a surgery triggers generating of output identifying the anatomical object (0026, 0029, 0030, 0048, 0049, 0073, 0076, 0097, 0103). In re claim 6, Roh ‘084 teaches wherein training the neural network includes training the neural network to interact with a surgeon using voice recognition (0007, 0154). In re claim 7, Roh ‘084 teaches wherein outputting the surgical guidance includes developing algorithms that further process and display output from the neural network in surgeon- and surgery-specific manners including movement efficiency metrics (0090, 0117) and intraoperative metrics to predict success of each surgery (0147). In re claim 8, Roh ‘084 teaches 1 wherein outputting the surgical guidance simultaneously processing data from input streams including pre-operative imaging (0057-0059), patient-specific risk factors (0031, 0084, 0086), surgical object cost data (0031, 0089, 0090, 0101, 0129, 0159), and/or intraoperative vital signs (0101). In re claim 9, Roh ‘084 teaches wherein outputting the surgical guidance includes overlaying the surgical guidance on the live feed of video images or onto a surgical field using augmented reality (0079, 0092, 0097) . Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-23-aia AIA The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows: 1. Determining the scope and contents of the prior art. 2. Ascertaining the differences between the prior art and the claims at issue. 3. Resolving the level of ordinary skill in the pertinent art. 4. Considering objective evidence present in the application indicating obviousness or nonobviousness. 07-20-02-aia AIA This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention. 07-21-aia AIA Claim (s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Roh ‘084 in view of Mann et al. (Segmentation of retinal blood vessels using artificial neural networks for early detection of diabetic retinopathy, AIP Conf. Proc. 1836, 020026 (2017), hereinafter Mann ‘2017) . In re claim 2, Roh ‘084 teaches wherein training the neural network to identify anatomical objects includes training the neural network to identify bone (0049-0050), muscle (0059, 0065, 0103-0108, 0112), tendons (note that Roh ‘084 already teaches ligament, 0103, cartilage, and muscle 0112, and it is known that tendon is also a connective tissue that connects muscle to bone, whereas cartilage is a soft, rubbery, gel-like coating on the ends of bones, where they articulate, that protects joints and facilitates movement, and whereas a ligament is an elastic band of tissue that connects bone to bone and provides stability to the joint. Hence, it would have been obvious to use neural network to identify every tissue that connects muscle to bone), organs (claim 12), blood vessels , and nerve (0049, 0085, 0086, etc.) roots, as well as abnormal tissues (0029) including tumor (0151). Note that it would have been obvious to use neural network to identify every single tissue to the extend that’s possible. Applicant has not specifically shown any details that would make identify any specific tissue any different than a black box neural network. Furthermore, Mann ‘2017 teaches wherein the neural network is trained to identify tissue including blood vessels (abstract, whole document). It would have been prima facie obvious to one of ordinary skills in the art at the time of invention to modify the method/device of Roh ‘084 to include the features of Mann ‘2017 in order to provide early detection of diabetic retinopathy . 07-21-aia AIA Claim (s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Roh ‘084 in view of Gurcan et al. (US 2016/0284084, hereinafter Gurcan ‘084) In re claim 3, Roh ‘084 teaches wherein training the neural network to identify tissue manipulations includes training the neural network to identify changes in contour of specific tissue types (0034, 0055) and wherein outputting surgical guidance includes outputting a warning when a change in contour for a tissue type identified (note when tissue shape are different, contour would be different) in the live video feed nears a damage threshold for the tissue type (0055, 0085, 0118, 0133, note too little bone surrounding the pilot hole is a change in contour). Furthermore, Gurcan ‘084 teaches configured to output a warning when a change in contour for a tissue type identified (0005, 0012-0014, 0046, 0047, 0052). It would have been prima facie obvious to one of ordinary skills in the art at the time of invention to modify the method/device of Roh ‘084 to include the features of Gurcan ‘084 in order to continuously monitor boundary of target tissue for a specific treatment requirements . 07-21-aia AIA Claim (s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Roh ‘084 in view of Semantic Segmentation (Semantic Segmentation using Fully Convolutional Networks over the years, Jun 1, 2017, http link see PTO592, hereinafter Semantic ‘2017 . In re claim 10, Roh ‘084 teaches using labeled surgical image frames for its neural network, but fails to teach wherein the neural network comprises a mask recurrent convolutional neural network and wherein training the neural network includes performing semi-supervised training of the mask recurrent neural network to detect patterns of the at least one of anatomical objects, surgical objects and tissue manipulations in the video frames in combination with supervised training of the mask recurrent convolutional neural network using labeled surgical image frames. Semantic ‘2017 teaches wherein the neural network comprises a mask recurrent convolutional neural network and wherein training the neural network includes performing semi-supervised training of the mask recurrent neural network to detect patterns of the at least one of anatomical objects, surgical objects and tissue manipulations in the video frames in combination with supervised training of the mask recurrent convolutional neural network using labeled surgical image frames (pages 12-13, Mask R-CNN, page 18-19). It would have been prima facie obvious to one of ordinary skills in the art at the time of invention to modify the method/device of Roh ‘084 to include the features of Semantic ‘2017 in order to use simple architecture and separate network for each task. Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to BO JOSEPH PENG whose telephone number is (571)270-1792. The examiner can normally be reached Monday thru Friday: 8:00 AM-5:00 PM EST. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ANNE M KOZAK can be reached at (571) 270-0552. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /BO JOSEPH PENG/Primary Examiner, Art Unit 3797 Application/Control Number: 19/047,522 Page 2 Art Unit: 3797 Application/Control Number: 19/047,522 Page 3 Art Unit: 3797 Application/Control Number: 19/047,522 Page 4 Art Unit: 3797 Application/Control Number: 19/047,522 Page 5 Art Unit: 3797 Application/Control Number: 19/047,522 Page 6 Art Unit: 3797 Application/Control Number: 19/047,522 Page 7 Art Unit: 3797 Application/Control Number: 19/047,522 Page 8 Art Unit: 3797 Application/Control Number: 19/047,522 Page 9 Art Unit: 3797 Application/Control Number: 19/047,522 Page 10 Art Unit: 3797 Application/Control Number: 19/047,522 Page 11 Art Unit: 3797 Application/Control Number: 19/047,522 Page 12 Art Unit: 3797 Application/Control Number: 19/047,522 Page 13 Art Unit: 3797
Prosecution Insights