Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Claims 1-20 are currently pending and under exam herein.
Claims 1-20 are rejected.
Priority
The instant application is a 371 of PCT US/2021/047770 filed on 8/26/2021 which claims priority from provisional application 63/070,698 filed on 08/26/2020. Thus, the effective filing date of the instant application is 08/26/2020.
Drawings
The Drawings filed on 02/24/2023 were considered.
Information Disclosure Statement
The application did not include an information disclosure statement.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “sufficient statistics model” in claims 1, 11, 16 and their dependent claims.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claims recite: (a) mathematical concepts, (e.g., mathematical relationships, formulas or equations, mathematical calculations); and (b) mental processes, i.e., concepts performed in the human mind, (e.g., observation, evaluation, judgement, opinion).
Subject matter eligibility evaluation in accordance with MPEP 2106:
Eligibility Step 1: Claims 1-20 are directed to a system and method of a surgical tool that uses a sensor and statistics to provide live feedback and assistance during surgery.
[Step 1: YES]
Eligibility Step 2A: First it is determined in Prong One whether a claim recites a judicial exception, and if
so, then it is determined in Prong Two whether the recited judicial exception is integrated into a
practical application of that exception.
Eligibility Step 2A Prong One: In determining whether a claim is directed to a judicial exception,
examination is performed that analyzes whether the claim recites a judicial exception, i.e., whether a
law of nature, natural phenomenon, or abstract idea is set forth or described in the claim.
Independent claim 1 recites the following steps which fall within the mental processes and/or mathematical concepts groupings of abstract ideas:
a feature extractor that generates a plurality of numerical features representing the time period from the sensor data (mathematical concept)
and a sufficient statistics model that generates a statistical parameter representing the plurality of sets of stored values, the statistical parameter being provided as part of the set of inputs (mathematical concept)
Dependent claim 6 recites the following steps which fall within the mental processes and/or mathematical concepts groupings of abstract ideas:
wherein the sufficient statistics model applies a wavelet decomposition to the plurality of sets of stored values to provide a set of wavelet coefficients, the set of inputs comprising a wavelet coefficient of the set of wavelet coefficients. (mathematical concept)
Dependent claim 7 recites the following steps which fall within the mental processes and/or mathematical concepts groupings of abstract ideas:
wherein the sufficient statistics model comprises a hidden Markov model that receives the plurality of sets of stored values as observations, the set of inputs comprising a probability value associated with the hidden Markov model (mathematical concept)
Dependent claim 8 recites the following steps which fall within the mental processes and/or mathematical concepts groupings of abstract ideas:
wherein the sufficient statistics model generates a cumulative sum likelihood from the plurality of sets of stored values (mathematical concept)
Dependent claim 9 recites the following steps which fall within the mental processes and/or mathematical concepts groupings of abstract ideas:
wherein the set of stored values represents only a set of time periods of the plurality of time periods that precede the time period (mathematical concept)
Dependent claim 10 recites the following steps which fall within the mental processes and/or mathematical concepts groupings of abstract ideas:
wherein the set of stored values represents all of the plurality of time periods (mathematical concept)
Independent claim 11 recites the following steps which fall within the mental processes and/or mathematical concepts groupings of abstract ideas:
generating a plurality of numerical features representing the time period from the sensor data (mathematical concept)
generating a statistical parameter representing a plurality of stored values from a memory at a sufficient statistics model (mathematical concept)
Dependent claim 14 recites the following steps which fall within the mental processes and/or mathematical concepts groupings of abstract ideas:
wherein storing a representation of the hidden layer of the recurrent neural network comprises: applying a transform to a set of values stored in the hidden layer to provide a set of transformed values (mathematical concept)
Independent claim 16 recites the following steps which fall within the mental processes and/or mathematical concepts groupings of abstract ideas:
and a sufficient statistics model that generates a statistical parameter representing the plurality of sets of stored values, the statistical parameter being provided as part of the set of inputs (mathematical concept)
Dependent claim 17 recites the following steps which fall within the mental processes and/or mathematical concepts groupings of abstract ideas:
wherein the sufficient statistics model generates a cumulative sum likelihood from the plurality of sets of stored values and applies a wavelet decomposition to the plurality of sets of stored values to provide a set of wavelet coefficients, the set of inputs comprising a wavelet coefficient of the set of wavelet coefficients and a value derived from the cumulative sum likelihood (mathematical concept)
Dependent claim 18 recites the following steps which fall within the mental processes and/or mathematical concepts groupings of abstract ideas:
wherein the sufficient statistics model further comprises a hidden Markov model that receives the plurality of sets of stored values as observations, the set of inputs further comprising a probability value associated with the hidden Markov model (mathematical concept)
The abstract ideas recited in the claims are evaluated under the broadest reasonable interpretation (BRI) of the claim limitations when read in light of and consistent with the specification. As noted in the foregoing section, the claims are determined to contain limitations that can practically be performed in the human mind with the aid of a pencil and paper, and therefore recite judicial exceptions from the mental process grouping of abstract ideas. Additionally, the recited limitations that are identified as judicial exceptions from the mathematical concepts grouping of abstract ideas are abstract ideas irrespective of whether or not the limitations are practical to perform in the human mind.
Therefore, claims 1-20 recite an abstract idea as the dependent claims will inherit the abstract ideas from the independent claims.
[Step 2A Prong One: YES]
Eligibility Step 2A Prong Two: In determining whether a claim is directed to a judicial exception, further
examination is performed that analyzes if the claim recites additional elements that when examined as a
whole integrates the judicial exception(s) into a practical application (MPEP 2106.04(d)). A claim that
integrates a judicial exception into a practical application will apply, rely on, or use the judicial exception
in a manner that imposes a meaningful limit on the judicial exception. The claimed additional elements
are analyzed to determine if the abstract idea is integrated into a practical application (MPEP
2106.04(d)(I); MPEP 2106.05(a-h)). If the claim contains no additional elements beyond the abstract
idea, the claim fails to integrate the abstract idea into a practical application (MPEP 2106.04(d)(III)).
The judicial exceptions identified in Eligibility Step 2A Prong One are not integrated into a practical application because of the reasons noted below.
The additional element in independent claim 1 includes:
A system comprising: a sensor positioned to monitor a surgical procedure on a patient, the surgical procedure comprising a plurality of surgical phases; a processor; and a non-transitory computer readable medium stores machine executable instructions for providing a surgical decision support system, the machine executable instructions being executed by the processor to provide
a sensor interface that receives sensor data from the sensor, the sensor data representing a time period of a plurality of time periods comprising the surgical procedure
a recurrent neural network, comprising a hidden layer, that receives a set of inputs and provides an output representing a surgical phase associated with the time period of the plurality of surgical phases, the set of inputs including the plurality of numerical features;
a memory that stores a representation of the hidden layer of the recurrent neural network as one of a plurality of sets of stored values
The additional element in dependent claim 2 includes:
wherein the at least one sensor comprises a camera that captures frame of video.
The additional element in dependent claim 3 includes:
wherein the feature extractor comprises a convolutional neural network.
The additional element in dependent claim 4 includes:
further comprising a network interface that provides the output representing the surgical phase associated with the time period to a surgical assisted decision making system.
The additional element in dependent claim 5 includes:
wherein the recurrent neural network is a long short term memory network.
The additional element in independent claim 11 includes:
A method comprising: receiving sensor data from the sensor, the sensor data representing a time period of a plurality of time periods comprising a surgical procedure;
providing an output, representing a surgical phase associated with the time period of the plurality of surgical phases, at a recurrent neural network from a set of inputs that includes the plurality of numerical features and the statistical parameter.
The additional element in dependent claim 12 includes:
further comprising storing a representation of a hidden layer of the recurrent neural network in the memory as one of the plurality stored values.
The additional element in dependent claim 13 includes:
wherein storing a representation of the hidden layer of the recurrent neural network comprises storing an output of the recurrent neural network in the memory.
The additional element in dependent claim 14 includes:
wherein storing a representation of the hidden layer of the recurrent neural network comprises: applying a transform to a set of values stored in the hidden layer to provide a set of transformed values;
and storing the set of transformed values in the memory.
The additional element in dependent claim 15 includes:
further comprising transmitting, at a network interface, a message to an individual at the facility in which the surgical procedure is performed to request an item of equipment in response to the output
The additional element in independent claim 16 includes:
A system comprising: a camera positioned to monitor a surgical procedure on a patient, the surgical procedure comprising a plurality of surgical phases;
a processor; and a non-transitory computer readable medium stores machine executable instructions for providing a surgical decision support system, the machine executable instructions being executed by the processor to provide:
a sensor interface that receives a frame of video from the camera, the frame of video representing a time period of a plurality of time periods comprising the surgical procedure;
a convolutional neural network that generates a plurality of numerical features representing the time period from the frame of video;
a long short term memory (LSTM) network, comprising a hidden layer, that receives a set of inputs and provides an output representing a surgical phase of the plurality of surgical phases associated with the time period, the set of inputs including the plurality of numerical features;
a memory that stores a representation of the hidden layer of the recurrent neural network as one of a plurality of sets of stored values, each of the plurality of sets of stored values representing one of the plurality of time periods;
The additional element in dependent claim 19 includes:
further comprising a user interface that provides the output representing the surgical phase to a human operator.
The additional element in dependent claim 20 includes:
further comprising a network interface that provides, in response to the output representing the surgical phase, a message to an individual at the facility in which the surgical procedure is performed to request an item of equipment
The additional elements of A system comprising: a sensor positioned to monitor a surgical procedure on a patient, the surgical procedure comprising a plurality of surgical phases (Claim 1), a sensor interface that receives sensor data from the sensor, the sensor data representing a time period of a plurality of time periods comprising the surgical procedure (Claim 1), wherein the at least one sensor comprises a camera that captures frame of video (Claim 2), wherein the feature extractor comprises a convolutional neural network (Claim 3), A method comprising: receiving sensor data from the sensor, the sensor data representing a time period of a plurality of time periods comprising a surgical procedure (Claim 11), providing an output, representing a surgical phase associated with the time period of the plurality of surgical phases, at a recurrent neural network from a set of inputs that includes the plurality of numerical features and the statistical parameter (Claim 11), a system comprising: a camera positioned to monitor a surgical procedure on a patient, the surgical procedure comprising a plurality of surgical phases (Claim 16), a convolutional neural network that generates a plurality of numerical features representing the time period from the frame of video (Claim 16), a long short term memory (LSTM) network, comprising a hidden layer, that receives a set of inputs and provides an output representing a surgical phase of the plurality of surgical phases associated with the time period, the set of inputs including the plurality of numerical features (Claim 16) are insignificant extra-solution activity that are part of the data gathering process used in the recited judicial exceptions (see MPEP 2106.05(g)).
The additional elements of a processor; and a non-transitory computer readable medium stores machine executable instructions for providing a surgical decision support system, the machine executable instructions being executed by the processor to provide (Claim 1), a recurrent neural network, comprising a hidden layer, that receives a set of inputs and provides an output representing a surgical phase associated with the time period of the plurality of surgical phases, the set of inputs including the plurality of numerical features (Claim 1), a memory that stores a representation of the hidden layer of the recurrent neural network as one of a plurality of sets of stored values (Claim 1), further comprising a network interface that provides the output representing the surgical phase associated with the time period to a surgical assisted decision making system (Claim 4), wherein the recurrent neural network is a long short term memory network (Claim 5), further comprising storing a representation of a hidden layer of the recurrent neural network in the memory as one of the plurality stored values (Claim 12), wherein storing a representation of the hidden layer of the recurrent neural network comprises storing an output of the recurrent neural network in the memory (Claim 13), and storing the set of transformed values in the memory (Claim 14), further comprising transmitting, at a network interface, a message to an individual at the facility in which the surgical procedure is performed to request an item of equipment in response to the output (Claim 15), a processor; and a non-transitory computer readable medium stores machine executable instructions for providing a surgical decision support system, the machine executable instructions being executed by the processor to provide (Claim 16), a sensor interface that receives a frame of video from the camera, the frame of video representing a time period of a plurality of time periods comprising the surgical procedure (Claim 16), a memory that stores a representation of the hidden layer of the recurrent neural network as one of a plurality of sets of stored values, each of the plurality of sets of stored values representing one of the plurality of time periods (Claim 16), further comprising a user interface that provides the output representing the surgical phase to a human operator (Claim 19), further comprising a network interface that provides, in response to the output representing the surgical phase, a message to an individual at the facility in which the surgical procedure is performed to request an item of equipment (Claim 20) fail to integrate a judicial exception into a practical application merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea, as discussed in MPEP § 2106.05(f).
Thus, the additionally recited elements merely invoke a computer as a tool, and/or amount to insignificant extra-solution data gathering activity, and as such, when all limitations in claims 1-20 have been considered as a whole, the claims are deemed to not recite any additional elements that would integrate a judicial exception into a practical application, and therefore claims 1-20 are directed to an abstract idea (MPEP 2106.04(d)).
[Step 2A Prong Two: NO]
Eligibility Step 2B: Because the claims recite an abstract idea, and do not integrate that abstract idea into a practical application, the claims are probed for a specific inventive concept. The judicial exception alone cannot provide that inventive concept or practical application (MPEP 2106.05). Identifying whether the additional elements beyond the abstract idea amount to such an inventive concept requires considering the additional elements individually and in combination to determine if they amount to significantly more than the judicial exception (MPEP 2106.05A i-vi).
The claims do not include any additional elements that are sufficient to amount to significantly more than the judicial exception(s) because of the reasons noted below.
The additional elements recited in claims 1-20 are identified above, and carried over from Step 2A: Prong Two along with their conclusions for analysis at Step 2B. Any additional element or combination of elements that was considered to be insignificant extra-solution activity at Step 2A: Prong Two was re-evaluated at Step 2B, because if such re-evaluation finds that the element is unconventional or otherwise more than what is well-understood, routine, conventional activity in the field, this finding may indicate that the additional element is no longer considered to be insignificant; and all additional elements and combination of elements were evaluated to determine whether any additional elements or combination of elements are other than what is well-understood, routine, conventional activity in the field, or simply append well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception, per MPEP 2106.05(d).
The additional elements of a system comprising: a sensor positioned to monitor a surgical procedure on a patient, the surgical procedure comprising a plurality of surgical phases (Claim 1), a sensor interface that receives sensor data from the sensor, the sensor data representing a time period of a plurality of time periods comprising the surgical procedure (Claim 1), wherein the at least one sensor comprises a camera that captures frame of video (Claim 2), a method comprising: receiving sensor data from the sensor, the sensor data representing a time period of a plurality of time periods comprising a surgical procedure (Claim 11), a system comprising: a camera positioned to monitor a surgical procedure on a patient, the surgical procedure comprising a plurality of surgical phases (Claim 16) are conventional and part of the data gathering process used in the recited judicial exceptions (see MPEP 2106.05(g)). Evidence for conventionality is shown by applicants own provisional application (63/070,698) in the appendix to the specification. Applicant admits cameras integrated with surgical tools have been done previously and lists several examples of machine learning software integrated into the camera containing surgical tools. Applicant then discusses their contribution to the field is the specific implication of the neural network architecture not the gathering of data with a conventional camera (pgs. 1-2).
The additional elements of wherein the feature extractor comprises a convolutional neural network (Claim 3), providing an output, representing a surgical phase associated with the time period of the plurality of surgical phases, at a recurrent neural network from a set of inputs that includes the plurality of numerical features and the statistical parameter (Claim 11), a convolutional neural network that generates a plurality of numerical features representing the time period from the frame of video (Claim 16), a long short term memory (LSTM) network, comprising a hidden layer, that receives a set of inputs and provides an output representing a surgical phase of the plurality of surgical phases associated with the time period, the set of inputs including the plurality of numerical features (Claim 16) are conventional and part of the data gathering process used in the recited judicial exceptions (see MPEP 2106.05(g)). Examiner asserts that LSTM and CNN in the instant application are being used to gather data for the statistical model. This is a conventional method of data gathering. Evidence for conventionality is shown by Herath et al (Herath et al, arxiv, Going Deeper into Action Recognition: A Survey, February 1, 2017) which discusses is a review and discusses how a combination LSTM-CNN in order to analyze videos can improve results (pg. 17, paragraph 5)
The additional elements of a processor; and a non-transitory computer readable medium stores machine executable instructions for providing a surgical decision support system, the machine executable instructions being executed by the processor to provide (Claim 1), a recurrent neural network, comprising a hidden layer, that receives a set of inputs and provides an output representing a surgical phase associated with the time period of the plurality of surgical phases, the set of inputs including the plurality of numerical features (Claim 1), a memory that stores a representation of the hidden layer of the recurrent neural network as one of a plurality of sets of stored values (Claim 1), further comprising a network interface that provides the output representing the surgical phase associated with the time period to a surgical assisted decision making system (Claim 4), wherein the recurrent neural network is a long short term memory network (Claim 5), further comprising storing a representation of a hidden layer of the recurrent neural network in the memory as one of the plurality stored values (Claim 12), wherein storing a representation of the hidden layer of the recurrent neural network comprises storing an output of the recurrent neural network in the memory (Claim 13), and storing the set of transformed values in the memory (Claim 14), further comprising transmitting, at a network interface, a message to an individual at the facility in which the surgical procedure is performed to request an item of equipment in response to the output (Claim 15), a processor; and a non-transitory computer readable medium stores machine executable instructions for providing a surgical decision support system, the machine executable instructions being executed by the processor to provide (Claim 16), a sensor interface that receives a frame of video from the camera, the frame of video representing a time period of a plurality of time periods comprising the surgical procedure (Claim 16), a memory that stores a representation of the hidden layer of the recurrent neural network as one of a plurality of sets of stored values, each of the plurality of sets of stored values representing one of the plurality of time periods (Claim 16), further comprising a user interface that provides the output representing the surgical phase to a human operator (Claim 19), further comprising a network interface that provides, in response to the output representing the surgical phase, a message to an individual at the facility in which the surgical procedure is performed to request an item of equipment (Claim 20) are conventional fail to integrate a judicial exception into a practical application merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea, as discussed in MPEP § 2106.05(f). In addition, MPEP 21.06.05 (d) states ii. Performing repetitive calculations, Flook, 437 U.S. at 594, 198 USPQ2d at 199 (recomputing or readjusting alarm limit values); Bancorp Services v. Sun Life, 687 F.3d 1266, 1278, 103 USPQ2d 1425, 1433 (Fed. Cir. 2012) ("The computer required by some of Bancorp’s claims is employed only for its most basic function, the performance of repetitive calculations, and as such does not impose meaningful limits on the scope of those claims."); and receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) are conventional.
When taken alone, all additional elements in claims 1-20 do not amount to significantly more than the above-identified judicial exception(s). Even when evaluated as a combination, the additional elements fail to transform the exception(s) into a patent-eligible application of that exception. Thus, claims 1-20 are deemed to not contribute an inventive concept, i.e., amount to significantly more than the judicial exception(s) (MPEP 2106.05(II)).
[Step 2B: NO]
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-5, 7,9-16, 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Twinanda et al. (Twinanda et al. EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos. IEEE Transactions on Medical Imaging 2017, 36 (1), 86–97.) in further view of Bodenstedt et al. (Bodenstedt et al, Unsupervised temporal context learning using convolutional neural networks for laparoscopic workflow analysis, arxiv, 2/17/2017) in further view of DiPietro et al. (DiPietro et al, Recognizing Surgical Activities with Recurrent Neural Networks, arxiv, 6/22/2026). The italicized text corresponds to the instant claim limitations.
With respect to the limitations of Claims 1, 5, 11, 12, 13, 14, 16, DiPietro et al. teaches a recurrent neural network (RNNs), and in particular long short-term memory (LSTM), to map kinematics to labels. Rather than operating only on local neighborhoods in time, LSTM maintains a memory cell and learns when to write to memory, when to reset memory, and when to read from memory, forming unaries that in principle depend on all inputs. In fact, we will rely only on these unary terms, or in other words assume that labels are independent given the sequence of kinematics. Despite this, we will see that predicted labels are smooth over time with no post-processing. Further, using a single model and a single set of hyperparameters, we match state-of-the-art performance for gesture recognition and improve over state-of-the-art performance for maneuver recognition, in terms of both accuracy and edit distance. The LSTM retains a memory of the video over time contributing to the overall prediction. (pg. 2, paragraph 1, a recurrent neural network, comprising a hidden layer, that receives a set of inputs and provides an output representing a surgical phase associated with the time period of the plurality of surgical phases, the set of inputs including the plurality of numerical features; a memory that stores a representation of the hidden layer of the recurrent neural network as one of a plurality of sets of stored values; (Claim 1), wherein the recurrent neural network is a long short term memory network (Claim 5) providing an output, representing a surgical phase associated with the time period of the plurality of surgical phases, at a recurrent neural network from a set of inputs that includes the plurality of numerical features and the statistical parameter. (Claim 11) storing a representation of a hidden layer of the recurrent neural network in the memory as one of the plurality stored values. (Claim 12), wherein storing a representation of the hidden layer of the recurrent neural network comprises storing an output of the recurrent neural network in the memory (Claim 13), wherein storing a representation of the hidden layer of the recurrent neural network comprises: applying a transform to a set of values stored in the hidden layer to provide a set of transformed values; and storing the set of transformed values in the memory (Claim 14), a long short term memory (LSTM) network, comprising a hidden layer, that receives a set of inputs and provides an output representing a surgical phase of the plurality of surgical phases associated with the time period, the set of inputs including the plurality of numerical features; a memory that stores a representation of the hidden layer of the recurrent neural network as one of a plurality of sets of stored values, each of the plurality of sets of stored values representing one of the plurality of time periods (Claim 16
With respect to the limitations of Claims 9, 10, DiPietro et al. teaches RNNs traditionally propagate information forward in time, forming predictions using only past and present inputs. Bidirectional RNNs can improve performance when operating offline by using future inputs as well. This essentially consists of running one RNN in the forward direction and one RNN in the backward direction, concatenating hidden states, and computing outputs jointly (pg. 3, paragraph 4, wherein the set of stored values represents only a set of time periods of the plurality of time periods that precede the time period (Claim 9) wherein the set of stored values represents all of the plurality of time periods (Claim 10)
DiPietro et al. does not explicitly teach
a sensor positioned to monitor a surgical procedure on a patient, the surgical procedure comprising a plurality of surgical phases (Claim 1))
a processor; and a non-transitory computer readable medium stores machine executable instructions for providing a surgical decision support system, the machine executable instructions being executed by the processor to provide: (Claim 1)
a feature extractor that generates a plurality of numerical features representing the time period from the sensor data (Claim 1)
a sensor interface that receives sensor data from the sensor, the sensor data representing a time period of a plurality of time periods comprising the surgical procedure (Claim 1)
wherein the at least one sensor comprises a camera that captures frame of video. (Claim 2)
wherein the feature extractor comprises a convolutional neural network. (Claim 3)
further comprising a network interface that provides the output representing the surgical phase associated with the time period to a surgical assisted decision making system (Claim 4),
generating a plurality of numerical features representing the time period from the sensor data (claim 11)
and a sufficient statistics model that generates a statistical parameter representing the plurality of sets of stored values, the statistical parameter being provided as part of the set of inputs (Claim 1),
receiving sensor data from the sensor, the sensor data representing a time period of a plurality of time periods comprising a surgical procedure (Claim 11)
a convolutional neural network that generates a plurality of numerical features representing the time period from the frame of video (Claim 16)
A system comprising: a camera positioned to monitor a surgical procedure on a patient, the surgical procedure comprising a plurality of surgical phases; a processor; and a non-transitory computer readable medium stores machine executable instructions for providing a surgical decision support system, the machine executable instructions being executed by the processor to provide: a sensor interface that receives a frame of video from the camera, the frame of video representing a time period of a plurality of time periods comprising the surgical procedure (Claim 16)
further comprising transmitting, at a network interface, a message to an individual at the facility in which the surgical procedure is performed to request an item of equipment in response to the output (Claim 15),
generating a statistical parameter representing a plurality of stored values from a memory at a sufficient statistics model (Claim 11), );
sufficient statistics model that generates a statistical parameter representing the plurality of sets of stored values, the statistical parameter being provided as part of the set of inputs (Claim 16)
further comprising a user interface that provides the output representing the surgical phase to a human operator (Claim 19),
further comprising a network interface that provides, in response to the output representing the surgical phase, a message to an individual at the facility in which the surgical procedure is performed to request an item of equipment (Claim 20).
With respect to the limitations of Claims 1, 2, 11, 16, Twinanda et al. teaches the camera in laparoscopic procedures is not static, resulting in motion blur and high variability of the observed scenes along the surgery (pg. 86, col. 2, paragraph 4, a sensor positioned to monitor a surgical procedure on a patient, the surgical procedure comprising a plurality of surgical phases (Claim 1)) Twinanda et al. teaches also teaches videos are captured at 25 fps and to reduce redundancy, we downsample the videos to 1 fps by taking the first frame from every 25 frames. In a preliminary study, we also experimented with sequences downsampled to 5 fps, but did not obtain any improvement. (pg. 89, col 2, paragraph 3 – pg. 90 col. 1, paragraph 1, a sensor interface that receives sensor data from the sensor, the sensor data representing a time period of a plurality of time periods comprising the surgical procedure (Claim 1) wherein the at least one sensor comprises a camera that captures frame of video. (Claim 2) receiving sensor data from the sensor, the sensor data representing a time period of a plurality of time periods comprising a surgical procedure (Claim 11), A system comprising: a camera positioned to monitor a surgical procedure on a patient, the surgical procedure comprising a plurality of surgical phases; a processor; and a non-transitory computer readable medium stores machine executable instructions for providing a surgical decision support system, the machine executable instructions being executed by the processor to provide: a sensor interface that receives a frame of video from the camera, the frame of video representing a time period of a plurality of time periods comprising the surgical procedure (Claim 16)
With respect to the limitations of Claims 1, 7, 8, 11, 16, Twinanda et al. teaches the use of an extension of HMM, namely a two-level Hierarchical hidden Markov model (HHMM). The top-level contains nodes that model the inter-phase dependencies, while the bottom-level nodes model the intra-phase dependencies. This is a sufficient statistics model according to applicant’s own provisional application where HMM models are considered sufficient. (pg. 89, col. 2, paragraph 1, and a sufficient statistics model that generates a statistical parameter representing the plurality of sets of stored values, the statistical parameter being provided as part of the set of inputs (Claim 1),wherein the sufficient statistics model comprises a hidden Markov model that receives the plurality of sets of stored values as observations, the set of inputs comprising a probability value associated with the hidden Markov model. (Claim 7), wherein the sufficient statistics model further comprises a hidden Markov model that receives the plurality of sets of stored values as observations, the set of inputs further comprising a probability value associated with the hidden Markov model (Claim 8), generating a statistical parameter representing a plurality of stored values from a memory at a sufficient statistics model (Claim 11),; and a sufficient statistics model that generates a statistical parameter representing the plurality of sets of stored values, the statistical parameter being provided as part of the set of inputs (Claim 16)
With respect to the limitations of Claims 1, 3, 11, 16, Bodenstedt et al. teaches that the aim of a computer-assisted surgery system (CAS) is to provide the surgeon with the right type of assistance at the right moment. In laparoscopic surgery, such a system could be used to compensate for some of the drawbacks typical to laparoscopy and presenting an unsupervised method for training a convolutional neural network (CNN) to differentiate between laparoscopic video frames on a temporal basis. (abstract, a processor; and a non-transitory computer readable medium stores machine executable instructions for providing a surgical decision support system, the machine executable instructions being executed by the processor to provide: (Claim 1) a feature extractor that generates a plurality of numerical features representing the time period from the sensor data (Claim 1) wherein the feature extractor comprises a convolutional neural network. (Claim 3) generating a plurality of numerical features representing the time period from the sensor data (claim 11) a convolutional neural network that generates a plurality of numerical features representing the time period from the frame of video (Claim 16)
With respect to the limitations of Claims 4, 15, 19, 20 , Bodenstedt et al. teaches computer-assisted surgery (CAS) aims to display the location of a tumor at the appropriate time or suggesting what instruments to prepare next, analyzing the surgical workflow is a prerequisite. Since laparoscopic interventions are performed via endoscope, the video signal is an obvious sensor modality to rely on for workflow analysis. Suggesting instruments is part of a surgical decision making system. This system inherently has the monitoring of surgical phases to determine the instruments needed to prepare. Additionally, transmitting any of this information wirelessly is an obvious variant. (abstract, further comprising a network interface that provides the output representing the surgical phase associated with the time period to a surgical assisted decision making system (Claim 4), further comprising transmitting, at a network interface, a message to an individual at the facility in which the surgical procedure is performed to request an item of equipment in response to the output (Claim 15), further comprising a user interface that provides the output representing the surgical phase to a human operator (Claim 19), further comprising a network interface that provides, in response to the output representing the surgical phase, a message to an individual at the facility in which the surgical procedure is performed to request an item of equipment (Claim 20).
A person of ordinary skill in the art would be motivated to modify Twinanda et al. with Bodenstedt et al. with DiPietro et al. are all in the same field of endeavor and address the same problem of video/ image analysis during surgery and automated surgical workflows. Twinanda et al. even expressly suggests adding LSTM architecture and discusses how limitations could be solved by using long short term memory (LSTM) architectures. Such an approach will form part of future efforts to improve phase recognition. (pg. 97, col. 1, paragraph 1). A person of ordinary skill in the art would be motivated to combine the LSTM architecture from both Bodenstedt et al. and DiPietro et al. with Twinanda et al. There is a reasonable expectation of success because the methods of data manipulation in neural network architectures are just being changed in known and predictable ways based on the prior art of Bodenstedt et al. and DiPietro et al. and Twinanda et al.
Claims 6, 8, 17 are rejected under 35 U.S.C. 103 as being unpatentable over Twinanda et al. in view of Bodenstedt et al. in further view of DiPietro et al. as applied to claims 1-5, 7,9-16, 18-20 above in further view of Callegari et al. (Callegari, C.; Giordano, S.; Pagano, M.; Pepe, T. WAVE-CUSUM: Improving CUSUM Performance in Network Anomaly Detection by Means of Wavelet Analysis. Computers & Security 2012, 31 (5), 727–735.) The italicized text corresponds to the instant claim limitations.
The limitations of claims 1-5, 7,9-16, 18-20 have been taught by Twinanda et al. in view of Bodenstedt et al. in further view of DiPietro et al. above.
Twinanda et al. in view of Bodenstedt et al. in further view of DiPietro et al. does not explicitly teach
wherein the sufficient statistics model applies a wavelet decomposition to the plurality of sets of stored values to provide a set of wavelet coefficients, the set of inputs comprising a wavelet coefficient of the set of wavelet coefficients (Claim 6)
wherein the sufficient statistics model generates a cumulative sum likelihood from the plurality of sets of stored values (Claim 8),
wherein the sufficient statistics model generates a cumulative sum likelihood from the plurality of sets of stored values and applies a wavelet decomposition to the plurality of sets of stored values to provide a set of wavelet coefficients, the set of inputs comprising a wavelet coefficient of the set of wavelet coefficients and a value derived from the cumulative sum likelihood (Claim 17)
However, these limitations were known in the art at the time of the effective filing date of the invention, as taught by Callegari et al.
With respect to the limitations of Claims 6, 8, 17, Callegari et al. teaches the use of wavelet decomposition combined cumulative sum for anomaly detection using both as model inputs. (pg. 730, Figure 2, wherein the sufficient statistics model applies a wavelet decomposition to the plurality of sets of stored values to provide a set of wavelet coefficients, the set of inputs comprising a wavelet coefficient of the set of wavelet coefficients (Claim 6) wherein the sufficient statistics model generates a cumulative sum likelihood from the plurality of sets of stored values (Claim 8), wherein the sufficient statistics model generates a cumulative sum likelihood from the plurality of sets of stored values and applies a wavelet decomposition to the plurality of sets of stored values to provide a set of wavelet coefficients, the set of inputs comprising a wavelet coefficient of the set of wavelet coefficients and a value derived from the cumulative sum likelihood (Claim 17)
A person of ordinary skill in the art would be motivated to use the method taught by Callegari et al. of a constant monitoring of data to determine anomalies with the surgical video analysis taught by Twinanda et al. in view of Bodenstedt et al. in further view of DiPietro et al. as video is a continuous stream of data and a person of ordinary skill in the art would understand methods that detect anomalies could be useful in a surgical context where doctors frequently remove anomalies. There is a reasonable expectation of success because the methods of data manipulation in a neural network architecture is just being changed in known and predictable ways based on the well-known method of Callegari et al.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Connor Beveridge whose telephone number is 571-272-2099. The examiner can normally be reached Monday - Thursday 9 am - 5 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Karlheinz Skowronek can be reached at 571-272-9047. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.H.B./Examiner, Art Unit 1687
/Karlheinz R. Skowronek/Supervisory Patent Examiner, Art Unit 1687
Prosecution Insights