Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The disclosure is objected to because of the following informalities:
Paragraph [0076] current states “Note that in terms of surgical action, the first type of partial activation includes the moment when the two jaws of the energy tool close on a tissue, i.e., close jaws 106 in FIG. 1; whereas the second type of partial activation includes the moment when the two jaws of the energy tool open up to release the tissue, i.e., close jaws 112 in FIG. 1.”. The Examiner notes that 112 in Fig. 1 corresponds to “Open Jaws” not close jaws. Appropriate correction is required.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 22-41 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-11 and 14 of U.S. Patent No. US 12,340,310. Although the claims at issue are not identical, they are not patentably distinct from each other.
Regarding Claim 22, please see the table below illustrating the shared limitations:
Instant claim 22 from 19/245,138
Reference claim 1 from U.S. Patent No. 12,340,310
Claim 22: A computer-implemented method, comprising: receiving a surgical video of a surgical procedure involving energy tool activations; applying a sequence of sampling windows to the surgical video to generate a sequence of windowed samples of the surgical video; for each windowed sample in the sequence of windowed samples, applying a deep- learning model to a sequence of video frames within the windowed sample to generate an activation/non-activation/partial-activation inference, thereby generating a sequence of activation/non-activation/partial-activation inferences;
Claim 1: A computer-implemented method, comprising: receiving a surgical video of a surgical procedure involving energy tool activations; applying a sequence of sampling windows to the surgical video to generate a sequence of windowed samples of the surgical video; for each windowed sample in the sequence of windowed samples, applying a deep- learning model to a sequence of video frames within the windowed sample to generate an activation/non-activation inference activation/non-activation inferences
and identifying a sequence of activation events based on the sequence of activation/non- activation/partial-activation inferences, wherein identifying an activation event in the sequence of activation events comprises:
and identifying a sequence of activation events based on the sequence of activation/non- activation inferences
identifying each instance of either or ii) multiple consecutive activation or partial-activation inferences located between two non-activation inferences, in the sequence of activation/non-activation/partial-activation inferences, as an identified activation event. (The Examiner notes that the language “either…or” allows for the interpretation of the claim limitation being associated with point i) or point ii).)
identifying multiple consecutive activation inferences located between two non- activation inferences, in the sequence of activation/non-activation inferences, as a single activation;
Regarding the system claim 34 corresponding to instant claim 22, see the similar system claim 14 from U.S. Patent No. 12,340,310.
Regarding Claim 23, please see the table below illustrating the shared limitations:
Instant claim 23 from 19/245,138
Reference claim 2 from U.S. Patent No. 12,340,310
Claim 23: The computer-implemented method of claim 22, wherein the method further comprises generating a total activation count for the surgical video by: incrementing an activation count by one in response to the multiple consecutive activation or partial-activation inferences; and outputting a final-updated activation count as the total activation count for the surgical video after processing the sequence of activation/non-activation/partial-activation inferences.
Claim 2: The computer-implemented method of claim 1,wherein the method further comprises generating a total activation count for the surgical video by: incrementing an activation count by one in response to the multiple consecutive activation inferences; and outputting the a final-updated activation count as the total activation count for the surgical video after processing the sequence of activation/non-activation inferences.
Regarding Claim 24, please see the table below illustrating the shared limitations:
Instant claim 24 from 19/245,138
Reference claim 1 from U.S. Patent No. 12,340,310
Claim 24: The computer-implemented method of claim 22 wherein the multiple consecutive activation or partial-activation inferences include multiple consecutive activation inferences, the method further comprising estimating a duration of the identified activation event by: identifying the first and the last inferences in the multiple consecutive activation inferences corresponding to two partial-activation windowed samples that partially overlap with the identified activation event; determining first and second amounts of partial-overlap between the two partial- activation windowed samples, respectively, and the identified activation event; and computing the duration of the identified activation event as a sum of i) the first and second amounts of partial-overlap and ii) full overlaps with the identified activation event of other windowed samples that are between the two partial-activation windowed samples and are associated with the multiple consecutive activation inferences. (The Examiner notes that claim 24 inherits all of the limitations of claim 22, which correspond to the crossed-out limitations shown in reference claim 1.)
Claim 1:
Regarding Claim 25, please see the table below illustrating the shared limitations:
Instant claim 25 from 19/245,138
Reference claim 3 from U.S. Patent No. 12,340,310
Claim 25: The computer-implemented method of claim 24, wherein determining the first and second amounts of partial-overlap includes multiplying a window length of the windowed sample with the first or the last inference.
Claim 3: The computer-implemented method of claim 1 wherein determining the first and second amounts of partial-overlap includes multiplying a window length of the sampling windowed sample
Regarding Claim 26, please see the table below illustrating the shared limitations:
Instant claim 26 from 19/245,138
Reference claim 4 from U.S. Patent No. 12,340,310
Claim 26: The computer-implemented method of claim 22, wherein the sequence of sampling windows has a common window length determined based on an activation duration distribution of a number of previously-identified activation events from a plurality of surgical videos of the surgical procedure.
Claim 4: The computer-implemented method of claim 1, wherein the sequence of sampling windows has a common window length determined based on an activation duration distribution of a large number of previously-identified activation events from a plurality of surgical videos of the surgical procedure.
System claim 41 from the instant application is a broader version of instant claim 26 and is similarly rejected.
Regarding Claim 27, please see the table below illustrating the shared limitations:
Instant claim 27 from 19/245,138
Reference claim 5 from U.S. Patent No. 12,340,310
Claim 27: The computer-implemented method of claim 22, wherein applying the sequence of sampling windows includes adding a predetermined amount of overlap between consecutive sampling windows.
Claim 5: The computer-implemented method of claim 1, wherein applying the sequence of sampling windows includes adding a predetermined amount of overlap between consecutive sampling windows.
System claim 35 from the instant application corresponds to claim 27 from the instant application and is similarly rejected.
Regarding Claim 28, please see the table below illustrating the shared limitations:
Instant claim 28 from 19/245,138
Reference claim 6 from U.S. Patent No. 12,340,310
Claim 28: The computer-implemented method of claim 22, wherein the method further comprises training the deep-learning model by: receiving a group of annotated surgical videos of the surgical procedure, wherein each annotated surgical video in the group of annotated surgical videos includes a set of identified activation events, wherein each identified activation event is annotated with a starting timestamp and an end timestamp; for each annotated surgical video, generating a set of labeled training data by sampling the annotated surgical video; adding the set of labeled training data into a training dataset; and training the deep-learning model using the training dataset.
Claim 6: The computer-implemented method of claim 1, wherein the method further comprises training the deep-learning model by: receiving a group of annotated surgical videos of the surgical procedure, wherein each annotated surgical video in the group of annotated surgical videos includes a set of identified activation events, wherein each identified activation event is annotated with a starting timestamp and an end timestamp; for each annotated surgical video in the group of annotated surgical videos, generating a set of labeled training data by sampling the annotated surgical video; adding the set of labeled training data into a training dataset; and training the deep-learning model using the training dataset.
Regarding Claim 29, please see the table below illustrating the shared limitations:
Instant claim 29 from 19/245,138
Reference claim 7 from U.S. Patent No. 12,340,310
Claim 29: The computer-implemented method of claim 28, wherein generating the set of labeled training data by sampling the annotated surgical video includes: sequentially applying a sequence of sampling windows to the annotated surgical video to generate a sequence of windowed samples of the annotated surgical video; and for each windowed sample in the sequence of windowed samples, acquiring a ground truth label for the windowed sample based on the temporal location of the windowed sample with respect to the set of annotated activation events in the annotated surgical video; and adding the labeled windowed sample into the set of labeled training data.
Claim 7: The computer-implemented method of claim 6, wherein generating the set of labeled training data by sampling the annotated surgical video includes: sequentially applying a sequence of sampling windows to the annotated surgical video to generate a sequence of windowed samples of the annotated surgical video; and for each windowed sample in the sequence of windowed samples, acquiring a ground truth label for the windowed sample based on the temporal location of the windowed sample with respect to the set of annotated activation events in the annotated surgical video; and adding the labeled windowed sample into the set of labeled training data.
Regarding Claim 30, please see the table below illustrating the shared limitations:
Instant claim 30 from 19/245,138
Reference claim 7 from U.S. Patent No. 12,340,310
Claim 30: The computer-implemented method of claim 29, wherein acquiring the ground truth label for the windowed sample based on the temporal location of the windowed sample includes: providing a first integer label of "1" to the windowed sample if the windowed sample is situated entirely inside an annotation activation event within the set of annotated activation events; and providing a second integer label of "0" to the windowed sample if the windowed sample is situated entirely outside of any of the set of annotated activation events.
Claim 8: The computer-implemented method of claim 7, wherein acquiring the ground truth label for the windowed sample based on the temporal location of the windowed sample includes: providing a first integer label of "1" to the windowed sample if the windowed sample is situated entirely inside an annotation activation event within the set of annotated activation events; and providing a second integer label of "0" to the windowed sample if the windowed sample is situated entirely outside of any of the set of annotated activation events.
Regarding Claim 31, please see the table below illustrating the shared limitations:
Instant claim 31 from 19/245,138
Reference claim 9 from U.S. Patent No. 12,340,310
Claim 31: The computer-implemented method of claim 30, wherein acquiring the ground truth label for the windowed sample further comprises: providing a float number label between "0" and "1" to the windowed sample if the windowed sample partially overlaps with an annotated activation event within the set of annotated activation events, wherein the float number label is computed based on the percentage of the windowed sample positioned inside the identified activation event.
Claim 9: The computer-implemented method of claim 8, wherein acquiring the ground truth label for the windowed sample further comprises: providing a float number label between "0" and "1" to the windowed sample if the windowed sample partially overlaps with an annotated activation event within the set annotated activation events, wherein the float number label is computed based on the percentage of the windowed sample positioned inside the identified activation event.
System claims 37 and 39-40 from the instant application are a broader version of instant claim 31 and are similarly rejected.
Regarding Claim 32, please see the table below illustrating the shared limitations:
Instant claim 32 from 19/245,138
Reference claim 10 from U.S. Patent No. 12,340,310
Claim 32: The computer-implemented method of claim 31, wherein the method further comprises: providing a negative sign to the float number label assigned to the windowed sample if the windowed sample overlaps with the beginning portion of the annotated activation event; and providing a positive sign to the float number label assigned to the windowed sample if the windowed sample overlaps with the ending portion of the annotated activation event.
Claim 10: The computer-implemented method of claim 9, wherein the method further comprises: providing a negative sign to the float number label assigned to the windowed sample if the windowed sample overlaps with the beginning portion of the annotated activation event; and providing a positive sign to the float number label assigned to the windowed sample if the windowed sample overlaps with the ending portion of the annotated activation event.
System claim 38 from the instant application is a broader version of instant claim 32 and is similarly rejected.
Regarding Claim 33, please see the table below illustrating the shared limitations:
Instant claim 33 from 19/245,138
Reference claim 11 from U.S. Patent No. 12,340,310
Claim 33: The computer-implemented method of claim 31, wherein the method further comprises: determining whether a center video frame within the windowed sample is inside the annotated activation event; and in response to determining that the center video frame is outside of the annotated activation event, excluding the windowed sample from the training dataset.
Claim 11: The computer-implemented method of claim 9, wherein the method further comprises: determining whether the center video frame within the windowed sample is inside the annotated activation event; and in response to determining that the center video frame is outside of the annotated activation event, excluding the windowed sample from the training dataset.
Regarding Claim 36, please see the table below illustrating the shared limitations:
Instant claim 36 from 19/245,138
Reference claims 8-9 from U.S. Patent No. 12,340,310
Claim 36: The system of claim 34 wherein an activation inference comprises a first integer, a non-activation inference comprises a second integer different than the first integer, and a partial-activation inference comprises a float number having a value between the first integer and the second integer. (The Examiner notes that the instant claim 36 is a broader version of reference claims 8 and 9, where reference claim 9 inherits the limitation of claim 8)
Claim 8: The computer-implemented method of claim 7, wherein acquiring the ground truth label for the windowed sample based on the temporal location of the windowed sample includes: providing a first integer label of "1" to the windowed sample if the windowed sample is situated entirely inside an annotation activation event within the set of annotated activation events; and providing a second integer label of "0" to the windowed sample if the windowed sample is situated entirely outside of any of the set of annotated activation events.
Claim 9: The computer-implemented method of claim 8, wherein acquiring the ground truth label for the windowed sample further comprises: providing a float number label between "0" and "1" to the windowed sample if the windowed sample partially overlaps with an annotated activation event within the set annotated activation events, wherein the float number label is computed based on the percentage of the windowed sample positioned inside the identified activation event.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 22-41 are rejected under 35 U.S.C. 101 because the claimed invention is
directed to an abstract idea without reciting elements that amount to significant more than
the abstract idea. The rationale for this rejection, under MPEP § 2106, for this finding is
explained below.
Step 1: Under step 1, the claims are analyzed to determine if the claim is directed to a
process, machine, article of manufacture, or composition of matter. For the claims in question,
claims 22-33 are directed towards a method and claims 34-41 are directed towards a machine.
Step 2A, Prong 1: Under step 2A, prong 1, the claims are evaluated to determine if the
claim recites a judicial exception, which includes the laws of nature, physical phenomena, or an
abstract idea. For independent claim 1 (and corresponding independent claim 34), the limitations regarding generating an activation/non-activation/partial activation inference and consequent sequence of activations, and using that information to identify a sequence of activation events based on identifying each instance of an activation inference or multiple consequent activation inferences between two non-activation inferences, is directed towards a mental process. For example, an individual watching a surgical video can reasonably identify surgical tool activation events and consequently identify/count instances of surgical tool activations, where both counting each instance of an activation sequence or counting the activation sequence between two non-activation sequences can both be performed mentally.
Step 2A, Prong 2: Under step 2A, prong 2, the claims are evaluated to determine
whether the claim as a whole integrates the recited judicial exception into a practical application
of the exception (see MPEP 2106.04(d)). The examiner notes that MPEP 2106.05(a) -(c) and (e)
generally concern limitations that are indicative of integration, whereas 2106.05(f)-(h) generally
concern limitations that are not indicative of integration.
In regards to independent claims 22 and 34, the limitations as described above are directed towards a mental process. The additional limitations regarding using a processor and memory, as well as receiving a surgical video and applying a sequence of sampling videos are recited at a high level of generality and are considered to be extra solution activities and/or are generic computing components, and do not constitute integration into a practical application nor significantly more.
In regards to dependent claims 22-33, the additional limitations are broadly recited and further disclose steps used to perform the judicial exception with any clear indication or detail which would indicate integration into a practical application as noted in MPEP 2106.05(a) or MPEP 2106.05(e). Therefore, the additional limitations of claims 2-6, 8-12, and 14-18 do not constitute integration into a practical application.
The examiner emphasizes MPEP 2106.05(a), which states that a limitation is indicative of integration into a practical application if the limitation identifies a manner in which an improvement is explicitly and specifically achieved and recited in the claims. The current claim language all are recited at a high level of generality which do not serve to integrate the limitations in view of MPEP 2106.05(f), and furthermore nothing precludes the current limitations from being interpreted under the mental processes grouping.
Furthermore, the examiner encourages the applicant to review the analysis of claim 2, example 47, of the most recent Patent Eligibility Guidance from July 17, 2024 (https://www.uspto.gov/patents/laws/examination-policy/subject-matter-eligibility, titled “2024 AI Examples 47 through 49”), wherein using an artificial neural network trained on input data to detect anomalies was deemed ineligible due to steps (b), (d), and (e) of claim 2 being related to a mental process. In regards to the instant claims of this application, they are directed towards what is otherwise/historically a mentally performed process of estimating player locations in a broadcast video, which has merely been made automatic by the use of a machine learning model, which cannot in itself serve to integrate at step 2A, prong 2, as an inventive concept (see MPEP 2106.05(f)).
Step 2B: Under step 2B, the claims are evaluated as a whole to determine if it amounts to
significantly more than the recited exception (i.e., whether any additional element, or
combination of additional elements, adds an inventive concept to the claim). The considerations
of step 2A, prong 2 and step 2B overlap, but differ in that 2B also requires considering the claim
as a whole/combination of limitations, and with reference to MPEP 2106.05(d) whether the
claims feature any “specific limitation(s) other than what is well - understood, routine,
conventional activity in the field” (WURC). The examiner asserts that, even when considered in
combination, the additional elements of claims 22-41 represent mere instruction to apply a mental process (identifying activation events in a surgical video) at a high level of generality that is generally linked to the field of applying machine learning techniques to surgical videos for identifying operative events.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 22-23 and 34 are rejected as being unpatentable over Venkataraman (US 2020/0303065, as noted as patent publication #2 in IDS filed on 10/29/2025; hereinafter “Venkataraman”) view of Liu et al. (US 2023/0316545; hereinafter “Liu”).
Regarding Claim 22, Venkataraman discloses a computer-implemented method, comprising (see Fig. 2):
receiving a surgical video of a surgical procedure involving energy tool activations ([0049-0050], Venkataraman discloses obtaining surgical videos which include surgical tool engagements.);
applying a sequence of sampling windows to the surgical video to generate a sequence of windowed samples of the surgical video ([0049-0050], Fig. 2, Venkataraman discloses segmenting a surgical video into a set of video sequences.);
for each windowed sample in the sequence of windowed samples, applying a deep- learning model to a sequence of video frames within the windowed sample to generate an activation/non-activation/partial-activation inference, thereby generating a sequence of activation/non-activation/partial-activation inferences ([0037], [0050-0052], Venkataraman discloses identifying surgical tool engagements using a deep neural network.); and
identifying each instance of either i) a single activation or partial-activation inference or ii) multiple consecutive activation or partial-activation inferences located between two non-activation inferences, (With regards to point i), Venkataraman discloses counting (i.e., identifying each instance) the number of times that a surgical tool is activated (see [0037-0038]).).
Venkataraman does not explicitly disclose identifying a sequence of activation events based on the sequence of activation/non- activation/partial-activation inferences, wherein identifying an activation event in the sequence of activation events comprises: identifying each instance of either i) a single activation or partial-activation inference or ii) multiple consecutive activation or partial-activation inferences located between two non-activation inferences (italicized for context), in the sequence of activation/non-activation/partial-activation inferences, as an identified activation event.
Liu discloses identifying a sequence of activation events based on the sequence of activation/non- activation/partial-activation inferences, wherein identifying an activation event in the sequence of activation events comprises: identifying each instance of either i) a single activation or partial-activation inference or ii) multiple consecutive activation or partial-activation inferences located between two non-activation inferences (italicized for context), in the sequence of activation/non-activation/partial-activation inferences, as an identified activation event ([0125], [0133-0134], [0165], Liu discloses annotating individual frames with whether a surgical tool is activated. The Examiner notes that the set of annotated frames is analogous to the claimed “sequence”. Also see [0049-0050], wherein Venkataraman discloses identifying a sequence of video frames to identify a surgical tool engagement.)).
Venkataraman and Liu are considered to be analogous to the claimed invention as they are in the same field of acquiring and analyzing image data from a surgical field. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Venkataraman by specifically identifying sequences of activation events as disclosed by Liu and consequently using the identified sequences to further identify each instance of an activation inference as disclosed by Venkataraman. The motivation for this combination being the ability to specifically define a subset sequence of data which should be processed, reducing the computational load required to process the data.
Claim 34 is the system claim corresponding to claim 22, and is similarly rejected (see Fig. 5, [0063], Venkataraman).
Regarding Claim 23, Venkataraman in view of Liu teaches the computer-implemented method of claim 22, wherein the method further comprises generating a total activation count for the surgical video by: incrementing an activation count by one in response to the multiple consecutive activation or partial-activation inferences; and outputting a final-updated activation count as the total activation count for the surgical video after processing the sequence of activation/non-activation/partial-activation inferences ([0037-0038], Venkataraman discloses a tool-usage counting module which keeps track of the number of uses for a surgical tool which is outputted to surgical staff.).
Claim 26 is rejected as being unpatentable over Venkataraman in view of Liu in view of Espeland and Riegler (US 2022/0296081; hereinafter “Espeland”) in view of Wolf et al. (US 2021/0012868, as noted as patent publication #3 in IDS filed on 10/29/2025; hereinafter “Wolf”).
Regarding Claim 26, Venkataraman in view of Liu teaches the computer-implemented method of claim 22.
Venkataraman in view of Liu does not explicitly teach wherein the sequence of sampling windows has a common window length determined based on an activation duration distribution of a number of previously-identified activation events from a plurality of surgical videos of the surgical procedure.
Espeland discloses wherein the sequence of sampling windows has a common window length ([0055-0060], Fig. 1, Fig. 3A, Espeland discloses a fixed size sliding window that is applied to a sequence of video frames.)
Venkataraman, Liu, and Espeland are considered to be analogous to the claimed invention as they are in the same field of feature extraction from videos. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention taught by Venkataraman in view of Liu by incorporating the sampling windows with same size taught by Espeland in order to make the method to detect energy tool activations in surgical videos. The motivation for this combination being the ability to have sampling sequences of uniform length allowing for uniform segmentation of a surgical video.
Venkataraman in view of Liu in view of Espeland does not teach sampling window size determined based on an activation duration distribution of a large number of previously-identified activation events from a large number of surgical videos of the surgical procedure.
Wolf teaches sampling window size determined based on an activation duration distribution of a large number of previously-identified activation events from a large number of surgical videos of the surgical procedure ([0167-0168], Fig. 6, Wolf teaches tagging specific timestamps within a surgical video with an event tag, such as when a tool is activated. Note that Wolf’s teaching includes identifying the duration of tool activation, which can further then be used to create a distribution of tool activations).
Venkataraman, Liu, Espeland, and Wolf are considered to be analogous to the claimed invention as they are in the same field of feature extraction from videos. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention taught by Venkataraman in view of Liu in view of Espeland by determining the sample size window based on a distribution of energy tool activations taught by Wolf in order to make the method to detect energy tool activations in surgical videos. The motivation for this combination being the ability to determine a distribution of tool activations gives a way to identify a most frequent tool activation time frame, which is beneficial when determining sampling window length to ensure that an activation event will be included within that window length.
Claim 41 is the system claim corresponding to claim 26 and is similarly rejected.
Claims 27 and 35 are rejected as being unpatentable over Venkataraman in view of Liu in view of Espeland.
Regarding Claim 27, Venkataraman in view of Liu teaches the computer-implemented method of claim 22.
Venkataraman in view of Liu does not explicitly teach wherein applying the sequence of sampling windows includes adding a predetermined amount of overlap between consecutive sampling windows.
Espeland discloses wherein applying the sequence of sampling windows includes adding a predetermined amount of overlap between consecutive sampling windows ([0055-0060], Fig. 1, Fig. 3A, Espeland discloses a fixed size sliding window that is applied to a sequence of video frames, wherein there can be overlap between sliding windows.).
Venkataraman, Liu, and Espeland are considered to be analogous to the claimed invention as they are in the same field of feature extraction from videos. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention taught by Venkataraman in view of Liu by ensuring that the sampling windows have an overlap in order to make the method to detect energy tool activations in surgical videos, as disclosed by Espeland. The motivation for this combination being the ability to handle situations when a segment of a video cannot be evenly sampled by a whole number of sampling windows.
Regarding Claim 35, Venkataraman in view of Liu teaches the computer-implemented method of claim 34.
Venkataraman in view of Liu does not explicitly teach wherein applying the sequence of sampling windows includes adding overlap between consecutive sampling windows.
Espeland discloses wherein applying the sequence of sampling windows includes overlap between consecutive sampling windows ([0055-0060], Fig. 1, Fig. 3A, Espeland discloses a fixed size sliding window that is applied to a sequence of video frames, wherein there can be overlap between sliding windows.).
Venkataraman, Liu, and Espeland are considered to be analogous to the claimed invention as they are in the same field of feature extraction from videos. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention taught by Venkataraman in view of Liu by ensuring that the sampling windows have an overlap in order to make the method to detect energy tool activations in surgical videos, as disclosed by Espeland. The motivation for this combination being the ability to handle situations when a segment of a video cannot be evenly sampled by a whole number of sampling windows.
Claims 28-29 are rejected as being unpatentable over Venkataraman in view of Liu in view of Wolf.
Regarding Claim 28, Venkataraman in view of Liu teaches the computer-implemented method of claim 22, wherein the method further comprises training the deep-learning model by ([0057], Venkataraman teaches training a machine learning model to identify tool use.):
for each annotated surgical video, generating a set of labeled training data by sampling the annotated surgical video (Fig. 2, Step 204, Step 206, [0049-0050], Venkataraman teaches segmenting a surgical video into a set of video sequences, wherein each set of video sequences is analogous to a sampling window.);
Venkataraman in view of Liu does not explicitly teach receiving a group of annotated surgical videos of the surgical procedure, wherein each annotated surgical video in the group of annotated surgical videos includes a set of identified activation events, wherein each identified activation event is annotated with a starting timestamp and an end timestamp; adding the set of labeled training data into a training dataset; and training the deep learning model using the training dataset.
Wolf discloses receiving a group of annotated surgical videos of the surgical procedure, wherein each annotated surgical video in the group of annotated surgical videos includes a set of identified activation events, wherein each identified activation event is annotated with a starting timestamp and an end timestamp ([0112-0116], [0167-0168], Wolf discloses training a machine learning model using training examples associated with intraoperative events and/or event characteristics with associated label indicating the location (i.e., timestamp).); adding the set of labeled training data into a training dataset; and training the deep learning model using the training dataset ([0083], Wolf teaches training a machine learning model with a desired label.).
Venkataraman, Liu, and Wolf are considered to be analogous to the claimed invention as they are in the same field of feature extraction from videos. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Venkataraman in view of Liu such that the machine learning model used to identify energy tool activations, as taught by Venkataraman in view of Liu, is trained based on the methods disclosed by Wolf involving annotating training samples with timestamp information regarding intraoperative events and/or event characteristics. The motivation for this combination being the ability to train a machine learning model with sampled surgical video sequences, such that the model can predict surgical tool activation and specify the timestamps where the activation occurs.
Regarding Claim 29, Venkataraman in view of Liu in view of Wolf teaches the computer-implemented method of claim 28, wherein generating the set of labeled training data by sampling the annotated surgical video includes: sequentially applying a sequence of sampling windows to the annotated surgical video to generate a sequence of windowed samples of the annotated surgical video (Fig. 2, Step 204, Step 206, [0049-0050], Venkataraman teaches segmenting a surgical video into a set of video sequences, wherein each set of video sequences is analogous to a sampling window); and for each windowed sample in the sequence of windowed samples, acquiring a ground truth label for the windowed sample based on the temporal location of the windowed sample with respect to the set of annotated activation events in the annotated surgical video; and adding the labeled windowed sample into the set of labeled training data ([0181], Venkataraman teaches assigning temporal annotations to surgical videos).
Claim 30 is rejected as being unpatentable over Venkataraman in view of Liu in view of Wolf in view of Idrees et al. (“The THUMOS Challenge on Action Recognition for Videos ‘in the Wild’”, DOI: 10.48550/arXiv.1604.06182, Publication Year: 2016; hereinafter “Idrees”).
Regarding Claim 30, Venkataraman in view of Liu in view of Wolf teaches the computer-implemented method of claim 29.
The current combination of Venkataraman in view of Liu in view of Wolf does not explicitly teach wherein acquiring the ground truth label for the windowed sample based on the temporal location of the windowed sample includes: providing a first integer label of "1" to the windowed sample if the windowed sample is situated entirely inside an annotation activation event within the set of annotated activation events; and providing a second integer label of "0" to the windowed sample if the windowed sample is situated entirely outside of any of the set of annotated activation events.
Idrees discloses wherein acquiring the ground truth label for the windowed sample based on the temporal location of the windowed sample includes: providing a first integer label of "1" to the windowed sample if the windowed sample is situated entirely inside an annotation activation event within the set of annotated activation events; and providing a second integer label of "0" to the windowed sample if the windowed sample is situated entirely outside of any of the set of annotated activation events (Fig. 3, 4. The THUMOS Dataset, Idrees discloses applying a binary classification label to frames of a video sample based on whether an action is present.)
Venkataraman, Liu, Wolf, and Idrees are considered to be analogous to the claimed invention as they are in the same field of feature extraction from videos. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Venkataraman in view of Liu in view of Wolf such that the ground truth annotations (taught by Venkataraman in view of Liu in view of Wolf), utilize integer values of “1” and “0” using the binary classification logic provided by Idrees (The Examiner notes the obviousness rationales provided in MPEP 2143, specifically the “Obvious to try” rationale which supports the motivation to try any set way to represent a binary decision such as using integer values “1” and “0”.). The motivation for this combination being the ability to specifically distinguish in the ground truth data when an event is present and not present, adding additional granularity which will aid in the training of a machine learning model.
Conclusion
No claims are allowed. Claims 24-25, 31-33, and 36-40 are free of prior art. Final allowability is contingent on overcoming all presented rejections.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PROMOTTO TAJRIAN ISLAM whose telephone number is (703)756-5584. The examiner can normally be reached Monday - Friday 8:30 am - 5:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached at (571) 272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PROMOTTO TAJRIAN ISLAM/Examiner, Art Unit 2669 /CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669
Prosecution Insights