Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim 1-13 are rejected under 35 USC 103 as being unpatentable over Liu et al. (US Patent 12,400,753) in view of Yu et al. (“Learning from a tiny dataset of manual annotations: a teacher/student approach for surgical phase recognition”, arXiv:1812.00033v3 [cs.LG] 30 Sep 2020).
Liu et al. teaches performing training of a self-knowledge distillation encoder(col. 6 lines 34-62, TTN (teacher network) conducts the weight self-ensembling part by replicating the average weights of the TSN (teacher network) instead of predictions.) using image of a surgical procedure by joint optimization (total loss) of a classification loss (adversarial loss) and feature similarity loss (MSE loss) through a student encoder network (TSN) and a teacher encoder network(TTN) (Fig. 2A and 2B);
providing a plurality of features extracted by the student encoder network to a self-knowledge distillation decoder (Fig. 2A and 2B);
performing training of the self-knowledge distillation decoder using the features, wherein the self-knowledge distillation decoder comprises a student decoder network and a teacher decoder network, and a plurality of soft labels (EMA prediction) generated by the teacher decoder network are used to regularize a prediction of the student decoder network (col. 8 lines 40-54, the EMA predictions and the base predictions are generated by the teacher model and the student model respectively with different augmentation applied to the target samples. Afterward, a mean-squared error (MSE) loss between the EMA and target predictions is calculated. Finally, the MSE loss together with the task-specific loss on the labeled source domain data is minimized to update the parameters of the student network ) ((Fig. 2A and 2B); and
combining a trained version of the self-knowledge distillation encoder and the self-knowledge distillation decoder as an image recognition model (Fig. 2A and 2B).
Liu et al. do not teach performing training of encoder sung a plurality of video frame of a surgical procedure and combining a trained version encoder and decoder as a phase recognition model to predict surgical phases of the surgical procedure in one or more videos.
Yu et al. teach performing training of encoder sung a plurality of video frame of a surgical procedure and combining a trained version encoder and decoder as a phase recognition model to predict surgical phases of the surgical procedure in one or more videos (page 2, introduction, , source of video data, which deep learning algorithms can leverage for surgical work flow analysis; surgical phase recognition using teacher/student approach).
At the time of effective filing, it would have been obvious to a person of ordinary skill in the art to recognize surgical phase of video using teacher/student approach in the method of Liu et al.
The suggestion/motivation for doing so would have been that to improve Yu et al.’s offline pseudo-labeling framework by adopting Liu et al.’s online self-knowledge distillation (EMA-updated teacher and joint MSE similarity loss) to extract dense, invariant visual features without relying on artificial offline label generation.
Therefore, it would have been obvious to combine Yu et al. with Liu et al. to obtain the invention as specified in claim 1.
With respect to claim 2, Liu et al. teach applying a different image augmentation to the video frames provided to each of the student encoder network and the teacher encoder network (col. 8 lines 40-54, the EMA predictions and the base predictions are generated by the teacher model and the student model respectively with different augmentation applied to the target samples.)
With respect to claim 3, Liu et al. teach that the student encoder network comprises a student backbone portion that generates a plurality of frame representations (Fig. 2A and 2B, encoder generates a plurality of image representations across multiple hierarchical levels of the network).
With respect to claim 4, Liu et al. teach forwarding the frame representations to a first task-specific head to perform classification optimization and a second task-specific head to perform similarity optimization (Fig. 2A and 2B)
With respect to claim 5, Liu et al. teach that the teacher encoder network generates a feature representation as a target for the student encoder network (Fig. 2A and 2B)
With respect to claim 6, Liu et al. teach that the student encoder network is trained for phase recognition and feature similarity (Fig. 2A and 2B)
With respect to claim 7, Liu et al. teach that the soft labels generated by the teacher decoder network in a current epoch are used to produce predictions for a same video frame as the student decoder network with variant logits (Fig. 2A and 2B, The EMA prediction generated by the Teacher is compared with current prediction produced by the student for the same image to form MSE error)
With respect to claim 8, Liu et al. teach minimizing teacher and student logits using a smoothing loss (Fig. 2A and 2B, training of the student model is iteratively improved (minimized) by the task-specific loss, a moving average (EMA) (smoothing) model (teacher) of the student model).
With respect to claim 9, claim 9 is rejected same reason as claim 1 above.
With respect to claim 10, claim 10 is rejected same reason as claim 1 above.
With respect to claim 11, claim 11 is rejected same reason as claim 7 above.
With respect to claim 12, claim 12 is rejected same reason as claim 2 above.
With respect to claim 13, claim 13 is rejected same reason as claims 3-5 above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Randolph Chu whose telephone number is 571-270-1145. The examiner can normally be reached on Monday to Thursday from 7:30 am - 5 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached on (571) 272-7778.
The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/RANDOLPH I CHU/
Primary Examiner, Art Unit 2667
