DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is responsive to the Application filed on 01/11/2024 Claims 1-20 are pending in the case. Claims 1, 8, and 15 are independent claims.
Claim Rejections - 35 U.S.C. § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. §§ 102 and 103 (or as subject to pre-AIA 35 U.S.C. §§ 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant are advised of the obligation under 37 C.F.R. § 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. § 102(b)(2)(C) for any potential 35 U.S.C. § 102(a)(2) prior art against the later invention.
Claims 1-20 are rejected under 35 U.S.C. § 103 as being unpatentable over Kim et al. (Kim, Kyungyul, ByeongMoon Ji, Doyoung Yoon, and Sangheum Hwang. "Self-knowledge distillation with progressive refinement of targets." In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6567-6576. 2021, hereinafter Kim) in view of Lee et al. (Lee, Hanbeen, Jeongho Kim, and Simon S. Woo. "Sliding cross entropy for self-knowledge distillation." In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp. 1044-1053. 2022., hereinafter Lee).
As to independent claim 1, Kim teaches:
A deep learning model training method using a self-knowledge distillation algorithm, the method comprising (Title and abstract):
inputting training data to a deep learning model at a first time to obtain first output vectors and inputting the training data to the deep learning model at a second time before the first time to obtain second output vectors (Figure 1, input X at two different epochs t);
generating soft target vectors at the first time point with respect to the training data using the second output vectors and label data (Page 6570, "compute the soft targets at t-th epoch" at two different epochs.);… and
training the deep learning model to minimize a first loss function determined on the basis of the first partial distribution and the second partial distribution (Page 6569, "the student is trained with the loss function LKD, given by...").
Kim does not appear to expressly teach sorting the first output vectors and the soft target vectors and generating a first partial distribution for the sorted first output vectors and a second partial distribution for the sorted soft target vectors.
Lee teaches sorting the first output vectors and the soft target vectors and generating a first partial distribution for the sorted first output vectors and a second partial distribution for the sorted soft target vectors (Figure 1, "SCE sorts soft target and model output in descending order based on soft target values").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the progressive self-knowledge distillation of Kim to include the sliding cross entropy of Lee to evenly considers all the inter-class relationships of a soft target during optimization (see Lee at abstract).
As to dependent claim 2, Kim further teaches a number of times of training the deep learning model at the first time and a number of times of training the deep learning model at the second time are different from each other (Figure 1, "epoch t, a student at epoch (t − 1)". The number of training iterations at the first time and the second time are different.).
As to dependent claim 3, Lee further teaches the generating the first partial distribution and the second partial distribution includes: sorting the soft target vectors on the basis of confidence scores for multiclass classification (Page 1047, "sort the soft target s in descending order"); and sorting the first output vectors in the same order as a class order of the sorted soft target vectors (Page 1047, "rearrange the output logit zs by order of the soft target").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the progressive self-knowledge distillation of Kim to include the sliding cross entropy of Lee to evenly considers all the inter-class relationships of a soft target during optimization (see Lee at abstract).
As to dependent claim 4, Lee further teaches the generating the first partial distribution and the second partial distribution includes generating the first partial distribution and the second partial distribution by dividing all classes included in the first output vectors and the soft target vectors by a preset number of classes (Figure 1, "each of both sorted representation is sliced according to the window size w and stride s. Finally, LSCE is calculated by each sliced pair, respectively". The predetermined window size used to divide the sorted class representations into partial class groups reads on the claimed present number of classes).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the progressive self-knowledge distillation of Kim to include the sliding cross entropy of Lee to evenly considers all the inter-class relationships of a soft target during optimization (see Lee at abstract).
As to dependent claim 5, Kim further teaches generating a first partial probability distribution and a second partial probability distribution from the first partial distribution and the second partial distribution using a softmax function (Page 6569, "For an input x and a K-dimensional one-hot target y, a model produces the logit vector z(x) = [z1(x), · · · , zK(x)], and then outputs the predicted probabilities P(x) = [p1(x), · · · , pK(x)] by a softmax function").
As to dependent claim 6, Lee further teaches the first loss function is determined on the basis of the difference between the first partial probability distribution and the second partial probability distribution (Abstract, "reduce the distance between sliced parts").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the progressive self-knowledge distillation of Kim to include the sliding cross entropy of Lee to evenly considers all the inter-class relationships of a soft target during optimization (see Lee at abstract).
As to dependent claim 7, Kim further teaches the training the deep learning model includes: determining a second loss function on the basis of the overall distributions of the first output vectors and the soft target vectors (Figure 1, loss for epoch t); and training the deep learning model to minimize a third loss function corresponding to a weighted sum of the first loss function and the second loss function (Figure 1, loss for a different epoch).
As to independent claim 8, Kim teaches:
A deep learning model interference device comprising: a memory configured to store a deep learning model and one or more instructions for performing inference using the deep learning model; and a processor configured to execute the one or more instructions stored in the memory, when executed by the processor, cause the processor to perform inference of the deep learning model, wherein the deep learning model is trained to (Title and abstract):
receive training data at a first time to obtain first output vectors and receive the training data at a second time before the first time to obtain second output vectors (Figure 1, input X at two different epochs t);
generate soft target vectors at the first time with respect to the training data using the second output vectors and label data (Page 6570, "compute the soft targets at t-th epoch" at two different epochs.);… and
minimize a first loss function determined on the basis of the first partial distribution and the second partial distribution, wherein output according to input data of the same domain as the training data is generated using the pre-trained model (Page 6569, "the student is trained with the loss function LKD, given by...").
Kim does not appear to expressly teach sort the first output vectors and the soft target vectors and generate a first partial distribution for the sorted first output vectors and a second partial distribution for the sorted soft target vectors.
Lee teaches sort the first output vectors and the soft target vectors and generate a first partial distribution for the sorted first output vectors and a second partial distribution for the sorted soft target vectors (Figure 1, "SCE sorts soft target and model output in descending order based on soft target values").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the progressive self-knowledge distillation of Kim to include the sliding cross entropy of Lee to evenly considers all the inter-class relationships of a soft target during optimization (see Lee at abstract).
As to dependent claim 9, Kim further teaches the number of times of training the deep learning model at the first time and the number of times of training the deep learning model at the second time are different from each other (Figure 1, "epoch t, a student at epoch (t − 1)". The number of training iterations at the first time and the second time are different).
As to dependent claim 10, Lee further teaches the deep learning model is trained to sort the soft target vectors on the basis of confidence scores for multiclass classification and to sort the first output vectors in the same order as a class order of the sorted soft target vectors (Page 1047, "sort the soft target s in descending order". Page 1047, "rearrange the output logit zs by order of the soft target").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the progressive self-knowledge distillation of Kim to include the sliding cross entropy of Lee to evenly considers all the inter-class relationships of a soft target during optimization (see Lee at abstract).
As to dependent claim 11, Lee further teaches the deep learning model is trained to generate the first partial distribution and the second partial distribution by dividing all classes included in the first output vectors and the soft target vectors by a preset number of classes (Figure 1, "each of both sorted representation is sliced according to the window size w and stride s. Finally, LSCE is calculated by each sliced pair, respectively". The predetermined window size used to divide the sorted class representations into partial class groups reads on the claimed present number of classes).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the progressive self-knowledge distillation of Kim to include the sliding cross entropy of Lee to evenly considers all the inter-class relationships of a soft target during optimization (see Lee at abstract).
As to dependent claim 12, Kim further teaches the deep learning model is trained to generate a first partial probability distribution and a second partial probability distribution from the first partial distribution and the second partial distribution using a softmax function (Page 6569, "For an input x and a K-dimensional one-hot target y, a model produces the logit vector z(x) = [z1(x), · · · , zK(x)], and then outputs the predicted probabilities P(x) = [p1(x), · · · , pK(x)] by a softmax function").
As to dependent claim 13, Lee further teaches the first loss function is determined on the basis of the difference between the first partial probability distribution and the second partial probability distribution (Abstract, "reduce the distance between sliced parts").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the progressive self-knowledge distillation of Kim to include the sliding cross entropy of Lee to evenly considers all the inter-class relationships of a soft target during optimization (see Lee at abstract).
As to dependent claim 14, Kim further teaches the deep learning model is trained to determine a second loss function on the basis of the overall distributions of the first output vectors and the soft target vectors and to minimize a third loss function corresponding to a weighted sum of the first loss function and the second loss function (Figure 1, loss for epoch t. Figure 1, loss for a different epoch).
As to independent claim 15, Kim teaches:
A non-transitory computer readable storage medium storing computer executable instructions, wherein the instructions, when executed by a processor, cause the processor to perform a deep learning model training method using a self-knowledge distillation algorithm, the method comprising (Title and abstract):
inputting training data to a deep learning model at a first time to obtain first output vectors and inputting the training data to the deep learning model at a second time before the first time to obtain second output vectors (Figure 1, input X at two different epochs t);
generating soft target vectors at the first time point with respect to the training data using the second output vectors and label data (Page 6570, "compute the soft targets at t-th epoch" at two different epochs);…and
training the deep learning model to minimize a first loss function determined on the basis of the first partial distribution and the second partial distribution (Page 6569, "the student is trained with the loss function LKD, given by...").
Kim does not appear to expressly teach sorting the first output vectors and the soft target vectors and generating a first partial distribution for the sorted first output vectors and a second partial distribution for the sorted soft target vectors.
Lee teaches sorting the first output vectors and the soft target vectors and generating a first partial distribution for the sorted first output vectors and a second partial distribution for the sorted soft target vectors (Figure 1, "SCE sorts soft target and model output in descending order based on soft target values").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the progressive self-knowledge distillation of Kim to include the sliding cross entropy of Lee to evenly considers all the inter-class relationships of a soft target during optimization (see Lee at abstract).
As to dependent claim 16, Kim further teaches the number of times of training the deep learning model at the first time and the number of times of training the deep learning model at the second time are different from each other (Figure 1, "epoch t, a student at epoch (t − 1)". The number of training iterations at the first time and the second time are different).
As to dependent claim 17, Lee further teaches the generating the first partial distribution and the second partial distribution includes: sorting the soft target vectors on the basis of confidence scores for multiclass classification (Page 1047, "sort the soft target s in descending order"); and sorting the first output vectors in the same order as a class order of the sorted soft target vectors (Page 1047, "rearrange the output logit zs by order of the soft target").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the progressive self-knowledge distillation of Kim to include the sliding cross entropy of Lee to evenly considers all the inter-class relationships of a soft target during optimization (see Lee at abstract).
As to dependent claim 18, Lee further teaches the generating of the first partial distribution and the second partial distribution includes generating the first partial distribution and the second partial distribution by dividing all classes included in the first output vectors and the soft target vectors by a preset number of classes (Figure 1, "each of both sorted representation is sliced according to the window size w and stride s. Finally, LSCE is calculated by each sliced pair, respectively". The predetermined window size used to divide the sorted class representations into partial class groups reads on the claimed present number of classes).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the progressive self-knowledge distillation of Kim to include the sliding cross entropy of Lee to evenly considers all the inter-class relationships of a soft target during optimization (see Lee at abstract).
As to dependent claim 19, Lee further teaches the first loss function is determined on the basis of the difference between a first partial probability distribution generated from the first partial distribution and a second partial probability distribution generated from the second partial distribution (Abstract, "reduce the distance between sliced parts").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the progressive self-knowledge distillation of Kim to include the sliding cross entropy of Lee to evenly considers all the inter-class relationships of a soft target during optimization (see Lee at abstract).
As to dependent claim 20, Kim further teaches the training of the deep learning model comprises: determining a second loss function on the basis of the overall distributions of the first output vectors and the soft target vectors (Figure 1, loss for epoch t); and training the deep learning model to minimize a third loss function corresponding to a weighted sum of the first loss function and the second loss function (Figure 1, loss for a different epoch).
Citation of Pertinent Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Fukuda (Int’l Pat. App. Pub. No. WO-2022121515-A1) teaches A method of training a student neural network. The method includes feeding a data set including a plurality of input vectors into a teacher neural network to generate a plurality of output values, and converting two of the plurality of output values from the teacher neural network for two corresponding input vectors into two corresponding soft labels. The method further includes combining the two corresponding input vectors to form a synthesized data vector, and forming a masked soft label vector from the two corresponding soft labels. The method further includes feeding the synthesized data vector into the student neural network, using the masked soft label vector to determine an error for modifying weights of the student neural network, and modifying the weights of the student neural network.
Conclusion
The prior art made of record and not relied upon is considered pertinent to Applicant's disclosure. Applicant is required under 37 C.F.R. § 1.111(c) to consider these references fully when responding to this action.
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33, 216 U.S.P.Q. 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 U.S.P.Q. 275, 277 (C.C.P.A. 1968)).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Casey R. Garner whose telephone number is 571-272-2467. The examiner can normally be reached Monday to Friday, 8am to 5pm, Eastern Time.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached on 571-270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center and the Private Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from Patent Center or Private PAIR. Status information for unpublished applications is available through Patent Center and Private PAIR to authorized users only. Should you have questions about access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/Casey R. Garner/Primary Examiner, Art Unit 2123
Prosecution Insights