DETAILED ACTION
This action is in response to the amendments and remarks filed 2/27/2026. Claims 1-19 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-3, 7, 9-13, 17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Hao et al. (AN END-TO-END ARCHITECTURE FOR CLASS-INCREMENTAL OBJECT DETECTION WITH KNOWLEDGE DISTILLATION, published 2019, 2019 IEEE International Conference on Multimedia and Expo (ICME)), hereafter referred to as Hao, in view of Zhao et al. (TRAINING METHOD AND TRAINING APPARATUS FOR A NEURAL NETWORK FOR OBJECT RECOGNITION, filed 11/4/2021, US 20220138454 A1), hereafter referred to as Zhao, and further in view of Zoldi et al. (Building Resilient Models To Address Dynamic Customer Data Use Rights, filed 4/17/2020, US 11709918 B2), hereafter referred to as Zoldi, and Kachare et al. (DYNAMIC QUANTIZATION IN STORAGE DEVICES USING MACHINE LEARNING, published 9/30/2021, US 2021/0303156 A1), hereafter referred to as Kachare.
Regarding claim 1, Hao discloses [a] method for performing continual learning by a learning apparatus including at least one processor, the method comprising:
(a) generating, by the learning apparatus, a teacher network and a student network from a pre-trained model using knowledge distillation: “Given an object detection model that is well-trained with images of certain old classes Co, our goal is to update the model with only images of certain new classes Cn, Co ∩ Cn = ∅” (Hao, page 2, right column, paragraph 2); “As shown in Fig. 1, we propose CIFRCN with the backbone structure of the Faster R-CNN, which contains the following three main components: … (2) the knowledge distillation mechanism in the FRCN branch is applied to preserve the knowledge of the old classes “ (Hao, page 2, right column, paragraph 3); “The self-imitated distillation structure is depicted in Fig. 3. The FRCN (pre-trained model) is also duplicated and frozen as the teacher (teacher network). Then, the output nodes of the classifier are expanded, and the weights of the newly added classes are randomly initialized. The modified network is treated as the student (student network)” (Hao, page 3, left column, paragraph 5)
(b) generating, by the learning apparatus, a representation memory including a plurality of storages respectively corresponding to classes and each having a predetermined range of storable feature representation values to store feature representation values in the teacher network and the student network: “In CIOD task, the RPN is expected to generate proposals containing objects of any classes observed so far … Inspired by Jung et al. [13], our work is designed to maintain the decision boundary of the classifier in RPN to preserve the knowledge gained from the old classes“ (Hao, page 2, right column, paragraph 5); “a feature-changing loss is proposed to reduce the difference of the feature maps between the old and new classes:
PNG
media_image1.png
137
758
media_image1.png
Greyscale
where x is the feature map of the image that is extracted by the feature extractor;
θ
T
R
P
N
and
θ
S
R
P
N
denote the weights of the teacher and student models, respectively;
f
L
-
1
R
P
N
(
∙
)
denotes the feature (feature representation value) before the classifier in RPN” (Hao, page 3, left column, paragraph 2). Feature representation values of both the teacher and student network must be stored in memory (representation memory) at some point to execute this loss equation.
(c) extracting, by the learning apparatus, a feature representation value for target data through the teacher network and storing the feature representation value in a corresponding one of the class-specific storages of a teacher-network representation memory before learning is entered: “a feature-changing loss is proposed to reduce the difference of the feature maps between the old and new classes:
PNG
media_image1.png
137
758
media_image1.png
Greyscale
where x is the feature map of the image (target data) that is extracted by the feature extractor;
θ
T
R
P
N
and
θ
S
R
P
N
denote the weights of the teacher and student models, respectively;
f
L
-
1
R
P
N
(
∙
)
denotes the feature before the classifier in RPN” (Hao, page 3, left column, paragraph 2).
(d) entering, by the learning apparatus, learning to extract the feature representation value for the target data through the student network and storing the feature representation value in a corresponding one of the class-specific storages of a student-network representation memory:
PNG
media_image1.png
137
758
media_image1.png
Greyscale
where x is the feature map of the image (target data) that is extracted by the feature extractor;
θ
T
R
P
N
and
θ
S
R
P
N
denote the weights of the teacher and student models, respectively;
f
L
-
1
R
P
N
(
∙
)
denotes the feature before the classifier in RPN” (Hao, page 3, left column, paragraph 2).
(e) calculating, by the learning apparatus, a representation loss based on mean-square differences between class-wise averaged values stored in the teacher-network representation memory and class-wise averaged values stored in the student-network representation memory: “a feature-changing loss (representation loss) is proposed to reduce the difference of the feature maps between the old and new classes:
PNG
media_image1.png
137
758
media_image1.png
Greyscale
where x is the feature map of the image that is extracted by the feature extractor;
θ
T
R
P
N
and
θ
S
R
P
N
denote the weights of the teacher and student models, respectively;
f
L
-
1
R
P
N
(
∙
)
denotes the feature before the classifier in RPN;
J
(
∙
)
measures the mean squared error (MSE)” (Hao, page 3, left column, paragraph 2).
Hao relates to continuous learning through knowledge distillation of neural networks and is analogous to the claimed invention.
While Hao fails to disclose the further limitations of the claim, Zhao discloses [a] method for performing continual learning by a learning apparatus including at least one processor: “the above-mentioned training apparatus 100 may be implemented in a variety of other forms, for example, it may be a general-purpose processor” (Zhao, [0129]).
Zhao’s method comprising:
(c) extracting, by the learning apparatus, a feature representation value for target data through the teacher network and storing the feature representation value in a corresponding one of class-specific storage of a teacher-network representation before learning is entered: “Subsequently, the normal image samples (target data) in the training image set input into the student network
M
S
are also input into the normal image teacher network
M
N
T
to extract the teacher feature
f
N
T
(feature representation) … the teacher features may be extracted and stored before the training (before learning) of the student network
M
S
, for use in the training process of the student network
M
S
” (Zhao, [0085])
(d) entering, by the learning apparatus, learning to extract the feature representation value for the target data through the student network and storing the feature representation value in a corresponding one of the class-specific storages of a student-network representation memory: “A training (learning) method of a neural network (student network) for object recognition, comprising: inputting a training image set containing an object to be recognized, which includes a set of normal image samples (target data) and a set of variation image samples, into the neural network to extract a student feature (feature representation) of each of the image samples” (Zhao, [0210])
Zhao relates to knowledge distillation for image-processing neural networks and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hao to extract teacher features before training the student network, as disclosed by Zhao. Doing so would enable the use of teaching features in training the student network. See Zhao, [0085].
While Hao and Zhao fail to disclose the further limitations of the claim, Zoldi discloses a method of (b) generating, by the learning apparatus, a representation memory including a plurality of storages respectively corresponding to classes and each having a predetermined range of storable feature representation values to store feature representation values in the teacher network and the student network: “To operationalize on a large scale, each variable (class) is discretized into value range (predetermined range) bins. Discretization can be done in one of several ways. In some implementations using equi-sized bins, each variable is divided into decile bins with equal population. In other implementations using entropy bins, each variable is divided into the bins such that the overall entropy is maximized. In all approaches, each bin should have a minimum number of data-points (feature representation values), η, for stable statistics. Bin creation can be part of the model design, and can vary based on application.” (Zoldi, column 5, paragraph 4). The ‘representation memory’ consists of the set of all storage bins across all variables. The set of bins for a variable is a storage, with the representation memory then consisting of a plurality of storages as many as the number of classes.
Zoldi relates to feature selection and storage for machine learning models and is analogous to the claimed invention. The combination of Hao and Zhao teaches a method of transferring knowledge from a teacher neural network to a student network. Zoldi teaches a method of separately storing feature data, organized by classes. It would have been obvious to one of ordinary skill in the art to combine the combination of Hao and Zhao with Zoldi by using separate storages to store the student and teacher feature data of Hao and Zhao. This would achieve the predictable result of storing features from multiple classes concurrently, with Hao and Zhao’s method and Zoldi’s storage performing the same together as they did separately. (MPEP 2143 I. (A) Combining prior art elements according to known methods to yield predictable results).
While Zoldi fails to disclose the further limitations of the claim, Kachare discloses (e) calculating, by the learning apparatus, a representation loss based on mean-square differences between class-wise averaged values stored in the teacher network representation memory and class-wise averaged values stored in the student network representation memory:
“The basic idea is to store an approximation of multiple closely-related datasets. Embodiments of the inventive concept may use a neural network-based technique to classify the input datasets into unique buckets or clusters (representation memor[ies]). For each such bucket or cluster, the SSD may store one representative dataset (values stored). When the host performs reads to any of the datasets belonging to a cluster or bucket, the stored single representative dataset may be returned to the host.” (Kachare, [0022])
“Each input dataset is assigned a Class ID, also known as a bucket ID or cluster ID. This Class ID may then be stored in a table against LBA and PBA. For each Class ID, only one dataset (class-wise values) may be stored” (Kachare, [0031])
“In yet other embodiments of the inventive concept, the representative data chunk may be updated in a manner that factors in earlier data written to the class ID. Examples of such techniques may include weighted averaging and centroid calculations. As an example of a weighted averaging, weights may be determined (which may be either specified at the time of manufacture of the storage device and not alterable or configurable by machine 105 of FIG. 1). The most recently received data chunk may be combined with the previous representative data chunk by multiplying the value of the most recently received data chunk by one weight and by multiplying the previous representative data chunk by another weight, and then summing the two products … The result of the sum (class-wise averaged values) of this calculation may then be stored as the new representative data chunk for the class ID” (Kachare, [0081])
Kachare relates to dividing memory into a plurality of class representative storages and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the existing combination to store and perform calculations on average class representative values in lieu of the full sets of student and teacher features, as disclosed by Kachare. Machine learning processes, such as the student-teacher knowledge distillation disclosed by the existing combination, often require very large amounts of data. Kachare’s method can significantly reduce the amount of data needed to be stored, thus providing a larger data storage capacity than would otherwise be possible. See Kachare, [0021-0022], [0030].
Regarding claim 2, the rejection of claim 1 in view of Hao, Zhao, Zoldi, and Kachare is incorporated. Hao further discloses a method, wherein
the step (a) is performed by knowledge distillation for transferring knowledge of the teacher network serving as a large model to the student network serving as a relatively small model by referring to the pre-trained model: “Thus, in order to address the catastrophic forgetting problem, knowledge distillation [5] is introduced into CIOD.“ (Hao, page 1, right column, paragraph 3); “Typically, knowledge from a well-trained teacher model, which is larger and more accurate, is transferred to shallower student networks” (Hao, page 2, right column, paragraph 1)
wherein the teacher network and the student network designate models trained in a previous task as a teacher model and a student model of a current task respectively: “For a multi-class detection dataset containing objects of C classes, the classes are divided into Ng class groups (groups of task[s]). For both training and testing sections of each class group, images that only contain objects of classes in this class group are selected from the corresponding section of the entire dataset, which forms the corresponding example groups. This process ensures that only images of new classes are utilized to perform class-incremental training“ (Hao, page 4, left column, paragraph 2); “For the first training session, the model is initialized with weights that were pre-trained on ImageNet [18]. Then, the model is trained by training data of group A (previous task group). For the following three incremental training sessions, the model from the last session is loaded, and a fixed duplication of the model as the teacher is made. The output nodes in FRCN are then extended to the un-fixed one as the student. After that, the student net is trained as our proposed method by the training data of classes in group B, C, and D (current task groups), individually” (Hao, page 4, right column, paragraph 4). The teacher and student models of a current task (tasks B, C, and D in this case) are derived from a model trained on a previous ask (task A in this case).
Regarding claim 3, the rejection of claim 1 in view of Hao, Zhao, Zoldi, and Kachare is incorporated. Hao further discloses a method, wherein the step (b) includes dividing the classes into ground truth classified according to tasks to be performed by the model, generating representation memories as many as the number of classes for storing the feature representation values, dividing the generated memory into a plurality of storages, and setting a limited range of storable values for each storage: “During training, the normalized feature f of each proposal is recorded and
μ
y
←
1
|
y
c
=
y
|
∑
ς
(
f
|
y
c
=
y
)
is used to obtain the prototype
μ
y
of class y, where
y
c
is the ground truth class of the proposal” (Hao, page 3, right column, paragraph 4).
While Hao fails to disclose the further limitations of the claim, Zoldi discloses a method, wherein the step (b) includes dividing the classes into ground truth classified according to the tasks to be performed by the model, generating representation memories as many as the number of classes for storing the feature representation values, dividing the generated memory into a plurality of storages, and setting a limited range of storable values for each storage: “To operationalize on a large scale, each variable (class) is discretized into value range bins (representation memories). Discretization can be done in one of several ways. In some implementations using equi-sized bins, each variable is divided into decile bins (plurality of storages) with equal population. In other implementations using entropy bins, each variable is divided into the bins such that the overall entropy is maximized. In all approaches, each bin should have a minimum number of data-points, η, for stable statistics. Bin creation can be part of the model design, and can vary based on application.” (Zoldi, column 5, paragraph 4).
Zoldi relates to feature selection and storage for machine learning models and is analogous to the claimed invention. The combination of Hao, Zhao, Zoldi, and Kachare teaches a method of transferring knowledge from a teacher neural network to a student network. The claimed invention improves upon this method by subdividing each class feature storage into a set of ranged buckets. Zoldi teaches a method of subdividing class feature storages into ranged buckets, applicable to the combination of Hao, Zhao, Zoldi, and Kachare. A person of ordinary skill in the art would have recognized that creating ranged storage buckets would lead to the predictable result of discretizing continuous feature values, and would improve the known device by allowing it to operate on continuous variables (MPEP 2143 I. (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results).
Regarding claim 7, the rejection of claim 1 in view of Hao, Zhao, Zoldi, and Kachare is incorporated. Hao further discloses a method, wherein the step (d) includes:
(d1) receiving the target data and inputting the target data to the student network:
PNG
media_image2.png
450
1062
media_image2.png
Greyscale
(Hao, page 2, right column, Fig. 1); “The teacher and student are simultaneously fed the same images (target data) of the new classes, and the class predictions are recorded as
P
T
and
P
S
, individually.” (Hao, page 3, right column, paragraph 1).
(d2) applying a softmax function to an output value to determine a value in a predetermined range: “
P
S
T
is a part of
P
S
and is the prediction on old classes made by the student.
y
o
'
and
y
n
'
are the transformed form of
P
T
and
P
S
T
(output value[s]), called “soft labels”. The transformer is a modified softmax function in which:
PNG
media_image3.png
89
343
media_image3.png
Greyscale
” (Hao, page 3, right column, paragraph 2). All softmax outputs are within the predetermined range of [0, 1].
(d3) extracting a feature map belonging to a last layer of the student network when the input target data is inferred as ground truth:
“a feature-changing loss is proposed to reduce the difference of the feature maps between the old and new classes:
PNG
media_image1.png
137
758
media_image1.png
Greyscale
where x is the feature map of the image (input target data) that is extracted by the feature extractor;
θ
T
R
P
N
and
θ
S
R
P
N
denote the weights of the teacher and student models, respectively;
f
L
-
1
R
P
N
(
∙
)
denotes the feature (feature map) before the classifier in RPN” (Hao, page 3, left column, paragraph 2).
“The feature before the classifier in the FRCN branch is denoted as
P
f
=
f
L
-
1
R
P
N
(
x
)
, and is normalized using
f
=
ς
P
f
=
P
f
|
|
P
f
|
|
.” (Hao, page 3, right column, paragraph 3)
“During training, the normalized feature f of each proposal is recorded and
μ
y
←
1
|
y
c
=
y
|
∑
ς
(
f
|
y
c
=
y
)
is used to obtain the prototype
μ
y
of class y, where
y
c
is the ground truth class of the proposal. For class y ∈
C
n
,
μ
y
is added to the classifier to expand its ability during class-incremental training” (Hao, page 3, right column, paragraph 4).
(d4) acquiring the feature representation value from the extracted feature map: “The feature before the classifier in the FRCN branch is denoted as
P
f
=
f
L
-
1
R
P
N
(
x
)
” (Hao, page 3, right column, paragraph 3).
f
L
-
1
R
P
N
, an acquired feature representation value, is stored in memory as
P
f
.
While Hao fails to disclose the further limitations of the claim, Zoldi further discloses a method, wherein the step (d) includes: … (d5) storing the acquired feature representation value in the storage corresponding to the determined value in the predetermined range: “Each feature is split into value ranges, called “bins,” and a customer (feature representation) is assigned to one of these bins for each feature” (Zoldi, column 5, paragraph 1).
Zoldi relates to feature selection and storage for machine learning models and is analogous to the claimed invention. The combination of Hao, Zhao, Zoldi, and Kachare teaches a method of transferring knowledge from a teacher neural network to a student network. The claimed invention improves upon this method by storing feature values in ranged buckets. Zoldi teaches a method of storing feature values in ranged buckets, applicable to the combination of Hao, Zhao, Zoldi, and Kachare. A person of ordinary skill in the art would have recognized that creating ranged storage buckets would lead to the predictable result of discretizing continuous feature values, and would improve the known device by allowing it to operate on continuous variables (MPEP 2143 I. (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results).
Regarding claim 9, the rejection of claim 1 in view of Hao, Zhao, Zoldi, and Kachare is incorporated. Hao further discloses a method, wherein the step (e) includes performing a mean square of average values stored in the storage of the teacher network and average values stored in the storage of the student network to calculate the representation loss: “Additionally, a feature-changing loss is proposed to reduce the difference of the feature maps between the old and new classes:
PNG
media_image4.png
133
759
media_image4.png
Greyscale
…
θ
T
R
P
N
and
θ
S
R
P
N
denote the weights of the teacher and student models, respectively; …
J
(
∙
)
measures the mean squared error (MSE)” (Hao, page 3, left column, paragraph 2). The mean squared error between features of the student and features of the teacher networks is calculated.
Regarding claim 10, Hao discloses instructions comprising:
(a) generating a teacher network and a student network from a pre-trained model using knowledge distillation: “Given an object detection model that is well-trained with images of certain old classes Co, our goal is to update the model with only images of certain new classes Cn, Co ∩ Cn = ∅” (Hao, page 2, right column, paragraph 2); “As shown in Fig. 1, we propose CIFRCN with the backbone structure of the Faster R-CNN, which contains the following three main components: … (2) the knowledge distillation mechanism in the FRCN branch is applied to preserve the knowledge of the old classes “ (Hao, page 2, right column, paragraph 3); “The self-imitated distillation structure is depicted in Fig. 3. The FRCN (pre-trained model) is also duplicated and frozen as the teacher (teacher network). Then, the output nodes of the classifier are expanded, and the weights of the newly added classes are randomly initialized. The modified network is treated as the student (student network)” (Hao, page 3, left column, paragraph 5)
(b) generating a representation memory including plural class-specific storages each having a predetermined storage value range to store feature representation values: “In CIOD task, the RPN is expected to generate proposals containing objects of any classes observed so far … Inspired by Jung et al. [13], our work is designed to maintain the decision boundary of the classifier in RPN to preserve the knowledge gained from the old classes“ (Hao, page 2, right column, paragraph 5); “a feature-changing loss is proposed to reduce the difference of the feature maps between the old and new classes:
PNG
media_image1.png
137
758
media_image1.png
Greyscale
where x is the feature map of the image that is extracted by the feature extractor;
θ
T
R
P
N
and
θ
S
R
P
N
denote the weights of the teacher and student models, respectively;
f
L
-
1
R
P
N
(
∙
)
denotes the feature (feature representation value) before the classifier in RPN” (Hao, page 3, left column, paragraph 2). Feature representation values of both the teacher and student network must be stored in memory (representation memory) at some point to execute this loss equation.
(c) extracting a feature representation value for target data through the teacher network and storing the feature representation value in a corresponding class-specific storage of a teacher-network representation memory before learning is entered: “a feature-changing loss is proposed to reduce the difference of the feature maps between the old and new classes:
PNG
media_image1.png
137
758
media_image1.png
Greyscale
where x is the feature map of the image (target data) that is extracted by the feature extractor;
θ
T
R
P
N
and
θ
S
R
P
N
denote the weights of the teacher and student models, respectively;
f
L
-
1
R
P
N
(
∙
)
denotes the feature before the classifier in RPN” (Hao, page 3, left column, paragraph 2).
(d) entering, by the learning apparatus, learning to extract the feature representation value for the target data through the student network and storing the feature representation value in a corresponding class-specific storage of a student-network representation memory:
PNG
media_image1.png
137
758
media_image1.png
Greyscale
where x is the feature map of the image (target data) that is extracted by the feature extractor;
θ
T
R
P
N
and
θ
S
R
P
N
denote the weights of the teacher and student models, respectively;
f
L
-
1
R
P
N
(
∙
)
denotes the feature before the classifier in RPN” (Hao, page 3, left column, paragraph 2).
(e) calculating a representation loss using class-wise averaged values stored in the teacher-network representation memory and class-wise averaged values stored in the student-network representation memory: “a feature-changing loss (representation loss) is proposed to reduce the difference of the feature maps between the old and new classes:
PNG
media_image1.png
137
758
media_image1.png
Greyscale
where x is the feature map of the image that is extracted by the feature extractor;
θ
T
R
P
N
and
θ
S
R
P
N
denote the weights of the teacher and student models, respectively;
f
L
-
1
R
P
N
(
∙
)
denotes the feature before the classifier in RPN;
J
(
∙
)
measures the mean squared error (MSE)” (Hao, page 3, left column, paragraph 2).
While Hao fails to disclose the further limitations of the claim, Zhao discloses [o]ne or more non-transitory computer-readable media storing instructions executable by one or more processors: “EE22. A non-transitory computer-readable storage medium storing executable instructions thereon, which, when executed, cause the processor to perform the training method of any ofEE1-EE17 and EE19” (Zhao, [0231])
Zhao’s instructions comprising:
(c) extracting a feature representation value for target data through the teacher network and storing the feature representation value in a corresponding class-specific storage of a teacher-network representation memory before learning is entered: “Subsequently, the normal image samples (target data) in the training image set input into the student network
M
S
are also input into the normal image teacher network
M
N
T
to extract the teacher feature
f
N
T
(feature representation) … the teacher features may be extracted and stored before the training (before learning) of the student network
M
S
, for use in the training process of the student network
M
S
” (Zhao, [0085])
(d) entering learning to extract the feature representation value for the target data through the student network and storing the feature representation value in a corresponding class-specific storage of a student-network representation memory: “A training (learning) method of a neural network (student network) for object recognition, comprising: inputting a training image set containing an object to be recognized, which includes a set of normal image samples (target data) and a set of variation image samples, into the neural network to extract a student feature (feature representation) of each of the image samples” (Zhao, [0210])
Zhao relates to knowledge distillation for image-processing neural networks and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hao to extract teacher features before training the student network, as disclosed by Zhao. Doing so would enable the use of teaching features in training the student network. See Zhao, [0085].
While Hao and Zhao fail to disclose the further limitations of the claim, Zoldi discloses a method of (b) generating a representation memory including plural class-specific storages each having a predetermined storage value range to store feature representation values: “To operationalize on a large scale, each variable (class) is discretized into value range (predetermined range) bins. Discretization can be done in one of several ways. In some implementations using equi-sized bins, each variable is divided into decile bins with equal population. In other implementations using entropy bins, each variable is divided into the bins such that the overall entropy is maximized. In all approaches, each bin should have a minimum number of data-points (feature representation values), η, for stable statistics. Bin creation can be part of the model design, and can vary based on application.” (Zoldi, column 5, paragraph 4). The ‘representation memory’ consists of the set of all storage bins across all variables. The set of bins for a variable is a storage, with the representation memory then consisting of a plurality of storages as many as the number of classes.
Zoldi relates to feature selection and storage for machine learning models and is analogous to the claimed invention. The combination of Hao and Zhao teaches a method of transferring knowledge from a teacher neural network to a student network. Zoldi teaches a method of separately storing feature data, organized by classes. It would have been obvious to one of ordinary skill in the art to combine the combination of Hao and Zhao with Zoldi by using separate storages to store the student and teacher feature data of Hao and Zhao. This would achieve the predictable result of storing features from multiple classes concurrently, with Hao and Zhao’s method and Zoldi’s storage performing the same together as they did separately. (MPEP 2143 I. (A) Combining prior art elements according to known methods to yield predictable results).
While Zoldi fails to disclose the further limitations of the claim, Kachare discloses (e) calculating a representation loss using class-wise averaged values stored in the teacher-network representation memory and class-wise averaged values stored in the student-network representation memory:
“The basic idea is to store an approximation of multiple closely-related datasets. Embodiments of the inventive concept may use a neural network-based technique to classify the input datasets into unique buckets or clusters (representation memor[ies]). For each such bucket or cluster, the SSD may store one representative dataset (values stored). When the host performs reads to any of the datasets belonging to a cluster or bucket, the stored single representative dataset may be returned to the host.” (Kachare, [0022])
“Each input dataset is assigned a Class ID, also known as a bucket ID or cluster ID. This Class ID may then be stored in a table against LBA and PBA. For each Class ID, only one dataset (class-wise values) may be stored” (Kachare, [0031])
“In yet other embodiments of the inventive concept, the representative data chunk may be updated in a manner that factors in earlier data written to the class ID. Examples of such techniques may include weighted averaging and centroid calculations. As an example of a weighted averaging, weights may be determined (which may be either specified at the time of manufacture of the storage device and not alterable or configurable by machine 105 of FIG. 1). The most recently received data chunk may be combined with the previous representative data chunk by multiplying the value of the most recently received data chunk by one weight and by multiplying the previous representative data chunk by another weight, and then summing the two products … The result of the sum (class-wise averaged values) of this calculation may then be stored as the new representative data chunk for the class ID” (Kachare, [0081])
Kachare relates to dividing memory into a plurality of class representative storages and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the existing combination to store and perform calculations on average class representative values in lieu of the full sets of student and teacher features, as disclosed by Kachare. Machine learning processes, such as the student-teacher knowledge distillation disclosed by the existing combination, often require very large amounts of data. Kachare’s method can significantly reduce the amount of data needed to be stored, thus providing a larger data storage capacity than would otherwise be possible. See Kachare, [0021-0022], [0030].
Regarding claim 11, Hao discloses [a] learning apparatus comprising:
an input unit comprising an input interface configured to receive at least one task and target data according to the task: “The FRCN branch takes the features and proposals as input (target data) and classifies (at least one task) the proposals whose coordinates are further adjusted” (Hao, page 3, left column, paragraph 3)
a storage unit comprising a memory device including a representation memory formed of plural class-specific storage each having a predetermined range of storable feature representation values: ““The feature before the classifier in the FRCN branch is denoted as
P
f
=
f
L
-
1
R
P
N
(
x
)
” (Hao, page 3, right column, paragraph 3).
f
L
-
1
R
P
N
, an acquired feature representation value, is stored in memory as
P
f
, used for further calculations (as
P
S
and
P
T
).
Hao further discloses instructions, comprising:
(a) generating a teacher network and a student network from a pre-trained model using knowledge distillation: “Given an object detection model that is well-trained with images of certain old classes Co, our goal is to update the model with only images of certain new classes Cn, Co ∩ Cn = ∅” (Hao, page 2, right column, paragraph 2); “As shown in Fig. 1, we propose CIFRCN with the backbone structure of the Faster R-CNN, which contains the following three main components: … (2) the knowledge distillation mechanism in the FRCN branch is applied to preserve the knowledge of the old classes “ (Hao, page 2, right column, paragraph 3); “The self-imitated distillation structure is depicted in Fig. 3. The FRCN (pre-trained model) is also duplicated and frozen as the teacher (teacher network). Then, the output nodes of the classifier are expanded, and the weights of the newly added classes are randomly initialized. The modified network is treated as the student (student network)” (Hao, page 3, left column, paragraph 5)
(b) generating the representation memory to store feature representation values in the teacher network and the student network: “In CIOD task, the RPN is expected to generate proposals containing objects of any classes observed so far … Inspired by Jung et al. [13], our work is designed to maintain the decision boundary of the classifier in RPN to preserve the knowledge gained from the old classes“ (Hao, page 2, right column, paragraph 5); “a feature-changing loss is proposed to reduce the difference of the feature maps between the old and new classes:
PNG
media_image1.png
137
758
media_image1.png
Greyscale
where x is the feature map of the image that is extracted by the feature extractor;
θ
T
R
P
N
and
θ
S
R
P
N
denote the weights of the teacher and student models, respectively;
f
L
-
1
R
P
N
(
∙
)
denotes the feature (feature representation value) before the classifier in RPN” (Hao, page 3, left column, paragraph 2). Feature representation values of both the teacher and student network must be stored in memory (representation memory) at some point to execute this loss equation.
(c) extracting a feature representation value for target data through the teacher network and storing the feature representation value in a corresponding class-specific storage of a teacher-network representation memory before learning is entered: “a feature-changing loss is proposed to reduce the difference of the feature maps between the old and new classes:
PNG
media_image1.png
137
758
media_image1.png
Greyscale
where x is the feature map of the image (target data) that is extracted by the feature extractor;
θ
T
R
P
N
and
θ
S
R
P
N
denote the weights of the teacher and student models, respectively;
f
L
-
1
R
P
N
(
∙
)
denotes the feature before the classifier in RPN” (Hao, page 3, left column, paragraph 2).
(d) entering the learning to extract the feature representation value for the target data through the student network and storing the feature representation value in a corresponding class-specific storage of a student-network representation memory:
PNG
media_image1.png
137
758
media_image1.png
Greyscale
where x is the feature map of the image (target data) that is extracted by the feature extractor;
θ
T
R
P
N
and
θ
S
R
P
N
denote the weights of the teacher and student models, respectively;
f
L
-
1
R
P
N
(
∙
)
denotes the feature before the classifier in RPN” (Hao, page 3, left column, paragraph 2).
(e) calculating a representation loss based on mean-square differences between class-wise averaged values stored in the teacher-network representation memory and class-wise averaged values stored in the student-network representation memory: “a feature-changing loss (representation loss) is proposed to reduce the difference of the feature maps between the old and new classes:
PNG
media_image1.png
137
758
media_image1.png
Greyscale
where x is the feature map of the image that is extracted by the feature extractor;
θ
T
R
P
N
and
θ
S
R
P
N
denote the weights of the teacher and student models, respectively;
f
L
-
1
R
P
N
(
∙
)
denotes the feature before the classifier in RPN;
J
(
∙
)
measures the mean squared error (MSE)” (Hao, page 3, left column, paragraph 2).
Hao relates to continuous learning through knowledge distillation of neural networks and is analogous to the claimed invention.
While Hao fails to disclose the further limitations of the claim, Zhao discloses a processor configured to execute a program for performing continual learning using the representation memory, wherein the program executed by the processor includes instructions: “EE22. A non-transitory computer-readable storage medium storing executable instructions thereon, which, when executed, cause the processor to perform the training method of any ofEE1-EE17 and EE19” (Zhao, [0231])
Zhao’s instructions comprising:
(c) extracting a feature representation value for target data through the teacher network and storing the feature representation value in a corresponding one of class-specific storage of a teacher-network representation before learning is entered: “Subsequently, the normal image samples (target data) in the training image set input into the student network
M
S
are also input into the normal image teacher network
M
N
T
to extract the teacher feature
f
N
T
(feature representation) … the teacher features may be extracted and stored before the training (before learning) of the student network
M
S
, for use in the training process of the student network
M
S
” (Zhao, [0085])
(d) entering learning to extract the feature representation value for the target data through the student network and storing the feature representation value in a corresponding class-specific storage of a student-network representation memory: “A training (learning) method of a neural network (student network) for object recognition, comprising: inputting a training image set containing an object to be recognized, which includes a set of normal image samples (target data) and a set of variation image samples, into the neural network to extract a student feature (feature representation) of each of the image samples” (Zhao, [0210])
Zhao relates to knowledge distillation for image-processing neural networks and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hao to extract teacher features before training the student network, as disclosed by Zhao. Doing so would enable the use of teaching features in training the student network. See Zhao, [0085].
While Hao and Zhao fail to disclose the further limitations of the claim, Zoldi discloses a storage unit comprising a memory device including a representation memory formed of plural class-specific storages each having a predetermined range of storable feature representation values: “To operationalize on a large scale, each variable (class) is discretized into value range bins. Discretization can be done in one of several ways. In some implementations using equi-sized bins, each variable is divided into decile bins with equal population. In other implementations using entropy bins, each variable is divided into the bins such that the overall entropy is maximized. In all approaches, each bin should have a minimum number of data-points (feature representation values), η, for stable statistics. Bin creation can be part of the model design, and can vary based on application.” (Zoldi, column 5, paragraph 4). The ‘representation memory’ consists of the set of all storage bins across all variables. The set of bins for a variable is a storage, with the representation memory then consisting of plural class-specific storages.
Zoldi relates to feature selection and storage for machine learning models and is analogous to the claimed invention. The combination of Hao and Zhao teaches a method of transferring knowledge from a teacher neural network to a student network. Zoldi teaches a method of separately storing feature data, organized by classes. It would have been obvious to one of ordinary skill in the art to combine the combination of Hao and Zhao with Zoldi by using separate storages to store the student and teacher feature data of Hao and Zhao. This would achieve the predictable result of storing features from multiple classes concurrently, with Hao and Zhao’s method and Zoldi’s storage performing the same together as they did separately. (MPEP 2143 I. (A) Combining prior art elements according to known methods to yield predictable results).
While Zoldi fails to disclose the further limitations of the claim, Kachare discloses (e) calculating a representation loss based on mean-square differences between class-wise averaged values stored in the teacher-network representation memory and class-wise averaged values stored in the student-network representation memory:
“The basic idea is to store an approximation of multiple closely-related datasets. Embodiments of the inventive concept may use a neural network-based technique to classify the input datasets into unique buckets or clusters (representation memor[ies]). For each such bucket or cluster, the SSD may store one representative dataset (values stored). When the host performs reads to any of the datasets belonging to a cluster or bucket, the stored single representative dataset may be returned to the host.” (Kachare, [0022])
“Each input dataset is assigned a Class ID, also known as a bucket ID or cluster ID. This Class ID may then be stored in a table against LBA and PBA. For each Class ID, only one dataset (class-wise values) may be stored” (Kachare, [0031])
“In yet other embodiments of the inventive concept, the representative data chunk may be updated in a manner that factors in earlier data written to the class ID. Examples of such techniques may include weighted averaging and centroid calculations. As an example of a weighted averaging, weights may be determined (which may be either specified at the time of manufacture of the storage device and not alterable or configurable by machine 105 of FIG. 1). The most recently received data chunk may be combined with the previous representative data chunk by multiplying the value of the most recently received data chunk by one weight and by multiplying the previous representative data chunk by another weight, and then summing the two products … The result of the sum (class-wise averaged values) of this calculation may then be stored as the new representative data chunk for the class ID” (Kachare, [0081])
Kachare relates to dividing memory into a plurality of class representative storages and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the existing combination to store and perform calculations on average class representative values in lieu of the full sets of student and teacher features, as disclosed by Kachare. Machine learning processes, such as the student-teacher knowledge distillation disclosed by the existing combination, often require very large amounts of data. Kachare’s method can significantly reduce the amount of data needed to be stored, thus providing a larger data storage capacity than would otherwise be possible. See Kachare, [0021-0022], [0030].
The analysis of claims 12-13, 17, and 19 mirrors that of claims 2-3, 7, and 9, with the exception that claims 12-13, 17, and 19 are directed to generic computer hardware which executes the methods of claims 2-3, 7, and 9 and generic units for input and storage. The generic units for input and storage are taught by Hao, while the generic hardware is taught by Zhao, as discussed regarding claim 11. Thus, claims 12-13, 17, and 19 are rejected under the same rationales used for claims 2-3, 7, and 9, respectively.
Claims 4-5 and 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Hao et al. (AN END-TO-END ARCHITECTURE FOR CLASS-INCREMENTAL OBJECT DETECTION WITH KNOWLEDGE DISTILLATION, published 2019, 2019 IEEE International Conference on Multimedia and Expo (ICME)), hereafter referred to as Hao, in view of Zhao et al. (TRAINING METHOD AND TRAINING APPARATUS FOR A NEURAL NETWORK FOR OBJECT RECOGNITION, filed 11/4/2021, US 20220138454 A1), hereafter referred to as Zhao, and further in view of Zoldi et al. (CLASSIFICATION APPARATUS, ROBOT, AND CLASSIFICATION METHOD, published 5/18/2017, US 20170140300 A1), hereafter referred to as Zoldi, Kachare et al. (DYNAMIC QUANTIZATION IN STORAGE DEVICES USING MACHINE LEARNING, published 9/30/2021, US 2021/0303156 A1), hereafter referred to as Kachare, and Ha (SYSTEM METHOD AND COMPUTER-ACCESSIBLE MEDIUM FOR CLASSIFYING TISSUE USING AT LEAST ONE CONVOLUTIONAL NEURAL NETWORK, published 11/26/2020, US 20200372637 A1).
Regarding claim 4, the rejection of claim 1 in view of Hao, Zhao, Zoldi, and Kachare is incorporated. Hao further discloses a method, wherein the step (c) includes:
(c1) receiving the target data and inputting the target data to the teacher network:
PNG
media_image2.png
450
1062
media_image2.png
Greyscale
(Hao, page 2, right column, Fig. 1); “The teacher and student are simultaneously fed the same images (target data) of the new classes, and the class predictions are recorded as
P
T
and
P
S
, individually.” (Hao, page 3, right column, paragraph 1).
(c2) applying a softmax function to an output value to determine a value in a predetermined range: “
P
S
T
is a part of
P
S
and is the prediction on old classes made by the student.
y
o
'
and
y
n
'
are the transformed form of
P
T
and
P
S
T
(output value[s]), called “soft labels”. The transformer is a modified softmax function in which:
PNG
media_image3.png
89
343
media_image3.png
Greyscale
” (Hao, page 3, right column, paragraph 2). All softmax outputs are within the predetermined range of [0, 1].
(c3) extracting a feature map belonging to a last layer of the teacher network when the input target data is inferred as ground truth:
“a feature-changing loss is proposed to reduce the difference of the feature maps between the old and new classes:
PNG
media_image1.png
137
758
media_image1.png
Greyscale
where x is the feature map of the image (input target data) that is extracted by the feature extractor;
θ
T
R
P
N
and
θ
S
R
P
N
denote the weights of the teacher and student models, respectively;
f
L
-
1
R
P
N
(
∙
)
denotes the feature (feature map) before the classifier in RPN” (Hao, page 3, left column, paragraph 2).
“The feature before the classifier in the FRCN branch is denoted as
P
f
=
f
L
-
1
R
P
N
(
x
)
, and is normalized using
f
=
ς
P
f
=
P
f
|
|
P
f
|
|
.” (Hao, page 3, right column, paragraph 3)
“During training, the normalized feature f of each proposal is recorded and
μ
y
←
1
|
y
c
=
y
|
∑
ς
(
f
|
y
c
=
y
)
is used to obtain the prototype
μ
y
of class y, where
y
c
is the ground truth class of the proposal. For class y ∈
C
n
,
μ
y
is added to the classifier to expand its ability during class-incremental training” (Hao, page 3, right column, paragraph 4).
While Hao fails to disclose the further limitations of the claim, Zoldi further discloses a method, wherein the step (c) includes: … (c5) storing the acquired feature representation value in the storage corresponding to the determined value in the predetermined range: “Each feature is split into value ranges, called “bins,” and a customer (feature representation) is assigned to one of these bins for each feature” (Zoldi, column 5, paragraph 1).
Zoldi relates to feature selection and storage for machine learning models and is analogous to the claimed invention. The combination of Hao, Zhao, Zoldi, and Kachare teaches a method of transferring knowledge from a teacher neural network to a student network. The claimed invention improves upon this method by storing feature values in ranged buckets. Zoldi teaches a method of storing feature values in ranged buckets, applicable to the combination of Hao, Zhao, Zoldi, and Kachare. A person of ordinary skill in the art would have recognized that creating ranged storage buckets would lead to the predictable result of discretizing continuous feature values, and would improve the known device by allowing it to operate on continuous variables (MPEP 2143 I. (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results).
While Hao, Zhao, Zoldi, and Kachare fail to disclose the further limitations of the claim, Ha discloses a method, comprising: (c4) calculating an average value of a feature map pooled by applying max pooling from the extracted feature map to acquire the feature representation value: “Downsampling of feature map size was implemented using a concatenated average and max pooling operation to decrease size by 75%.” (Ha, [0076])
Ha relates to processing image data feature maps and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Hao, Zhao, Zoldi, and Kachare to use max pooled and averaged feature maps in lieu of raw feature maps, as disclosed by Ha. Doing so would significantly reduce the size of the features, by 75% in Ha’s case. See Ha, [0076].
Regarding claim 5, the rejection of claim 4 in view of Hao, Zhao, Zoldi, Kachare, and Ha is incorporated. Hao further discloses a method, wherein the step (c3) includes using a property that a portion similar to a feature of a source domain is present in data predicted as ground truth even when the input target data is different from a domain trained by the teacher network:
“a feature-changing loss is proposed to reduce the difference of the feature maps between the old (source domain data) and new classes (target data):
PNG
media_image1.png
137
758
media_image1.png
Greyscale
where x is the feature map of the image that is extracted by the feature extractor;
θ
T
R
P
N
and
θ
S
R
P
N
denote the weights of the teacher and student models, respectively;
f
L
-
1
R
P
N
(
∙
)
denotes the feature before the classifier in RPN;
J
(
∙
)
measures the mean squared error (MSE); and
γ
R
P
N
is used to balance the losses” (Hao, page 3, left column, paragraph 2). As one of ordinary skill in the art would know, by minimizing this loss, features in the student network are pushed to be similar to features in the teacher network given the same input image(s).
“Then, the model is trained by training data of group A. For the following three incremental training sessions, the model from the last session is loaded, and a fixed duplication of the model as the teacher is made. The output nodes in FRCN are then extended to the un-fixed one as the student. After that, the student net is trained as our proposed method by the training data of classes in group B, C, and D, individually” (Hao, page 4, right column, paragraph 4). An explicit example of the teacher being trained on a source domain (group A), and a student being trained on a different target domain (groups B, C, and D).
The analysis of claims 14-15 mirrors that of claims 4-5, with the exception that claims 14-15 are directed to generic computer hardware which executes the methods of claims 4-5 and generic units for input and storage. The generic units for input and storage are taught by Hao, while the generic hardware is taught by Zhao, as discussed regarding claim 11. Thus, claims 14-15 are rejected under the same rationales used for claims 4-5, respectively.
Claims 6, 8, 16, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Hao et al. (AN END-TO-END ARCHITECTURE FOR CLASS-INCREMENTAL OBJECT DETECTION WITH KNOWLEDGE DISTILLATION, published 2019, 2019 IEEE International Conference on Multimedia and Expo (ICME)), hereafter referred to as Hao, in view of Zhao et al. (TRAINING METHOD AND TRAINING APPARATUS FOR A NEURAL NETWORK FOR OBJECT RECOGNITION, filed 11/4/2021, US 20220138454 A1), hereafter referred to as Zhao, and further in view of Zoldi et al. (CLASSIFICATION APPARATUS, ROBOT, AND CLASSIFICATION METHOD, published 5/18/2017, US 20170140300 A1), hereafter referred to as Zoldi, Kachare et al. (DYNAMIC QUANTIZATION IN STORAGE DEVICES USING MACHINE LEARNING, published 9/30/2021, US 2021/0303156 A1), hereafter referred to as Kachare, Ha (SYSTEM METHOD AND COMPUTER-ACCESSIBLE MEDIUM FOR CLASSIFYING TISSUE USING AT LEAST ONE CONVOLUTIONAL NEURAL NETWORK, published 11/26/2020, US 20200372637 A1), and Marra et al. (Incremental learning for the detection and classification of GAN-generated images, published 10/6/2019, arXiv:1910.01568v2), hereafter referred to as Marra.
Regarding claim 6, the rejection of claim 4 in view of Hao, Zhao, Zoldi, Kachare, and Ha is incorporated. While Hao, Zhao, and Ha fail to disclose the further limitations of the claim, Marra discloses a method, comprising: (c6) calculating respective average values of the feature representation values stored in the storage and updating all the storages of the teacher network, when all the pieces of input target data have been traversed:
“Initialization: let s classes be given, with associated training sets X1, … , Xs (input target data). A CNN is trained on such data to minimize the classification error. When the training is over, the output of a suitable layer is used to associate a unit-norm feature vector, ϕ(x) (feature representation value), with each input image x. For each class, y = 1, …, s, M / s images are selected to form the class-y exemplar set, Py, and a class template vector is computed as the average of the corresponding feature vectors
PNG
media_image5.png
92
286
media_image5.png
Greyscale
” (Marra, page 2, right column, paragraph 2).
“Updating: let t - s new classes be given, with associated training sets Xs+1, … , Xt. iCaRL performs a three-step updating procedure:
1) the weights of the CNN are updated, using only the new training sets and the set of exemplar images P;
2) exemplar sets are created for each of the new classes;
3) a suitable number of images are discarded from old exemplar sets to keep a fixed memory budget M.” (Marra, page 2, right column, paragraph 3). Network storage for the weights and exemplars used to calculate averages are updated.
“
g
y
(
∙
)
is the classification score for class y. The distillation term, is the KL-divergence loss with temperature T, as proposed in [22]:
PNG
media_image6.png
111
871
media_image6.png
Greyscale
(5) where
g
~
T
(
∙
)
is the old classifier, that is, before the current updating phase” (Marra, page 2, right column, paragraph 4). The old classifier, whose storages are updated to form the new classifier, is a teacher network.
Marra relates to continuous learning with knowledge distillation and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Hao, Zhao, Zoldi, Kachare, and Ha to use the iCaRL method for storing and updating class exemplars / averages computed by said exemplars, as disclosed by Marra. iCaRL allows for continuously learning new classes a model isn’t initially trained on, a key aspect of continuous learning, without forgetting information learned about prior classes, all done in a memory-efficient manner. See Marra, page 2, left column, paragraphs 3-4; page 2, right column, paragraph 1; and page 3, left column, paragraph 4.
The analysis of claim 16 mirrors that of claim 6, with the exception that claim 16 is directed to generic computer hardware which executes the methods of claim 6 and generic units for input and storage. The generic units for input and storage are taught by Hao, while the generic hardware is taught by Zhao, as discussed regarding claim 11. Thus, claim 16 is rejected under the same rationale used for claim 6.
Claims 8 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Hao et al. (AN END-TO-END ARCHITECTURE FOR CLASS-INCREMENTAL OBJECT DETECTION WITH KNOWLEDGE DISTILLATION, published 2019, 2019 IEEE International Conference on Multimedia and Expo (ICME)), hereafter referred to as Hao, in view of Zhao et al. (TRAINING METHOD AND TRAINING APPARATUS FOR A NEURAL NETWORK FOR OBJECT RECOGNITION, filed 11/4/2021, US 20220138454 A1), hereafter referred to as Zhao, and further in view of Zoldi et al. (CLASSIFICATION APPARATUS, ROBOT, AND CLASSIFICATION METHOD, published 5/18/2017, US 20170140300 A1), hereafter referred to as Zoldi, Kachare et al. (DYNAMIC QUANTIZATION IN STORAGE DEVICES USING MACHINE LEARNING, published 9/30/2021, US 2021/0303156 A1), hereafter referred to as Kachare, and Marra et al. (Incremental learning for the detection and classification of GAN-generated images, published 10/6/2019, arXiv:1910.01568v2), hereafter referred to as Marra.
Regarding claim 8, the rejection of claim 7 in view of Hao, Zhao, Zoldi, and Kachare is incorporated. While Hao, Zhao, Zoldi, and Kachare fail to disclose the further limitations of the claim, Marra discloses a method, comprising: (d6) calculating average values of the feature representation values stored in the storage to update all the storage of the student network, when all the pieces of input target data have been traversed:
“Initialization: let s classes be given, with associated training sets X1, … , Xs (input target data). A CNN is trained on such data to minimize the classification error. When the training is over, the output of a suitable layer is used to associate a unit-norm feature vector, ϕ(x) (feature representation value), with each input image x. For each class, y = 1, …, s, M / s images are selected to form the class-y exemplar set, Py, and a class template vector is computed as the average of the corresponding feature vectors
PNG
media_image5.png
92
286
media_image5.png
Greyscale
” (Marra, page 2, right column, paragraph 2).
“Updating: let t - s new classes be given, with associated training sets Xs+1, … , Xt. iCaRL performs a three-step updating procedure:
1) the weights of the CNN are updated, using only the new training sets and the set of exemplar images P;
2) exemplar sets are created for each of the new classes;
3) a suitable number of images are discarded from old exemplar sets to keep a fixed memory budget M.” (Marra, page 2, right column, paragraph 3). Network storage for the weights and exemplars used to calculate averages are updated.
“
g
y
(
∙
)
is the classification score for class y. The distillation term, is the KL-divergence loss with temperature T, as proposed in [22]:
PNG
media_image6.png
111
871
media_image6.png
Greyscale
(5) where
g
~
T
(
∙
)
is the old classifier, that is, before the current updating phase” (Marra, page 2, right column, paragraph 4). The new classifier, distilled from the old, is a student network with fully updated storages.
Marra relates to continuous learning with knowledge distillation and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the existing combination to use the iCaRL method for storing and updating class exemplars / averages computed by said exemplars, as disclosed by Marra. iCaRL allows for continuously learning new classes a model isn’t initially trained on, a key aspect of continuous learning, without forgetting information learned about prior classes, all done in a memory-efficient manner. See Marra, page 2, left column, paragraphs 3-4; page 2, right column, paragraph 1; and page 3, left column, paragraph 4.
The analysis of claim 18 mirrors that of claim 8, with the exception that claim 18 is directed to generic computer hardware which executes the methods of claim 8 and generic units for input and storage. The generic units for input and storage are taught by Hao, while the generic hardware is taught by Zhao, as discussed regarding claim 11. Thus, claim 18 is rejected under the same rationale used for claim 8.
Response to Arguments
The following responses address arguments and remarks made in the instant remarks dated 02/27/2026.
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Objections
In light of the instant amendments, objections to the specification, abstract, drawings, and claims have been withdrawn.
112(f) Interpretation and 112 Rejections
In light of the instant amendments, the claims are no longer interpreted under 35 U.S.C. 112(f).
Additionally, rejections under 35 U.S.C. 112(a) and 112(b) have been withdrawn.
101 Rejections
On page 20 of the instant remarks, the Applicant argues that the claimed invention is practically integrated through improvement to a technological field:
“The claims improve a technological field - continual learning - The amendments
clarify that the invention solves catastrophic forgetting using a novel partitioned
representation memory architecture; teacher-student dual-memory system; and class-wise
averaged representation-loss mechanism.
This constitutes a specific technological improvement, consistent with (1) Enfish v.
Microsoft (improved data structure); (2) McRO v. Bandai (specific rules improving computer
operation); and (3) Finjan v. Blue Coat (non-abstract security data structures).”
The Applicant’s arguments above, in light of the amendments and paragraphs [4-9] and [31-36] of the instant specification describing technical problems overcome by the claimed invention, have been fully considered and are persuasive. The independent claims practically integrate their recited judicial exceptions through improvements to existing technology. Thus, previous rejections under 35 U.S.C. 101 have been withdrawn.
103 Rejections
On pages 22-26 of the instant remarks, the Applicant argues that the amended claims are not fully disclosed by the cited references:
“The cited references do not teach or suggest:
(1) A representation memory including a plurality of storages respectively
corresponding to classes AND each having a predetermined range of storable feature
representation values;
(2) Teacher-network representation memory distinct from student-network
representation memory;
(3) Storing feature representation values in corresponding class-specific storages both
before and after learning is entered;
(4) Computing representation loss based on mean-square differences between class-wise
averaged values stored in two different representation memories; and
(5) Using class-based memory ranges as part of continual learning without source
domain data.
None of these features are taught or suggested in any cited reference alone or in
combination. Thus, the rejection cannot stand.
…
Amended claim 1 recites: "a representation memory including a plurality of storages
respectively corresponding to classes and each having a predetermined range of storable
feature representation values."
None of Hao, Zhao, Zaidi, Ha, nor Marra teaches this.
…
Claim 1 requires two separate structural memories (1) teacher-network representation
memory; and (2) student-network representation memory.
None of the cited references disclose or suggest dual distinct memories
…
No reference teaches or suggests storing teacher representations in a first memory
and student representations in a second memory, each with class-specific storable ranges
…
Claim 1 (e) recites: "calculating ... a representation loss based on mean-square
differences between class-wise averaged values stored in the teacher-network
representation memory and class-wise averaged values stored in the student-network
representation memory
This is a key inventive concept of the CoReD system. No reference performs this
operation.”
In response to the Applicant's argument that the individual references to disclose limitations of amended claim 1, the Examiner notes that one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Regarding “(b) generating, by the learning apparatus, a representation memory including a plurality of storages respectively corresponding to classes and each having a predetermined range of storable feature representation values to store feature representation values in the teacher network and the student network”, the second limitation of amended claim 1, Hao discloses a method of storing feature representation values for the teacher network and the student network (Hao, page 3, left column, paragraph 2). While Hao fails to disclose the further aspects of this limitation, this deficiency is remedied by Zoldi, which discloses storing data in a plurality of storage buckets, each corresponding to a class, and each having a predetermined range of values (Zoldi, column 5, paragraph 4).
Regarding “(c) extracting, by the learning apparatus, a feature representation value for target data through the teacher network and storing the feature representation value in a corresponding one of the class-specific storages of a teacher-network representation memory before learning is entered”, Hao discloses extracting feature representations from teacher network after feeding it target data and storing them (Hao, page 3, left column, paragraph 2). While Hao fails to disclose storing values in class-specific storages or performing this extraction before learning is entered, these deficiencies are remedied by Zhao and Zoldi. Zoldi discloses class-specific storages, as discussed above. Zhao discloses extracting feature representations from the teacher network and storing them in memory, explicitly before training is performed (Zhao, [0085]).
Regarding “(d) entering, by the learning apparatus, learning to extract the feature representation value for the target data through the student network and storing the feature representation value in a corresponding one of the class-specific storages of a student-network representation memory”, Hao discloses extracting feature representations from student network after feeding it target data storing them (Hao, page 3, left column, paragraph 2). While Hao fails to disclose storing values in class-specific storages or performing this explicitly as part of the learning process, these deficiencies are remedied by Zhao and Zoldi. Zoldi discloses class-specific storages, as discussed above. Zhao discloses extracting feature representations from the student network and storing them in memory, through learning of the model, explicitly as part of the learning process (Zhao, [0085], [0210]).
Regarding “(e) calculating, by the learning apparatus, a representation loss based on mean-square differences between class-wise averaged values stored in the teacher-network representation memory and class-wise averaged values stored in the student-network representation memory”, Hao discloses calculating the mean (average) squared error of teacher network values and student network values (Hao, page 3, left column, paragraph 2). While Hao fails to disclose computing class-wise averages for these values, this deficiency is remedied by Kachare, which discloses storing and retrieving class-wise averaged values in lieu of individual values from a class (Kachare, [0022], [0031], [0081]).
Thus, the limitations of amended claim 1 are taught by the combination of Hao, Zhao, Zoldi, and Kachare. Similar reasoning is applicable to substantially similar independent claims 10 and 11. Upon further search and consideration, claims 1-19 are found to be obvious in view of Hao, Zhao, Zoldi, Kachare, Ha, and Marra. See the 103 rejections section for more detail. No rejections are withdrawn on these grounds.
On pages 25 of the instant remarks, the Applicant argues that the claimed limitations require a specific order of operations not represented by the cited references:
“The Examiner treats the limitations as modular or interchangeable. They are not.
The claims require a specific ordered sequence: (1) Generate teacher/student networks;
(2) Create partitioned representation memory; (3) Extract teacher-side feature values before
learning; (4) Extract student-side feature values after learning enters; and (5) Compute
class-wise averaged representation loss.
This ordering is: (1) never taught by Hao; (2) contradicted by Zhao (domain adaptation);
(3) absent from Marra (distillation); and (4) absent from Ha and Zaidi.
The art does not recognize the necessity of separating teacher and student
representations into partitioned storage for continual learning.”
In response to applicant's arguments above, it is noted that the specific ordering upon which the Applicant relies is not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification or arguments are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
The only explicit ordering specified by the claims is performing step (c) “before learning is entered”, which is disclosed by the relied upon references, as discussed in previous responses to arguments above. Thus, the Examiner respectfully disagrees with the assertion that the claims require the argued ordered sequence, or that the relied upon references fail to disclose the sequence disclosed by the claims. No rejections are withdrawn on this basis.
On page 26 of the instant remarks, the Applicant argues that there’s no sufficient motivation to combine the cited references:
“NO MOTIVATION TO COMBINE:
Even if the references disclosed isolated pieces (they do not), the proposed
combination lacks motivation: (1) The cited art does not recognize catastrophic forgetting as
solved by class-partitioned representation memories; (2) The cited art does not suggest dual-memory
architectures; (3) The cited art does not suggest predetermined ranges for storable
values; (4) The cited art does not suggest class-wise averaged representation-loss
operations; and (5) The cited art does not use memory structures in continual learning
systems.
A POSIT A would have no reason to create such a memory structure because the cited
references never hint at this architecture or its benefits.”
In response to applicant's arguments above, it is noted that the catastrophic forgetting and dual-memory architectures upon which the Applicant relies are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
As described in previous responses above, Zoldi discloses “a representation memory including a plurality of storages respectively corresponding to classes and each having a predetermined range of storable feature representation values to store feature representation values”. Combining Zoldi’s storage with the teacher-student learning method of the existing combination would have been obvious to one of ordinary skill in the art, as this would achieve the predictable result of storing features from different classes concurrently, with the elements of Zoldi and the existing combination performing the same combined as they would separately (MPEP 2143 I. (A) Combining prior art elements according to known methods to yield predictable results).
As noted in previous responses above, Kachare discloses “class-wise averaged values stored in the representation memory”. One of ordinary skill in the art would have been motivated to combine Kachare with Hao’s loss (and the rest of the existing combination), as Kachare explicitly describes the benefits of its storage method, stating that it can significantly reduce the needed storage for a large dataset, something often needed by machine learning processes (Kachare, [0021-0022], [0030]).
As noted in previous responses above, Hao discloses storing data for the teacher and student networks in memory. No motivation is needed to utilize the primary reference in 35 U.S.C. 103 analysis.
Motivation to combine every secondary reference under 35 U.S.C. 103 analysis is provided for each claim reciting secondary references. Thus, no rejections are withdrawn on these grounds.
On pages 26-27 of the instant remarks, the Applicant argues that the claimed invention produces unexpected results not predictable from the relied upon references:
“The combination of: class-specific memory partitions; predetermined storable ranges;
teacher/student dual-memory storage; and class-wise averaged representation-loss
computation produces a new and unexpected result: significant reduction in
catastrophic forgetting without storing source-domain training data.
This result is: not predictable from Hao; not suggested by Zhao; not achievable using
Marra's distillation; and not remotely implicated by Zaidi or Ha
Under Ex parte Rubin, In re Burhans, and In re Gibson, such unexpected results
overcome prima facie obviousness.”
Regarding the Applicant’s arguments above, the Examiner respectfully disagrees. “without storing source-domain training data” is not a result, it’s a facet of the method not represented by the claim language. The result of “significant reduction in catastrophic forgetting” would not be unexpected to one of ordinary skill in the art in view of Hao, Zhao, Zoldi, Kachare, Ha, and Marra. Hao explicitly discloses that avoiding catastrophic forgetting is one of the primary goals of its method (Hao, page 2, left column, paragraph 2). Avoiding forgetting of network information is also explicitly described by Zhao (Zhao, [0108]) and Marra (Marra, page 2, left column, paragraph 3). While this might not be mentioned explicitly by other cited references, it would have been obvious that the cited modifications to Hao’s catastrophic forgetting remediation system would maintain the result of reducing catastrophic forgetting. Thus, no rejections are withdrawn on these grounds.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Rebuffi et al. (iCaRL: Incremental Classifier and Representation Learning, published 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition pp. 5533 – 5542) discloses a method of incremental learning through prototypical feature representations
Jeon et al. (T-GD: Transferable GAN-generated Images Detection Framework, published 8/10/2020, arXiv:2008.04115v1) discloses a method of using knowledge distillation to produce a domain-adapted GAN
Bengio et al. (Representation Learning: A Review and New Perspectives, published 2014, arXiv:1206.5538v3) discloses a survey of methods for CNN representation learning
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Aaron P Gormley whose telephone number is (571)272-1372. The examiner can normally be reached Monday - Friday 12:00 PM - 8:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle T Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AG/Examiner, Art Unit 2148 /MICHELLE T BECHTOLD/Supervisory Patent Examiner, Art Unit 2148
Prosecution Insights