Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-5, 7-12, and 14-19 are pending. Claims 6, 13 and 20 have been cancelled. This Office Action is responsive to the amendment filed on 06/13/2025, which has been entered in the above identified application.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1, 8 and 15 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1, 8 and 15 recites the limitations "assigning one task to each group of the N groups of convolutional layers" and “manipulating gradients of the N groups of convolutional layers”. There is insufficient antecedent basis for these limitations in the claim. At best, the previous limitation recites dividing filters of deeper convolutional layers into N groups, not dividing the convolutional layers themselves into N groups. For the purposes of examination, these limitations will be interpreted as “assigning one task to each group of the N groups of filters of convolutional layers” and “manipulating gradients of the N groups of filters of convolutional layers” respectively.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-5, 7-12, and 14-19 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Independent Claims
Step 1 – Claim 1 is drawn to a method, claim 8 is drawn to a non-transitory computer-readable storage medium, and claim 15 is drawn to a system. Therefore, each of these claims fall under one of the four categories of statutory subject matter (process/method, machine/product/apparatus, manufacture or composition of matter).
Step 2A Prong 1 – Claims 1, 8 and 15 are directed to a judicially recognized exception of an abstract idea without significantly more. Claims 1, 8 and 15 recite a method comprising:
training a neural network model that alleviates a gradient conflict resulting from performing the visual human analysis tasks simultaneously on an image using the neural network model – This limitation is directed towards the abstract idea of a mathematical calculation (see MPEP § 2106.04(a)(2), section I, C). In Paragraph [00025] of the specification, it states “the exemplary methods propose a training scheme dubbed Gradient Split (or GradSplit) that enables each task to learn its essential features without interference from other tasks. Instead of using each task loss to update all filters of convolution in the shared backbone, GradSplit explicitly makes it only impact a subset of the filters.” BRI in light of the specification would support that “training a neural network” would encompass parameter updating based on loss calculations and fall under the mathematical concepts grouping.
during backpropagation, dividing filters of deeper layers of convolutional layers of the feature extractor into N groups, N being a number of tasks – This limitation recites the abstract idea of a mathematical relationship, specifically organizing information and manipulating information through mathematical correlations (see MPEP § 2106.04(a)(2), section I, A). In Paragraph [00063] of the specification, it states “During back-propagation, in the GradSplit model310, the exemplary methods uniformly divide the weights into N = 2 groups. Thus, each task loss only influences one specific filter group. The first filter group, G1, includes the bottom weights or bottom group only (horizontally aligned with designation G1), whereas the second filter group, G2, includes the top weights ot top group only (horizontally aligned with designation G2).” BRI in light of the specification and the plain meaning of “backpropagation” would support that “dividing filters” would encompass organizing neural network parameters into equal groups based on associated tasks for gradient computation and fall under the mathematical concepts grouping.
assigning one task to each group of the N groups of convolutional layers – This limitation is directed towards the abstract idea of a mental process, or a concept that can be performed in the human mind, including observation, evaluation, judgement or opinion (see MPEP § 2106.04(a)(2), subsection III, C). In Paragraph [00067] of the specification, it states “To better understand the training procedure, consider a system that has 10 filters in the last or deepest layer of the feature extractor when the tasks are A and B. A conventional training algorithm updates 10 filters to minimize the sum of losses of tasks A and B. The exemplary method, however, updates the first 5 filters to minimize the loss of task A and updates the remaining 5 filters to minimize the task B loss. This makes the first 5 filters to predict the features specifically required for task A.” BRI in light of the specification would support that “assigning tasks” would encompass a mental process with or without the aid of pen and paper of evenly distributing resources amongst a group of recipients.
manipulating gradients of the N groups of convolutional layers so that each task loss updates only one subset of filters associated with one task assigned to the N groups by discarding gradients of filters associated with other tasks – This limitation is directed towards the abstract idea of a mathematical calculation (see MPEP § 2106.04(a)(2), section I, C). In Paragraph [00031] of the specification, it states “Regarding intuitive understanding of GradSplit as regularization, consider manipulating gradients with respect to θt as a weighted linear sum of task gradients:
m
t
∘
∇
θ
t
L
t
+
∑
t
'
≠
t
m
t
'
∇
θ
t
L
t
'
. When mt = 1 and mt’ = 0 (t ≠ t’), the above equation become
∇
θ
t
G
S
L
. When mt is a probabilistic binary mask, it is equivalent to dropping-out gradients.” BRI in light of the specification would support that “manipulating gradients” as claimed would fall within the mathematical calculations grouping, and therefore, the claim recites the abstract idea of a mathematical concept.
Step 2A Prong 2 – The following additional limitations are recited:
extracting images from training data having a plurality of datasets – This limitation recites an insignificant extra-solution activity of mere data gathering (see MPEP § 2106.05(g)) and thus, fails to integrate the exception into a practical application.
each dataset associated with one task from a plurality of visual human analysis tasks that utilize identity-variant attributes and identity-invariant features – This limitation recites the insignificant extra-solution activity of selecting a particular data source or type of data to be manipulated (see MPEP § 2106.05(g)) and thus, fails to integrate the exception into a practical application.
the neural network model including a feature extractor and task-specific heads, wherein the feature extractor has a feature extractor shared component and a feature extractor task-specific component – This limitation amounts to no more than generally linking the use of the judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h)). It merely limits the use of the abstract idea to ---a multi-task model with shared and task-specific components and thus, fails to integrate the exception into a practical application.
Step 2B – The additional elements in Step 2A Prong 2, view individually or wholistically, do not provide an inventive concept or otherwise amount to significantly more than the abstract idea itself.
extracting images from training data having a plurality of datasets – This limitation is directed towards an insignificant extra-solution activity of mere data gathering (see MPEP § 2106.05(g)), which is well-understood, routine and conventional activity similar to cases reviewed by the courts involving storing and retrieving information from memory ((see MPEP § 2106.05(d)(II)) and thus, fails to provide significantly more to the judicial exception.
each dataset associated with one task from a plurality of visual human analysis tasks that utilize identity-variant attributes and identity-invariant features – This limitation recites the insignificant extra-solution activity of selecting a particular data source or type of data to be manipulated (see MPEP § 2106.05(g)), which is well-understood, routine and conventional activity similar to cases reviewed by the courts involving electronically scanning or extracting data from a physical document ((see MPEP § 2106.05(d)(II)) and thus, fails to provide significantly more to the judicial exception.
the neural network model including a feature extractor and task-specific heads, wherein the feature extractor has a feature extractor shared component and a feature extractor task-specific component – This limitation amounts to no more than generally linking the use of the judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h)). It merely limits the use of the abstract idea to ---a multi-task model with shared and task-specific components and thus, fails to provide significantly more to the judicial exception.
As such, claims 1, 8 and 15 are not patent eligible.
Dependent Claims
Claims 2-5, 7, 9-12, 14 and 16-19 merely narrow the previously cited abstract idea limitations. For the reasons described above with respect to independent claims 1, 8 and 15, these judicial exceptions are not meaningfully integrated into a practical application, nor amount to significantly more than the abstract ideas. The claims disclose similar limitations described for the independent claims above and do not provide anything more than the mental processes that are practically capable of being performed in the human mind with the assistance of pen and paper and mathematical concepts that are achievable through mathematical computation. Therefore, claims ---2-5, 7, 9-12, 14 and 16-19 also recite abstract ideas that do not integrate into a practical application or amount to significantly more than the judicial exception, and are rejected under U.S.C. § 101.
Step 1 – Claims 2-5 and 7 are drawn to a method, claims 9-12 and 14 are drawn to a non-transitory computer-readable storage medium, and claims 16-19 are drawn to a system. Therefore, each of these claims fall under one of the four categories of statutory subject matter (process/method, machine/product/apparatus, manufacture or composition of matter).
Step 2A Prong 1 – These claims are directed to a judicially recognized exception of an abstract idea without significantly more.
Claims 3, 10 and 17:
wherein parameters in the feature extractor task-specific component are updated to minimize a loss of its assigned task only – This limitation is directed towards the abstract idea of a mental process, or a concept that can be performed in the human mind, including observation, evaluation, judgement or opinion (see MPEP § 2106.04(a)(2), subsection III, C). In paragraph [00067] of the specification, it states “A conventional training algorithm updates 10 filters to minimize the sum of losses of tasks A and B. The exemplary method, however, updates the first 5 filters to minimize the loss of task A and updates the remaining 5 filters to minimize the task B loss.” BRI in light of the specification would support that “updating to minimize an assigned loss only” would encompass a mental process with or without the aid of pen and paper of determining a minimum value for different groups separately from each other.
Claims 4, 11 and 18:
wherein, during training, each group of the N groups is only updated by its corresponding task gradients – This limitation is directed towards the abstract idea of a mathematical calculation (see MPEP § 2106.04(a)(2), section I, C). In Paragraph [00028] of the specification, it states “The exemplary methods denote the parameters assigned to the tth task as
θ
∈
R
h
×
w
×
c
i
×
n
t
, where nt is the number of output channels assigned to the task t. Then, one iteration of parameter update using GradSplit is formulated as:
θ
t
←
θ
t
-
α
∇
θ
t
G
S
L
,
w
h
e
r
e
∇
θ
t
G
S
L
=
∇
θ
t
L
t
. Therefore, GradSplit updates parameter θt using the gradients from its assigned task only while discarding gradients are not averaged over tasks.” BRI in light of the specification would support that “only updating by corresponding task gradients” as claimed would fall within the mathematical calculations grouping, and therefore, the claim recites the abstract idea of a mathematical concept.
Claims 5, 12 and 19:
wherein each task learns its features without interference from other tasks – This limitation is directed towards the abstract idea of a mental process, or a concept that can be performed in the human mind, including observation, evaluation, judgement or opinion (see MPEP § 2106.04(a)(2), subsection III, C). In Paragraph [00025] of the specification, it states “A well-known issue for multi-task learning is that if the tasks have conflicts (e.g., identity-invariant feature versus identity-variant attributes), then joint optimization leads to sub-optimal solutions. To alleviate this, the exemplary methods propose a training scheme dubbed Gradient Split (or GradSplit) that enables each task to learn its essential features without interference from other tasks. Instead of using each task loss to update all filters of convolution in the shared backbone, GradSplit explicitly makes it only impact a subset of the filters.” BRI in light of the specification would support that “learning features without interference” would encompass a mental process with or without the aid of pen and paper of conducting processes independently from each other.
Step 2A Prong 2 – These limitations do not recite any additional elements which integrate the abstract idea into a practical application.
Claims 2, 9 and 16:
wherein the feature extractor generates a feature map from an image of the extracted images and the task-specific heads output task predictions based on the generated feature map – This limitation merely recites the idea of generating a feature map and making predictions, but fails to recite details of how the generation or predicting is accomplished. Reciting the idea of a solution or outcome without detailing how the result is accomplished is equivalent to saying “apply it” (see MPEP § 2106.05(f)), and thus, fails to integrate the exception into a practical application.
Claims 7 and 14:
wherein a round-robin batch-level update mechanism is applied – This limitation amounts to no more than generally linking the use of the judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h)). It merely limits the use of the abstract idea to a round-robin batch-level updating scheme and thus, fails to integrate the exception into a practical application.
Step 2B – These limitations, as a whole, do not amount to significantly more than the judicial exception.
Claims 2, 9 and 16:
wherein the feature extractor generates a feature map from an image of the extracted images and the task-specific heads output task predictions based on the generated feature map – This limitation is directed towards an insignificant extra-solution activity of mere data output (see MPEP § 2106.05(g)), which is well-understood, routine and conventional activity similar to cases reviewed by the courts involving receiving or transmitting data over a network ((see MPEP § 2106.05(d)(II)) and thus, fails to provide significantly more to the judicial exception.
Claims 7 and 14:
wherein a round-robin batch-level update mechanism is applied – This limitation amounts to no more than generally linking the use of the judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h)). It merely limits the use of the abstract idea to a round-robin batch-level updating scheme and thus, fails to provide significantly more to the judicial exception.
As such, claims 2-7, 9-14 and 16-20 are not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 7-12, and 14-19 are rejected under 35 U.S.C. 103 as being unpatentable over Suteu et al. (“Regularizing Deep Multi-Task Networks using Orthogonal Gradients”, published 12/14/2019), hereinafter Suteu; in view of Bragman et al. (“Stochastic Filter Groups for Multi-Task CNNs: Learning Specialist and Generalist Convolution Kernels”, published 10/29/2019), hereinafter Bragman; in further view of Ranjan et al. (“An All-In-One Convolutional Neural Network for Face Analysis”, published 06/29/2017), hereinafter Ranjan. Suteu and Bragman were cited in the previous Office Action.
Regarding Claim 1, Suteu teaches a method for multi-task learning via gradient split for rich human analysis (Suteu: “We propose a new gradient regularization term to the multi-task objective that explicitly minimizes the squared cosine between task gradients” [Section 1. Introduction]), the method comprising:
extracting images from training data having a plurality of datasets, each dataset associated with one task (Suteu: “Each of the inputs in X is associated to a set of labels for the tasks in T, forming the dataset
D
=
{
x
i
,
y
i
t
1
,
.
.
.
,
y
i
t
T
}
i
∈
N
of N observations” [Section 3. Orthogonal Task Gradients]); and
training a neural network model that alleviates a gradient conflict resulting from performing the visual human analysis tasks simultaneously on an image using the neural network model (Suteu: “Deep neural networks have proven to be very successful at solving isolated tasks in a variety of fields ranging from computer vision to NLP. In contrast to this single task setup, multi-task learning aims to train one model on several problems simultaneously.” [Section 1. Introduction]; “The problem of conflicting gradients or task interference has been previously explored in multi-task learning as well as continual learning. Zhao et al. (2018) introduce a modulation module that reduces destructive gradient interference between tasks that are unrelated. Du et al. (2018) choose to ignore the gradients of auxiliary tasks if they are not sharing a similar direction with the main task. Riemer et al. (2018) maximize the dot product between task gradients in order to overcome catastrophic forgetting. These methods have in common the interpretation that two tasks are in conflict if the cosine between their gradients is negative, while alignment should be encouraged. Our work differs from this perspective by additionally penalizing task gradients that have a similar direction, arguing that by decorrelating updates the shared encoder is able to maximize its representational capacity.” [Section 2. Related Works]),
the neural network model including a feature extractor and task-specific heads, wherein the feature extractor has a feature extractor shared component and a feature extractor task-specific component (Suteu: “We define a multi-task neural network as a shared encoder
f
θ
s
h
and a set of task-specific decoders
f
θ
t
i
, for each of the T tasks
T
=
{
t
1
,
.
.
.
,
t
T
}
. The encoder creates a mapping between the input space X and a latent feature space
R
d
that is used by each of the decoders to predict the task specific labels
Y
t
i
.” [Section 3. Orthoganal Task Gradients]), the training includes:
dividing filters of convolutional layers of the feature extractor (Suteu: “A similar observation regarding orthogonal parameters is made by Rodriguez et al. (2016) who propose a weight regularization term for single task learning that decorrelates filters in convolutional neural networks.” [Section 2. Related Work]); and
assigning one task to each group convolutional layers (Suteu: “We perform our experiments using a convolutional neural network architecture. The shared encoder consists of two convolutional layers, while the decoders have one convolutional and two fully connected layers. The decoders contain convolutions in order to encourage the learning of task-specific filters. For simplicity our convolutional layers lack bias terms and use stride to replace maxpooling layers.” [Section 4. Empirical Analysis]).
However, Suteu fails to expressly disclose each dataset associated with one task from a plurality of visual human analysis tasks that utilize identity-variant attributes and identity-invariant features; the training includes: during backpropagation, dividing filters of deeper layers of convolutional layers of the feature extractor into N groups, N being a number of tasks; assigning one task to each group of the N groups; and manipulating gradients of the N groups of convolutional layers so that each task loss updates only one subset of filters associated with one task assigned to the N groups by discarding gradients of filters associated with other tasks.
In the same field of endeavor, Bragman teaches the training includes: during backpropagation, dividing filters of deeper layers of convolutional layers of the feature extractor into N groups, N being a number of tasks (Bragman: “We propose stochastic filter groups (SFG), a probabilistic mechanism to partition kernels in each convolution layer into “specialist” groups or a “shared” group, which are specific to or shared across different tasks, respectively.” [Section 3. Methods]; See [Figure 1] and [Figure 3]);
assigning one task to each group of the N groups of convolutional layers (Bragman: “We propose stochastic filter groups (SFG), a probabilistic mechanism to partition kernels in each convolution layer into “specialist” groups or a “shared” group, which are specific to or shared across different tasks, respectively.” [Section 3. Methods]; “Filter Assignment: each kernel
w
k
(
l
)
is stochastically assigned to either: i) the “task-1 specific group”
G
1
(
l
)
, ii) “shared group”
G
s
(
l
)
or iii) “task-2 specific group”
G
2
(
l
)
with respective probabilities
p
l
,
k
=
p
1
l
,
k
,
p
s
l
,
k
,
p
2
l
,
k
∈
[
0,1
]
3
. Convolving with the respective filter groups yields distinct sets of features
F
1
(
l
)
,
F
s
(
l
)
,
F
2
(
l
)
.
Fig.2 illustrates this operation and Fig. 3 shows different learnable patterns.” [Section 3.1. Stochastic Filter Groups]); and
manipulating gradients of the N groups of convolutional layers so that each task loss updates only one subset of filters associated with one task assigned to the N groups by discarding gradients of filters associated with other tasks (Bragman: “(ii) shows the directions of the gradient flow from the task losses L1 and L2. The red and blue arrows denote the gradients that step from L1 and L2, respectively. The task-specific groups G1, G2 are only updated based on the associated losses” [Figure 4]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated that the training includes: during backpropagation, dividing filters of deeper layers of convolutional layers of the feature extractor into N groups, N being a number of tasks; assigning one task to each group of the N groups; and manipulating gradients of the N groups of convolutional layers so that each task loss updates only one subset of filters associated with one task assigned to the N groups by discarding gradients of filters associated with other tasks, as taught by Bragman, to the method of Suteu because both of these methods are directed towards minimizing task interference in multi-task learning. In making this combination and manipulating the gradients of task-specific filter groups independently from each other, it would allow the method of Suteu to introduce sparsity into the filter computations, which “reduces computational cost and number of parameters without compromising accuracy” (Bragman: [Section 3.1. Stochastic Filter Groups]).
Suteu and Bragman still fail to expressly disclose each dataset associated with one task from a plurality of visual human analysis tasks that utilize identity-variant attributes and identity-invariant features.
In the same field of endeavor, Ranjan teaches each dataset associated with one task from a plurality of visual human analysis tasks that utilize identity-variant attributes and identity-invariant features (Ranjan: “We divide the tasks into two groups: 1) subject independent tasks which include face detection, keypoints localization and visibility, pose estimation and smile prediction, and 2) subject-dependent tasks which include age estimation, gender prediction and face recognition.” [Section II.B. Network Architecture]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated that each dataset associated with one task from a plurality of visual human analysis tasks that utilize identity-variant attributes and identity-invariant features, as taught by Ranjan to the method of Suteu and Bragman because both of these methods are directed towards image analysis with multi-task learning. Facial recognition is a specialized field of image analysis. In making this combination and determining human facial recognition related tasks for the multi-task model, it would allow the model of Suteu and Bragman to be applied for “simultaneously detect[ing] faces, predict[ing] landmarks locations, pose angles, smile expression, gender, age as well as the identity from any unconstrained face image” (Ranjan: [Section I. Introduction])
Regarding Claim 2, Suteu, Bragman and Ranjan teach the method of Claim 1, wherein the feature extractor generates a feature map from an image of the extracted images and the task-specific heads output task predictions based on the generated feature map (Suteu: “The encoder creates a mapping between the input space X and a latent feature space Rd that is used by each of the decoders to predict the task specific labels Yt” [Section 3. Orthogonal Task Gradients]).
Regarding Claim 3, Suteu, Bragman and Ranjan teach the method of Claim 1, wherein parameters in the feature extractor task-specific component are updated to minimize a loss of its assigned task only (Bragman: “Fig. 4(ii) shows that such sparse connectivity ensures the parameters of
G
1
(
l
)
and
G
2
(
l
)
are only learned based on the respective task losses” [Section 3.1. Stochastic Filter Groups]).
Regarding Claim 4, Suteu, Bragman and Ranjan teach the method of Claim 1, wherein, during training, each group of the