DETAILED ACTION
This action is in response to the amendment filed 04/21/2026. Claims 1-4, 6-11, 13-17, and 19-20 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement filed 01/12/2022 fails to comply with 37 CFR 1.98(a)(2), which requires a legible copy of each cited foreign patent document; each non-patent literature publication or that portion which caused it to be listed; and all other information or that portion which caused it to be listed. The information disclosure statement filed 01/12/2022 listed non patent literature documents that were not in the application file. It has been placed in the application file, but the information referred to therein has not been considered.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claim 1 is rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. Specifically, “wherein a same neural network encoder is used for all diverse tasks” is not supported by the specification. Paragraphs 0014-0015 describe sharing an encoder, however these paragraphs are describing standard and known inventions. Further, Figure 1, which displays the invention’s architecture, depicts more than one encoder f. Additionally, Figure 1 does not show the many encoders f operating on diverse tasks. Instead, it depicts all encoders operating on tasks represented by a circle.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-4, 6-11, 13-17, and 19-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1:
Subject Matter Eligibility Analysis Step 1:
Claim 1 recites a process, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 1 recites
Measuring the diverse tasks with disjoint input and output variable sets in a shared space z (This limitation is a mental process as it encompasses a human mentally measuring tasks.)
for each diverse task, encoding by a function f a value of each observed input variable xi given its shared space location vector zi within shared space z,(This limitation is a mental process as it encompasses a human mentally encoding each task.)
aggregating encodings by elementwise addition (This limitation is a mental process as it encompasses a human mentally aggregating encodings.)
decoding by a function g the aggregated encodings to predict output variable yj at its location vector zj, wherein zi and zj are variable embeddings (This limitation is a mental process as it encompasses a human mentally decoding a task.)
Therefore, claim 1 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 1 further recites additional elements of
a process, implemented in a computing environment, for training a single model comprised of multiple neural networks across diverse tasks from multiple domains (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
wherein the encoding is implemented by a first neural network encoder, wherein a same neural network encoder is used for all diverse tasks (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
and further wherein the decoding is implemented by a single neural network decoder(This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
Therefore, claim 1 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 1 do not provide significantly more than the abstract idea itself, taken alone and in combination because
a process, implemented in a computing environment, for training a single model comprised of multiple neural networks across diverse tasks from multiple domains uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
wherein the encoding is implemented by a first neural network encoder, wherein a same neural network encoder is used for all diverse tasks uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
and further wherein the decoding is implemented by a single neural network decoder uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 1 is subject-matter ineligible.
Regarding Claim 2:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 2 recites the same abstract ideas as claim 1. Therefore, claim 2 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 2 further recites additional elements of
wherein the encoding and decoding are conditioned on the variable embeddings via Feature-wise Linear Modulation (FiLM) layers. (This element does not integrate the abstract idea into a practical application because it recites a technological environment in which to apply a judicial exception (see MPEP 2106.05(h)).)
Therefore, claim 2 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 2 do not provide significantly more than the abstract idea itself, taken alone and in combination because
wherein the encoding and decoding are conditioned on the variable embeddings via Feature-wise Linear Modulation (FiLM) layers. specifies a particular technological environment to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(h)).
Therefore, claim 2 is subject-matter ineligible.
Regarding Claim 3:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 3 recites
wherein function g is decomposed (This limitation is a mental process as it encompasses a human mentally re-decomposing a function.)
Therefore, claim 3 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 3 further recites additional elements of
a core, g1, which is independent of output variable yj(This element does not integrate the abstract idea into a practical application because it recites a generic computing component on which to perform the abstract idea (see MPEP 2106.05(f)).)
a decoder, g2, which is conditioned on output variable yj (This element does not integrate the abstract idea into a practical application because it recites a generic computing component on which to perform the abstract idea (see MPEP 2106.05(f)).)
Therefore, claim 3 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 3 do not provide significantly more than the abstract idea itself, taken alone and in combination because
a core, g1, which is independent of output variable yj uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
a decoder, g2, which is conditioned on output variable yj uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 3 is subject-matter ineligible.
Regarding Claim 4:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 4 recites
wherein the single model is in the form of:
E
[
y
j
|
x
]
=
g
2
(
g
1
(
∑
i
=
1
n
f
(
x
,
,
z
i
)
)
,
z
j
). (This limitation is an abstract idea because it recites a mathematical equation)
Therefore, claim 4 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 4 does not recites any additional elements to integrate the abstract idea into a practical application. Therefore, claim 4 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements to provide significantly more than the abstract idea itself, taken alone and in combination, claim 4 is subject-matter ineligible.
Regarding Claim 6:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 6 recites the same abstract ideas as claim 1. Therefore, claim 6 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 6 further recites additional elements of
wherein the neural networks are residual block networks (This element does not integrate the abstract idea into a practical application because it recites generic computer components on which to perform the abstract idea (see MPEP 2106.05(f)).)
Therefore, claim 6 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 6 do not provide significantly more than the abstract idea itself, taken alone and in combination because
wherein the neural networks are residual block networks uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 6 is subject-matter ineligible.
Regarding Claim 7:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 7 recites
wherein the shared space z is 2-Dimensional (This limitation is an abstract idea because it further describes the mental process of measuring tasks as recited in claim 1.)
Therefore, claim 7 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 7 does not recites any additional elements to integrate the abstract idea into a practical application. Therefore, claim 7 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements to provide significantly more than the abstract idea itself, taken alone and in combination, claim 7 is subject-matter ineligible.
Regarding Claim 8:
Subject Matter Eligibility Analysis Step 1:
Claim 8 recites a non-transitory computer readable medium and is thus a product, one of the four statutory categories of patentable subject matter
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 8 recites
measuring the diverse tasks with disjoint input and output variable sets in a shared space z (This limitation is a mental process as it encompasses a human mentally measuring tasks.)
for each diverse task, encoding by a function f a value of each observed variable xi given its shared space location zi within shared space z (This limitation is a mental process as it encompasses a human mentally encoding each task.)
aggregating encodings by elementwise addition (This limitation is a mental process as it encompasses a human mentally aggregating encodings.)
decoding by a function g the aggregated encodings to predict yj at its location zj, wherein zi and zj are variable embeddings (This limitation is a mental process as it encompasses a human mentally decoding a task.)
Therefore, claim 8 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 8 further recites additional elements of
At least one computer-readable medium storing instructions that, when executed by a computer, perform a process for training a single model across diverse tasks (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
Wherein the encoding is implemented by a neural network encoder, wherein a same neural network encoder is used for all diverse tasks (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
And further wherein the decoding is implemented by a single neural network decoder (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
Therefore, claim 8 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 8 do not provide significantly more than the abstract idea itself, taken alone and in combination because
At least one computer-readable medium storing instructions that, when executed by a computer, perform a process for training a single model across diverse tasks uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Wherein the encoding is implemented by a neural network encoder, wherein a same neural network encoder is used for all diverse tasks uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
And further wherein the decoding is implemented by a single neural network decoder uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 8 is subject-matter ineligible.
Regarding claim 9, claim 9 recites substantially similar limitations as claim 2, and is therefore rejected under the same analysis.
Regarding claim 10, claim 10 recites substantially similar limitations as claim 3, and is therefore rejected under the same analysis.
Regarding claim 11, claim 11 recites substantially similar limitations as claim 4, and is therefore rejected under the same analysis.
Regarding claim 13, claim 13 recites substantially similar limitations as claim 6, and is therefore rejected under the same analysis.
Regarding claim 14, claim 14 recites substantially similar limitations as claim 7, and is therefore rejected under the same analysis.
Regarding Claim 15:
Subject Matter Eligibility Analysis Step 1:
Claim 15 recites a single universal prediction model, thus a product, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 15 recites
aggregating the encoder outputs (This limitation is a mental process as it encompasses a human mentally aggregating outputs.)
generating… a prediction yj given its location in the shared space z. (This limitation is a mental process as it encompasses a human mentally generating a prediction.)
Therefore, claim 15 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 15 further recites additional elements of
A single universal prediction model comprised of multiple neural networks trained across diverse tasks from multiple domains in a shared space with disjoint input and output variable sets (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
An encoder, f, which is conditioned on a shared space vector zi (This element does not integrate the abstract idea into a practical application because it recites a generic computing component on which to perform the abstract idea (see MPEP 2106.05(f)).)
generating by a neural network encoder an encoder output for a task variable xi given its location in the shared space z, wherein a same neural network encoder is used for all diverse tasks (This element does not integrate the abstract idea into a practical application because it recites insignificant extra-solution activity of data gathering (see MPEP 2106.05(g)).)
an aggregator (This element does not integrate the abstract idea into a practical application because it recites a generic computing component on which to perform the abstract idea (see MPEP 2106.05(f)).)
a core, g1, which is independent of output (This element does not integrate the abstract idea into a practical application because it recites a generic computing component on which to perform the abstract idea (see MPEP 2106.05(f)).)
a decoder, g2, which is conditioned on shared space vector zj (This element does not integrate the abstract idea into a practical application because it recites a generic computing component on which to perform the abstract idea (see MPEP 2106.05(f)).)
generating by a single neural network decoder (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
Therefore, claim 15 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 15 do not provide significantly more than the abstract idea itself, taken alone and in combination because
A single universal prediction model comprised of multiple neural networks trained across diverse tasks from multiple domains in a shared space with disjoint input and output variable sets uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
An encoder, f, which is conditioned on shared space vector zi sets uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
generating by a neural network encoder an encoder output for a task variable xi given its location in the shared space z, wherein a same neural network encoder is used for all diverse tasks is the well understood, routine, and conventional activity of “transmitting or receiving data over a network” (see MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network)).
an aggregator uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
a core, g1, which is independent of output uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
a decoder, g2, which is conditioned on shared space vector zj uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
generating by a single neural network decoder uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 15 is subject-matter ineligible.
Regarding claim 16, claim 16 recites substantially similar limitations as claim 4, and is therefore rejected under the same analysis.
Regarding Claim 17:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 17 recites the same abstract ideas as claim 15. Therefore, claim 17 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 17 further recites additional elements of
wherein vector zi and zj are variable embeddings. (This element does not integrate the abstract idea into a practical application because it recites a technological environment in which to apply a judicial exception (see MPEP 2106.05(h)).)
Therefore, claim 17 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 17 do not provide significantly more than the abstract idea itself, taken alone and in combination because
wherein vector zi and zj are variable embeddings specifies a particular technological environment to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(h)).
Therefore, claim 17 is subject-matter ineligible.
Regarding claim 19, claim 19 recites substantially similar limitations as claim 6, and is therefore rejected under the same analysis.
Regarding claim 20, claim 20 recites substantially similar limitations as claim 2, and is therefore rejected under the same analysis.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1, 7, 8, and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kuga et al. (“Multi-task Learning using Multi-modal Encoder-Decoder Networks with Shared Skip Connections”) (hereafter referred to as Kuga) in view of Rabideau et al. (US 2022/0383994 A1) (hereafter referred to as Rabideau) and Malone et al. (US 2019/0325995 A1) (hereafter referred to as Malone).
Regarding claim 1, Kuga teaches
A process, implemented in a computing environment, for training a single model comprised of multiple neural networks across diverse tasks from multiple domains (Kuga, page 403, abstract, “We propose a multi-modal encoder-decoder networks to harness the multi-modal nature of multi-task scene recognition. In additional to the shared latent representation among encoder-decoder pairs, our model also has shared skip connections from different encoders” where “we describe details of the cost functions defined for semantic label decoder, image decoder, and depth decoder” (Kuga, page 406, 1st column, 3rd paragraph) and where “we discuss details of the proposed network by taking the task of depth and semantic label estimation from RGB images assuming a training dataset consisting with three modalities: image, depth and semantic labels” (Kuga, page 405, 1st column last paragraph). Examiner notes that the diverse tasks from multiple domains are the label decoding, image decoding, and depth decoding.) , comprising:
Measuring the diverse tasks with disjoint input and output variable sets in a shared space (Kuga, page 406, 1st column, 3rd paragraph, “we describe details of the cost functions defined for semantic label decoder, image decoder, and depth decoder” where “we discuss details of the proposed network by taking the task of depth and semantic label estimation from RGB images assuming a training dataset consisting with three modalities: image, depth and semantic labels” (Kuga, page 405, 1st column last paragraph) and “to exploit the commonality among different tasks, all encoder/decoder pairs are connected with each other via the shared latent representation” (Kuga, page 404, 2nd column, last paragraph). Examiner notes that the diverse tasks with disjoint input and output are the tasks, where the diverse tasks are the image, depth, and semantic labels. Examiner further notes that measuring the tasks is the cost functions and the shared space is latent representation.);
for each diverse task, encoding by a function f a value of each observed input variable xi given its shared space location vector zi within shared space z, wherein the encoding is implemented by a neural network encoder (Kuga, page 405, 2nd column, last paragraph, “all encoder/decoder pairs are connected with the shared latent representation. Letting x ∈{xi, xs, xd } be the input modalities for each encoder; image, semantic label and depth map, and E ∈ {Ei,Es,Ed} be the corresponding encoder functions, then the outputs from each encoder r, which means latent representation, is defined by r ∈ {Ei(xi),Es(xs),Ed(xd)}.” Examiner notes that for each modalities, or task, a function E is encoding an input value x where the input value x is the value of each observed variable. Examiner further notes that the shared space location is the shared latent representation and the encoder is the first neural network.);
wherein a same neural network encoder is used for all diverse tasks (Kuga, page 405, Figure 1,
PNG
media_image1.png
580
835
media_image1.png
Greyscale
Examiner notes that Figure 1 shows the encoder in the left box. )
and decoding by a function g the aggregated encodings to predict output variable yj at its location vector zj within shared space z, wherein zi and zj are variable embeddings, and further wherein the decoding is implemented by a single neural network decoder(Kuga, page 405, 2nd column, 1st paragraph, “each convolution layer (Conv) in the encoder is followed by a batch-normalization layer (Norm) and activation function (ReLU). Two maxpooling operations are placed in the middle of seven Conv+Norm+ReLU components, which makes the dimension of the latent representation 1/16 of the input. Similarly, the decoder network consists of seven Conv+Norm+ReLU components except for the final layer, while max-pooling operations are replaced by un-pooling operations for expanding a feature map. The max-pooling operation pools a feature map by taking maximum values, and the un-pooling operation restores the pooled feature into un-pooled feature map by embedding the same values. The final output of the decoder is then rescaled to the original input size. The rescaled output is further fed into a softmax layer to produce the class probability distribution at the label decoder” where “we can obtain the outputs y ∈ {Di(r),Ds(r),Dd(r)} from any r, where D ∈ {Di,Ds,Dd} are decoder functions” (Kuga, page 406, 1st column, 1st paragraph) and Kuga, page 405, Figure 1,
PNG
media_image1.png
580
835
media_image1.png
Greyscale
Examiner notes that the aggregated encodings are the max pooled feature maps, the prediction is the class probability, and the location is the feature map or latent representation. Examiner further notes that the feature map is created by embedding values, and so the feature map is made of the variable embeddings. Additionally, the decoder is the second neural network shown in Figure 1 in the right circle.).
Kuga teaches encoding by a function f a value of each observed input variable xi given its shared space location …zi within shared space z and decoding by a function g the aggregated encodings to predict output variable yj at its location…zj within shared space z. Kuga does not teach that the latent representation is a vector. Rabideau teaches
encoding … given its shared space location vector zi (Rabideau, page 10, paragraph 0054, “In the Variational AutoEncoder (VAE) described herein, the learned output of the encoder is a latent vector in a latent representation space 107.”)
decoding … to predict output variable yj at its location vector zj (Rabideau, page 10, paragraph 0054, “decoder 108 similarly constructed as a multilayered neural network, has an input layer 119 (in this the input layer receives a latent vector representation from the latent representation space 107) a hidden layer 120 and an output layer 121.”)
Kuga and Rabideau are analogous to the claimed invention because they use an encoder-decoder structure to produce an output. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to have modified Kuga to use vectors as latent representations like in Rabideau. Doing so allows for “the loss for this objective function is found such that backpropagation can be performed in order to adjust the weights on the Variational AutoEncoder’s computational graph such that this objective function is maximized” (Rabideau, page 12, paragraph 0067).
Kuga in view of Rabideau does not teach, but Malone does teach
aggregating encodings by elementwise addition (Malone, page 18, paragraph 0134, “For each input, the output vectors of the encoding functions are added element-wise and the resulting vector used in the margin-based contrastive loss function”. Examiner notes that the elementwise addition is summing the encodings.);
Kuga, Rabideau and Malone are analogous to the claimed invention because they use an encoder structure to produce an output. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Kuga and Rabideau to aggregate encodings by elementwise addition like in Malone. Doing so “allows to learn embeddings for the missing data values” (Malone, page 14, paragraph 0099).
Regarding claim 7, Kuga in view of Rabideau and Malone teaches the process according to claim 1. Kuga further teaches
wherein the shared space z is 2-Dimensional (Kuga, page 405, 2nd column, 1st paragraph, “Two max-pooling operations are placed in the middle of seven Conv+Norm+ReLU components, which makes the dimension of the latent representation 1/16 of the input” where “the input image size is fixed to 96 x 96” (Kuga, page 406, 2nd column, last paragraph). Examiner notes that since the laten representation is created from pooling a 2D input image, the latent representation, or shared space is also 2 dimensional. ).
Regarding claim 8, Kuga teaches
perform a process for training a single model comprised of multiple neural networks across diverse tasks from multiple (Kuga, page 403, abstract, “We propose a multi-modal encoder-decoder networks to harness the multi-modal nature of multi-task scene recognition. In additional to the shared latent representation among encoder-decoder pairs, our model also has shared skip connections from different encoders” where “we describe details of the cost functions defined for semantic label decoder, image decoder, and depth decoder” (Kuga, page 406, 1st column, 3rd paragraph) and where “we discuss details of the proposed network by taking the task of depth and semantic label estimation from RGB images assuming a training dataset consisting with three modalities: image, depth and semantic labels” (Kuga, page 405, 1st column last paragraph). Examiner notes that the diverse tasks from multiple domains are the label decoding, image decoding, and depth decoding.), comprising:
measuring the diverse tasks with disjoint input and output variable sets in a shared space z (Kuga, page 406, 1st column, 3rd paragraph, “we describe details of the cost functions defined for semantic label decoder, image decoder, and depth decoder” where “we discuss details of the proposed network by taking the task of depth and semantic label estimation from RGB images assuming a training dataset consisting with three modalities: image, depth and semantic labels” (Kuga, page 405, 1st column last paragraph) and “to exploit the commonality among different tasks, all encoder/decoder pairs are connected with each other via the shared latent representation” (Kuga, page 404, 2nd column, last paragraph). Examiner notes that the tasks with disjoint input and output are the tasks, where the tasks are the image, depth, and semantic labels. Examiner further notes that measuring the tasks is the cost functions and the shared space is latent representation.);
for each diverse task, encoding by a function f a value of each observed input variable xi given its shared space location vector zi within shared space z, wherein the encoding is implemented by a neural network encoder(Kuga, page 405, 2nd column, last paragraph, “all encoder/decoder pairs are connected with the shared latent representation. Letting x ∈{xi, xs, xd } be the input modalities for each encoder; image, semantic label and depth map, and E ∈ {Ei,Es,Ed} be the corresponding encoder functions, then the outputs from each encoder r, which means latent representation, is defined by r ∈ {Ei(xi),Es(xs),Ed(xd)}.” Examiner notes that for each modalities, or task, a function E is encoding an input value x where the input value x is the value of each observed variable. Examiner further notes that the shared space location is the shared latent representation and the encoder is the first neural network.);
wherein a same neural network encoder is used for all diverse tasks (Kuga, page 405, Figure 1,
PNG
media_image1.png
580
835
media_image1.png
Greyscale
Examiner notes that Figure 1 shows the encoder in the left box. )
and decoding by a function g the aggregated encodings to predict output variable yj at its location vector zj within shared space z, wherein zi and zj are variable embeddings, and further wherein the decoding is implemented by a single neural network decoder(Kuga, page 405, 2nd column, 1st paragraph, “each convolution layer (Conv) in the encoder is followed by a batch-normalization layer (Norm) and activation function (ReLU). Two maxpooling operations are placed in the middle of seven Conv+Norm+ReLU components, which makes the dimension of the latent representation 1/16 of the input. Similarly, the decoder network consists of seven Conv+Norm+ReLU components except for the final layer, while max-pooling operations are replaced by un-pooling operations for expanding a feature map. The max-pooling operation pools a feature map by taking maximum values, and the un-pooling operation restores the pooled feature into un-pooled feature map by embedding the same values. The final output of the decoder is then rescaled to the original input size. The rescaled output is further fed into a softmax layer to produce the class probability distribution at the label decoder” where “we can obtain the outputs y ∈ {Di(r),Ds(r),Dd(r)} from any r, where D ∈ {Di,Ds,Dd} are decoder functions” (Kuga, page 406, 1st column, 1st paragraph) and Kuga, page 405, Figure 1,
PNG
media_image1.png
580
835
media_image1.png
Greyscale
Examiner notes that the aggregated encodings are the max pooled feature maps, the prediction is the class probability, and the location is the feature map or latent representation. Examiner further notes that the feature map is created by embedding values, and so the feature map is made of the variable embeddings. Additionally, the decoder is the second neural network shown in Figure 1 in the right circle.).
Kuga teaches encoding by a function f a value of each observed input variable xi given its shared space location …zi within shared space z and decoding by a function g the aggregated encodings to predict output variable yj at its location…zj within shared space z. Kuga does not teach that the latent representation is a vector. Rabideau teaches
encoding … given its shared space location vector zi (Rabideau, page 10, paragraph 0054, “In the Variational AutoEncoder (VAE) described herein, the learned output of the encoder is a latent vector in a latent representation space 107.”)
decoding … to predict output variable yj at its location vector zj (Rabideau, page 10, paragraph 0054, “decoder 108 similarly constructed as a multilayered neural network, has an input layer 119 (in this the input layer receives a latent vector representation from the latent representation space 107) a hidden layer 120 and an output layer 121.”)
Kuga and Rabideau are analogous to the claimed invention because they use an encoder-decoder structure to produce an output. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to have modified Kuga to use vectors as latent representations like in Rabideau. Doing so allows for “the loss for this objective function is found such that backpropagation can be performed in order to adjust the weights on the Variational AutoEncoder’s computational graph such that this objective function is maximized” (Rabideau, page 12, paragraph 0067).
Kuga in view of Rabideau does not teach, but Malone does teach
at least one non-transitory computer-readable medium storing instructions that when executed by a computer perform a process (Malone, page 11, paragraph 0035, “The present invention provides a tangible, non-transitory computer-readable medium having instructions thereon which, upon execution by one or more processors with memory, provide for execution of a method”.)
aggregating encodings by elementwise addition (Malone, page 18, paragraph 0134, “For each input, the output vectors of the encoding functions are added element-wise and the resulting vector used in the margin-based contrastive loss function”. Examiner notes that the elementwise addition is summing the encodings.);
Kuga, Rabideau and Malone are analogous to the claimed invention because they use an encoder structure to produce an output. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Kuga and Rabideau to aggregate encodings by elementwise addition like in Malone. Doing so “allows to learn embeddings for the missing data values” (Malone, page 14, paragraph 0099). Additionally, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have implemented Kuga and Rabideau on a computer-readable medium storing instructions executed by a computer. Thus, this would be applying a known technique (multi-modal encoder-decoder networks) to a known device (computing device such as a processor) ready for improvement to yield predictable results (scene recognition outputs) (MPEP 2143 I. (C) Use of known technique to improve similar devices (methods, or products) in the same way).
Regarding claim 14, claim 14 recites substantially similar limitations as claim 7, and is therefore rejected under the same analysis.
Claim(s) 2 and 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kuga in view of Rabideau and Malone in further view of Meseguer-Brocal et al. (“Conditioned-U-Net: Introducing a Control Mechanism in the U-Net for Multiple Source Separations”) (hereafter referred to as Brocal).
Regarding claim 2, Kuga in view of Rabideau and Malone teach the process according to claim 1. Kuga in view of Rabideau and Malone do not teach, but Brocal does teach
wherein the encoding and decoding are conditioned on the variable embeddings via Feature-wise Linear Modulation (FiLM) layers (Brocal, page 3, 1st column, 5th paragraph, “our conditioned network is a standard U-Net that computes a set of generic source separation filters that we use to separate the various instruments. It adapts itself through the control mechanism (the condition generator) with FiLM layers inserted at different depths. Our external data is a condition vector…(a one-hot-encoding) which specify the instrument to be separated….The vector…is the input to the control mechanism/condition generator.” where “a FiLM layer is inserted inside each encoding block” (Brocal, page 4, 1st column, 2nd paragraph) and where “the decoder maps the encoder, with 6 decoders blocks” (Brocal, page 4, 1st column, 1st paragraph). Examiner notes that the variable embeddings is the condition vector.).
Kuga in view of Rabideau, Malone and Brocal are analogous to the claimed invention because they use an encoder-decoder structure to produce an output. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Kuga in view of Rabideau and Malone to use FiLM layers like in Brocal. Doing so, “can do several source separation tasks without losing performance as it does not introduce any limitation and makes use of the commonalities of the distinct instruments” (Brocal, page 6, 2nd column, 3rd paragraph).
Regarding claim 9, claim 9 recites substantially similar limitations as claim 2, and is therefore rejected under the same analysis.
Claim(s) 3-4, 6, 10-11, and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kuga in view of Rabideau and Malone in further view of Li et al. (“Autoencoder Based Residual Deep Networks for Robust Regression Prediction and Spatiotemporal Estimation”) (hereafter referred to as Li).
Regarding claim 3, Kuga in view of Rabideau and Malone teaches the process according to claim 1. Kuga in view of Rabideau and Malone does not teach, but Li does teach
wherein function g is decomposed into a core, g1, which is independent of output variable yj, and a decoder ,g2, which is conditioned on output variable yj (Li, page 2, Fig. 1 (see below),
PNG
media_image2.png
229
410
media_image2.png
Greyscale
Examiner notes that the function g is the decoder, code y is g1, the rest of the decoder is the g2, and output x’ is the output variable.)
Kuga in view of Rabideau, Malone and Li are analogous to the claimed invention because they use an encoder-decoder structure to produce an output. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Kuga in view of Rabideau and Malone to use a decomposed decoder like in Li. Doing so “aims to learn an efficient data representation… typically for dimensionality reduction” (Li, page 2, 2nd column, first paragraph).
Regarding claim 4, Kuga in view of Rabideau, Malone and Li teaches the process according to claim 3. Kuga further teaches
f (xi, zi) (Kuga, page 405, 2nd column, last paragraph, “all encoder/decoder pairs are connected with the shared latent representation. Letting x ∈{xi, xs, xd } be the input modalities for each encoder; image, semantic label and depth map, and E ∈ {Ei,Es,Ed} be the corresponding encoder functions, then the outputs from each encoder r, which means latent representation, is defined by r ∈ {Ei(xi),Es(xs),Ed(xd)}.” Examiner notes that xi is x, zi is the latent representation, and f is the encoder that takes x as input and creates the latent representation.)
E
[
y
j
|
x
]
=
g
1
(
g
2
(
∑
i
=
1
n
f
(
x
i
,
z
i
)
)
,
z
j
)
(Kuga, page 405, 2nd column, 1st paragraph, “each convolution layer (Conv) in the encoder is followed by a batch-normalization layer (Norm) and activation function (ReLU). Two maxpooling operations are placed in the middle of seven Conv+Norm+ReLU components, which makes the dimension of the latent representation 1/16 of the input. Similarly, the decoder network consists of seven Conv+Norm+ReLU components except for the final layer, while max-pooling operations are replaced by un-pooling operations for expanding a feature map. The max-pooling operation pools a feature map by taking maximum values, and the un-pooling operation restores the pooled feature into un-pooled feature map by embedding the same values. The final output of the decoder is then rescaled to the original input size. The rescaled output is further fed into a softmax layer to produce the class probability distribution at the label decoder” where “we can obtain the outputs y ∈ {Di(r),Ds(r),Dd(r)} from any r, where D ∈ {Di,Ds,Dd} are decoder functions” (Kuga, page 406, 1st column, 1st paragraph). Examiner notes that
∑
i
=
1
n
f
(
x
i
,
z
i
)
is the max pooled feature maps,
E
[
y
j
|
x
]
is the class probability, g2 is the decoder, and zj is the feature map or latent representation.)
While Kuga in view of Rabideau and Malone disclose the decoding of
∑
i
=
1
n
f
(
x
i
,
z
i
)
, Kuga in view of Rabideau and Malone do not explicitly disclose the decomposition part of the formula. Li, however does disclose the decomposition portion of the formula. In combination with Kuga, Rabideau, and Malone, Li teaches
g
1
(
g
2
(
∑
i
=
1
n
f
(
x
i
,
z
i
)
)
,
z
j
)
(Li, page 2, Fig. 1 (see below),
PNG
media_image2.png
229
410
media_image2.png
Greyscale
Examiner notes that the combination of Kuga, Rabideau, Malone, and Li disclose the entire formula with Li teaching g1 which is the code and g2 which are the decoder layers.)
Kuga in view of Rabideau, Malone and Li are analogous to the claimed invention because they use an encoder-decoder structure to produce an output. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Kuga in view of Rabideau, and Malone to use a decomposed decoder like in Li. Doing so “aims to learn an efficient data representation… typically for dimensionality reduction” (Li, page 2, 2nd column, first paragraph).
Regarding claim 6, Kuga in view of Rabideau and Malone in further view of Li teaches the process according to claim 1. Li further teaches
wherein the neural networks are residual block networks (Li, page 3, 2nd column, 4th paragraph, “Shortcut connections have been added to neural network to address vanishing/exploding gradients and degradation in accuracy in residual CNN. We use the shortcut connection of identity mapping from the encoding layer to its corresponding mirrored decoding layer to implement residual networks in a nested way from the outermost to the innermost layers.”).
Kuga in view of Rabideau, Malone and Li are analogous to the claimed invention because they use an encoder-decoder structure to produce an output. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Kuga in view of Rabideau and Malone to use residual block networks for the neural networks as in Li because “such residual connections can compensate the loss in error backpropagation through the long path of regular deep layers and collaborate with regular connections to reduce the known problem of degradation in accuracy” (Li, page 10, last paragraph 1st column – first paragraph 2nd column).
Regarding claim 10, claim 10 recites substantially similar limitations as claim 3, and is therefore rejected under the same analysis.
Regarding claim 11, claim 11 recites substantially similar limitations as claim 4, and is therefore rejected under the same analysis.
Regarding claim 13, claim 13 recites substantially similar limitations as claim 6, and is therefore rejected under the same analysis.
Claim(s) 15-17 and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kuga in view of Rabideau in further view of Malone and Li and .
Regarding claim 15, Kuga teaches
a single universal prediction model comprised of multiple neural networks trained across diverse tasks from multiple domains in a shared space with disjoint input and output variable sets (Kuga, page 403, abstract, “We propose a multi-modal encoder-decoder networks to harness the multi-modal nature of multi-task scene recognition. In additional to the shared latent representation among encoder-decoder pairs, our model also has shared skip connections from different encoders” where “we describe details of the cost functions defined for semantic label decoder, image decoder, and depth decoder” (Kuga, page 406, 1st column, 3rd paragraph) and where “we discuss details of the proposed network by taking the task of depth and semantic label estimation from RGB images assuming a training dataset consisting with three modalities: image, depth and semantic labels” (Kuga, page 405, 1st column last paragraph) and “to exploit the commonality among different tasks, all encoder/decoder pairs are connected with each other via the shared latent representation” (Kuga, page 404, 2nd column, last paragraph). Examiner notes that the tasks with disjoint input and output are the tasks, where the tasks are the image, depth, and semantic labels. Examiner further notes that measuring the tasks is the cost functions and the shared space is latent representation.);
an encoder, f, which is conditioned on a shared spaced vector zi, for generating by a neural network encoder an encoder output for a task variable xi given its location in the shared space z (Kuga, page 405, 2nd column, last paragraph, “all encoder/decoder pairs are connected with the shared latent representation. Letting x ∈{xi, xs, xd } be the input modalities for each encoder; image, semantic label and depth map, and E ∈ {Ei,Es,Ed} be the corresponding encoder functions, then the outputs from each encoder r, which means latent representation, is defined by r ∈ {Ei(xi),Es(xs),Ed(xd)}.” Examiner notes that for each modalities, or task, a function E is encoding an input value x where the input value x is the value of each observed variable. Examiner further notes that the shared space location and zi is the shared latent representation and the encoder is the first neural network.);
wherein a same neural network encoder is used for all diverse tasks (Kuga, page 405, Figure 1,
PNG
media_image1.png
580
835
media_image1.png
Greyscale
Examiner notes that Figure 1 shows the encoder in the left box. )
a decoder, g2, which is conditioned on shared space vector zj, for generating by a single neural network decoder a prediction yj given its location in the shared space z (Kuga, page 405, 2nd column, 1st paragraph, “each convolution layer (Conv) in the encoder is followed by a batch-normalization layer (Norm) and activation function (ReLU). Two maxpooling operations are placed in the middle of seven Conv+Norm+ReLU components, which makes the dimension of the latent representation 1/16 of the input. Similarly, the decoder network consists of seven Conv+Norm+ReLU components except for the final layer, while max-pooling operations are replaced by un-pooling operations for expanding a feature map. The max-pooling operation pools a feature map by taking maximum values, and the un-pooling operation restores the pooled feature into un-pooled feature map by embedding the same values. The final output of the decoder is then rescaled to the original input size. The rescaled output is further fed into a softmax layer to produce the class probability distribution at the label decoder” where “we can obtain the outputs y ∈ {Di(r),Ds(r),Dd(r)} from any r, where D ∈ {Di,Ds,Dd} are decoder functions” (Kuga, page 406, 1st column, 1st paragraph) and Kuga, page 405, Figure 1,
PNG
media_image1.png
580
835
media_image1.png
Greyscale
Examiner notes that the aggregated encodings are the max pooled feature maps, the prediction is the class probability, and the location is the feature map or latent representation. Examiner further notes that the feature map is created by embedding values, and so the feature map is made of the variable embeddings. Additionally, the decoder is the second neural network shown in Figure 1 in the right circle.).
Kuga teaches an encoder, f, which is conditioned on … zi and a decoder, g2, which is conditioned on …zj. Kuga does not teach that the latent representation is a vector. Rabideau teaches
an encoder, f, which is conditioned on shared space vector zi (Rabideau, page 10, paragraph 0054, “In the Variational AutoEncoder (VAE) described herein, the learned output of the encoder is a latent vector in a latent representation space 107.”)
a decoder, g2, which is conditioned on shared space vector zj (Rabideau, page 10, paragraph 0054, “decoder 108 similarly constructed as a multilayered neural network, has an input layer 119 (in this the input layer receives a latent vector representation from the latent representation space 107) a hidden layer 120 and an output layer 121.”)
Kuga and Rabideau are analogous to the claimed invention because they use an encoder-decoder structure to produce an output. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to have modified Kuga to use vectors as latent representations like in Rabideau. Doing so allows for “the loss for this objective function is found such that backpropagation can be performed in order to adjust the weights on the Variational AutoEncoder’s computational graph such that this objective function is maximized” (Rabideau, page 12, paragraph 0067).
Kuga in view of Rabideau does not teach, but Malone does teach
an aggregator for aggregating the encoder outputs (Malone, page 14, paragraph 0100, “FIG. 3 shows an example of a submodel 50 which learns embeddings 52 for combined heart rate and blood pressure data as a data modality 44, including when some of the raw patient data 46 is missing….The submodel 50 consists of two function, fd- and fm…..According to an embodiment, element-wise addition is used to combine the embeddings 52 of the episode snapshot 42a.” Examiner notes that the submodel is the aggregator and the embeddings are the encoder outputs.);
Kuga, Rabideau and Malone are analogous to the claimed invention because they use an encoder structure to produce an output. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Kuga and Rabideau to aggregate encodings by elementwise addition like in Malone. Doing so “allows to learn embeddings for the missing data values” (Malone, page 14, paragraph 0099).
Kuga in view of Rabideau and Malone do not teach, but Li does teach
a core, g1, which is independent of output (Li, page 2, Fig. 1 (see below),
PNG
media_image2.png
229
410
media_image2.png
Greyscale
Examiner notes that code y is g1, and the output variable is output x).;
Kuga in view Rabideau, Malone and Li are analogous to the claimed invention because they use an encoder-decoder structure to produce an output. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Kuga in view of Rabideau and Malone to use a decomposed decoder like in Li. Doing so “aims to learn an efficient data representation… typically for dimensionality reduction” (Li, page 2, 2nd column, first paragraph).
Regarding claim 16, Kuga in view of Rabideau, Malone and Li teach the single universal prediction model of claim 15. Kuga further teaches
f (xi, zi) (Kuga, page 405, 2nd column, last paragraph, “all encoder/decoder pairs are connected with the shared latent representation. Letting x ∈{xi, xs, xd } be the input modalities for each encoder; image, semantic label and depth map, and E ∈ {Ei,Es,Ed} be the corresponding encoder functions, then the outputs from each encoder r, which means latent representation, is defined by r ∈ {Ei(xi),Es(xs),Ed(xd)}.” Examiner notes that xi is x, zi is the latent representation, and f is the encoder that takes x as input and creates the latent representation.)
E
[
y
j
|
x
]
=
g
1
(
g
2
(
∑
i
=
1
n
f
(
x
i
,
z
i
)
)
,
z
j
)
(Kuga, page 405, 2nd column, 1st paragraph, “each convolution layer (Conv) in the encoder is followed by a batch-normalization layer (Norm) and activation function (ReLU). Two maxpooling operations are placed in the middle of seven Conv+Norm+ReLU components, which makes the dimension of the latent representation 1/16 of the input. Similarly, the decoder network consists of seven Conv+Norm+ReLU components except for the final layer, while max-pooling operations are replaced by un-pooling operations for expanding a feature map. The max-pooling operation pools a feature map by taking maximum values, and the un-pooling operation restores the pooled feature into un-pooled feature map by embedding the same values. The final output of the decoder is then rescaled to the original input size. The rescaled output is further fed into a softmax layer to produce the class probability distribution at the label decoder” where “we can obtain the outputs y ∈ {Di(r),Ds(r),Dd(r)} from any r, where D ∈ {Di,Ds,Dd} are decoder functions” (Kuga, page 406, 1st column, 1st paragraph). Examiner notes that
∑
i
=
1
n
f
(
x
i
,
z
i
)
is the max pooled feature maps,
E
[
y
j
|
x
]
is the class probability, g2 is the decoder, and zj is the feature map or latent representation.)
Kuga does not teach that the latent representations are vectors. Kuga in view of Rabideau, however teaches
vector zi and zj (Rabideau, page 10, paragraph 0054, “In the Variational AutoEncoder (VAE) described herein, the learned output of the encoder is a latent vector in a latent representation space 107. A decoder 108 similarly constructed as a multilayered neural network, has an input layer 119 (in this the input layer receives a latent vector representation from the latent representation space 107) a hidden layer 120 and an output layer 121.”)
While Kuga in view of Rabideau and Malone disclose the decoding of
∑
i
=
1
n
f
(
x
i
,
z
i
)
, Kuga in view of Rabideau and Malone do not explicitly disclose the decomposition part of the formula. Li, however does disclose the decomposition portion of the formula. In combination with Kuga, Rabideau, and Malone, Li teaches
g
1
(
g
2
(
∑
i
=
1
n
f
(
x
i
,
z
i
)
)
,
z
j
)
(Li, page 2, Fig. 1 (see below),
PNG
media_image2.png
229
410
media_image2.png
Greyscale
Examiner notes that the combination of Kuga, Rabideau, Malone, and Li disclose the entire formula with Li teaching g1 which is the code and g2 which are the decoder layers.)
Kuga in view of Rabideau, Malone and Li are analogous to the claimed invention because they use an encoder-decoder structure to produce an output. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Kuga in view of Rabideau, and Malone to use a decomposed decoder like in Li. Doing so “aims to learn an efficient data representation… typically for dimensionality reduction” (Li, page 2, 2nd column, first paragraph).
Regarding claim 17, Kuga in view of Rabideau, Malone, and Li teaches the single universal prediction model of claim 15. Kuga further teaches
wherein vector zi and zj are variable embeddings (Kuga, page 405, 2nd column, 1st paragraph, “each convolution layer (Conv) in the encoder is followed by a batch-normalization layer (Norm) and activation function (ReLU). Two maxpooling operations are placed in the middle of seven Conv+Norm+ReLU components, which makes the dimension of the latent representation 1/16 of the input. Similarly, the decoder network consists of seven Conv+Norm+ReLU components except for the final layer, while max-pooling operations are replaced by un-pooling operations for expanding a feature map. The max-pooling operation pools a feature map by taking maximum values, and the un-pooling operation restores the pooled feature into un-pooled feature map by embedding the same values. The final output of the decoder is then rescaled to the original input size. The rescaled output is further fed into a softmax layer to produce the class probability distribution at the label decoder” where “we can obtain the outputs y ∈ {Di(r),Ds(r),Dd(r)} from any r, where D ∈ {Di,Ds,Dd} are decoder functions” (Kuga, page 406, 1st column, 1st paragraph). Examiner notes that the aggregated encodings are the max pooled feature maps, the prediction is the class probability, and the location is the feature map or latent representation. Examiner further notes that the feature map is created by embedding values, and so the feature map is made of the variable embeddings.).
Kuga does not teach that the latent representations zi and zj are vectors. However Rabideau teaches
vector zi and zj (Rabideau, page 10, paragraph 0054, “In the Variational AutoEncoder (VAE) described herein, the learned output of the encoder is a latent vector in a latent representation space 107. A decoder 108 similarly constructed as a multilayered neural network, has an input layer 119 (in this the input layer receives a latent vector representation from the latent representation space 107) a hidden layer 120 and an output layer 121.”)
Kuga and Rabideau are analogous to the claimed invention because they use an encoder-decoder structure to produce an output. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to have modified Kuga to use vectors as latent representations like in Rabideau. Doing so allows for “the loss for this objective function is found such that backpropagation can be performed in order to adjust the weights on the Variational AutoEncoder’s computational graph such that this objective function is maximized” (Rabideau, page 12, paragraph 0067).
Regarding claim 19, Kuga in view Rabideau, Malone and Li teaches the single universal prediction of claim 15. Li further teaches
wherein the neural networks are residual block networks (Li, page 3, 2nd column, 4th paragraph, “Shortcut connections have been added to neural network to address vanishing/exploding gradients and degradation in accuracy in residual CNN. We use the shortcut connection of identity mapping from the encoding layer to its corresponding mirrored decoding layer to implement residual networks in a nested way from the outermost to the innermost layers.”).
Kuga in view Rabideau, Malone and Li are analogous to the claimed invention because they use an encoder-decoder structure to produce an output. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Kuga in view Rabideau, Malone and Li to use residual block networks for the neural networks as in Li because “such residual connections can compensate the loss in error backpropagation through the long path of regular deep layers and collaborate with regular connections to reduce the known problem of degradation in accuracy” (Li, page 10, last paragraph 1st column – first paragraph 2nd column).
Regarding claim 20, Kuga in view of Rabideau, Malone, and Li teach the single universal prediction of claim 15. Kuga in view of Rabideau, Malone, and Li do not teach
wherein the encoding and decoding are conditioned on the variable embeddings via Feature-wise Linear Modulation (FiLM) layers (Brocal, page 3, 1st column, 5th paragraph, “our conditioned network is a standard U-Net that computes a set of generic source separation filters that we use to separate the various instruments. It adapts itself through the control mechanism (the condition generator) with FiLM layers inserted at different depths. Our external data is a condition vector…(a one-hot-encoding) which specify the instrument to be separated….The vector…is the input to the control mechanism/condition generator.” where “a FiLM layer is inserted inside each encoding block” (Brocal, page 4, 1st column, 2nd paragraph) and where “the decoder maps the encoder, with 6 decoders blocks” (Brocal, page 4, 1st column, 1st paragraph). Examiner notes that the variable embeddings is the condition vector.).
Kuga in view of Rabideau, Malone, Li and Brocal are analogous to the claimed invention because they use an encoder-decoder structure to produce an output. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Kuga in view of Rabideau, Malone, and Li to use FiLM layers like in Brocal. Doing so, “can do several source separation tasks without losing performance as it does not introduce any limitation and makes use of the commonalities of the distinct instruments” (Brocal, page 6, 2nd column, 3rd paragraph).
Response to Arguments
On page 6, Applicant argues:
The Applicant has reviewed the IDS and finds the following NPL was not submitted:
• Y. Yang and T. M. Hospedales. A unified perspective on multi -domain and multi-task learning. In Proc. Of ICLR, 2014.
A copy is provided herewith. It appears to the Applicant that any other required copies were submitted, noting that copies of US patent documents including patent applications, issued patents and patent publications are not required to be submitted.
Examiner notes that the NPL listed above was considered. Examiner notes that a significant number of NPL documents are still required and are missing based on the IDS filed 01/12/2022.
On page 7, Applicant argues:
Claim 15 has been amended to address the first two points raised in the rejection. With respect to the Examiner's issue with "'output variable' in line 6 of claim 15" we find no such language in claim 15.
Examiner notes that the 112(b) rejections have been overcome in light of the instant amendments.
On page 5, Applicant argues:
Regarding the rejection to claims 8-14 as not falling into one of the four categories of patent-eligible subject matter, the clams are amended herein to recite the non-transitory nature of the medium.
Regarding the Applicant’s argument that the claims are amended to now fall into one of the four categories of patent-eligible subject matter, Examiner agrees. Thus, the 101 rejections regarding claims 8-14 not falling into one of the four statutory categories have been withdrawn.
On page 7, Applicant argues:
The Examiner acts directly contrary to guidance and essentially dumps every single additional element into the amorphous "apply it on a computer" category from MPEP 2106.05(f). Examiners were initially explicitly warned against using this rationale in the August 4, 2025 Memorandum from Charles Kim to Technology Centers 2100, 2600 and 3600 ("Kim Memo f'), "Examiners are cautioned not to oversimplify claim limitations and expand the application of the "apply it" consideration." And then, in the wake of the PTAB's precedential decision in Ex Parte Dejardins (hereafter Dejardins), the warning was added to the MPEP as described in the December 5, 2025 Memorandum from Charles Kim to the Patent Examining Corps:
Regarding the Applicant’s argument that the memorandums changed the analysis of claims under 101, the Examiner respectfully disagree. Specifically, Examiner notes that the memo did not change the analysis of claims under 101. As such, Examiner further notes that the claim limitations were not oversimplified. “A process, implemented in a computing environment, for training a single model comprised of multiple neural networks across diverse tasks from multiple domains”, “wherein the encoding is implemented by a first neural network encoder”, and “wherein a same neural network encoder is used for all diverse tasks, and further wherein the decoding is implemented by a single neural network decoder” do not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)) since the neural networks are implementing the abstract ideas.
On page 9, Applicant argues:
The reasoning in Dejardins and Baker is directly applicable to the present rejection and the present claims given that the problem and claimed solution addressed by the pending claims is improvements in artificial intelligence (AI) technology. As articulated in the BACKGROUND of the present application:
[0012] Deep multi-task learning (Deep MTL) has shown a similar ability to adapt knowledge across tasks whose observed variables are embedded in a shared space. Examples include vision, where the input for all tasks (photograph, drawing, or otherwise) is pixels arranged in a 2D plane; natural language, speech processing and genomics, which exploit the ID structure of text, waveforms, and nucleotide sequences; and video game-playing, where interactions are organized across space and time. Yet, many real-world prediction tasks have no such spatial organization; their input and output variables are simply labeled values, e.g., the height of a tree, the cost of a haircut, or the score on a standardized test. To make matters worse, these sets of variables are often disjoint across a set of tasks.
[0013] These challenges have led the MTL community to avoid such tasks, despite the fact that general knowledge about how to make good predictions can arise from solving seemingly "unrelated" tasks.
[0016] .... Existing methods address the challenge of disjoint output and input
c), spaces by using separate decoders and encoders for each domain (Table 1,
and thus they require some other method of sharing model parameters across tasks, such as sharing some of their layers or drawing their parameters from a shared pool. For many datasets, the separate encoder and decoder absorbs too much functionality to share optimally, and their complexity makes it difficult to analyze the relationships between tasks .....
[0017] None of these methods could optimally propose multi task encoder decoder decompositions in cross domain settings. In the background of foregoing limitations, there exists a need for a solution that can extend the notion of task embeddings in order to apply the idea in the cross-domain setting.
Regarding the Applicant’s argument that paragraphs 0012-0013 and 0016-0017 provide an improvement, Examiner respectfully disagrees. Specifically, Examiner notes that these paragraphs provide a problem, but not the solution and thus cannot provide an improvement. Examiner further notes that the judicial exception alone cannot provide the improvement (MPEP 2106.05(a)). Instead, the improvement must be provided by the additional elements. Since the additional elements fall under the "apply it on a computer" category from MPEP 2106.05(f), the additional elements cannot provide an improvement either.
On page 10, Applicant argues:
These elements cannot practically be performed in the human mind. Accordingly, the Applicant challenges the Examiner's finding of any abstract idea in the first place. Further, divorcing the elements from the implementing neural network encoders and decoders and then arguing that the neural networks are essentially generic computing
components, is also contrary the requirement that the claims be analyzed as a whole.
Regarding the Applicant’s argument that the elements cannot be practically performed in the human mind, Examiner respectfully disagrees. Specifically, “measuring the diverse tasks with disjoint input and output variable sets in a shared space z” encompasses a human mentally measuring tasks and is thus an evaluation, “for each diverse task, encoding by a function f a value of each observed input variable xi given its shared space location vector zi within shared space z,” encompasses a human mentally encoding each task and is thus an evaluation, “aggregating encodings by elementwise addition” encompasses a human mentally aggregating encodings and is thus an evaluation, “decoding by a function g the aggregated encodings to predict output variable yj at its location vector zj, wherein zi and zj are variable embeddings” encompasses a human mentally decoding a task and is thus an evaluation. Since these elements encompass evaluations, under MPEP 2106.04(a), these elements are mental processes.
On page 11, Applicant argues:
When the present claims are analyzed as required, the present claims are clearly directed to a combination of elements which implements the technical solution summarized in paragraph [0017] and integrate any judicial exceptions into a practical application, i.e., multi task encoder decoder decompositions in cross domain settings.
Regarding the Applicant’s argument that claim 0017 provides and summarizes the technical solution, Examiner respectfully disagrees. Specifically, Examiner notes that claim 0017 does not provide a solution. It merely states a need for a solution.
On page 11-12, Applicant argues:
In view of Dejardin, Carmody and Baker, the Board has already determined that claims directed to improving deep neural networks (DNN) are patent-eligible. See also, e.g., Ex Parte Mixer, Appeal No. 2023-003543 (Nov. 6, 2024) (reversing§ 101 rejection because "combination of elements recited by the claims result in an improvement to existing artificial neural network technology"); Ex Parte Helenius et. al., Appeal No. 2024-000079 (Aug. 13, 2024) (reversing§ 101 rejection for artificial intelligence claims because the claims improve the "generating a product path with higher likelihood of success."); Ex Parte Nair, et. al., Appeal No. 2024- 000911 (Aug. 7, 2024) (reversing§ 101 rejection for artificial neural network training claims because they "enable[] deep learning based on an exponentially larger range of possible test values then prior FF16 system could process"); Ex Parte Holtmann-Rice et al., Appeal 2024- 000046 (Mar. 27, 2024) (reversing§ l0lfor machine learning claims); Ex Parte Adachi, Appeal No. 2023-003642 (July 9, 2024) (reversing§ 101 rejection because the claim "recites an improvement to a neural network that includes an integrated layer that combines a dropout layer and a fully connected layer").
Regarding the Applicant’s argument that the memorandums changed the analysis of claims under 101, the Examiner respectfully disagree. Specifically, Examiner notes that the memo did not change the analysis of claims under 101. As such, Examiner further notes that the claim limitations were not oversimplified. “A process, implemented in a computing environment, for training a single model comprised of multiple neural networks across diverse tasks from multiple domains”, “wherein the encoding is implemented by a first neural network encoder”, and “wherein a same neural network encoder is used for all diverse tasks, and further wherein the decoding is implemented by a single neural network decoder” do not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)) since the neural networks are implementing the abstract ideas.
On page 13, Applicant argues:
A key distinction from the prior art cited is that the claims use a single neural network for encoding all tasks and a single neural network for decoding aggregated encodings for all diverse tasks. As is clearly shown in Figure 2 of Kuga, each task requires its own encoder/decoder pair.
Neither Rabideau nor Malone teach this. For at least this reason, it is most respectfully submitted that the combination of Kuga, Rabideau and Malone does not establish a prima facie case of unpatentability of claims 1, 7, 8 or 14 or any such prima facie case is hereby rebutted.
Regarding the Applicant’s argument that Kuga does not teach a single neural network for encoding all tasks and a single neural network for decoding aggregated encodings for all diverse tasks. Examiner respectfully disagrees. Specifically, Examiner notes that claim 1 does not recite a single encoder. Instead, claim 1 recites “a single model comprised of multiple neural networks”. Examiner additionally, notes that even so, Kuga teaches “wherein a same encoder is used for all diverse tasks” and “wherein decoding is implemented by a single neural network decoder” through Figure 1 (shown below)
PNG
media_image1.png
580
835
media_image1.png
Greyscale
Examiner notes that the left box is the encoder and the right circle is the decoder of the model in Kuga.
On page 13, Applicant argues:
Meseguer-Brocal does not remedy the deficiencies of the art as discussed above.
Accordingly, claims 2 and 9 are patentable over the combination of Kuga, Rabideau, Malone and Meseguer-Brocal by virtue of their dependence on claims 1 and 8.
Regarding the Applicant’s argument that the dependent claims are allowable at least due in part to their dependency on the independent claims, the Examiner respectfully disagrees and notes the instant rejections and response to arguments regarding the independent claims above.
On page 14, Applicant argues:
Li et al. does not remedy the deficiencies of the art as discussed above. Accordingly, claims 3-4, 6, 10-11 and 13 are patentable over the combination of Kuga, Rabideau, Malone and Li et al. by virtue of their dependence on claims 1 and 8.
Regarding the Applicant’s argument that the dependent claims are allowable at least due in part to their dependency on the independent claims, the Examiner respectfully disagrees and notes the instant rejections and response to arguments regarding the independent claims above.
On page 115, Applicant argues:
A key distinction from the prior art cited is that the claims use a single neural network for encoding all tasks and a single neural network for decoding aggregated encodings for all diverse tasks. As is clearly shown in Figure 2 of Kuga, each task requires its own encoder/decoder pair.
Neither Rabideau, Malone nor Li teach this. For at least this reason, it is most respectfully
submitted that the combination of Kuga, Rabideau, Malone and Li does not establish a prima facie case of unpatentability of claims 15-17 and 19-20 or any such prima facie case is hereby rebutted.
Regarding the Applicant’s argument that Kuga does not teach a single neural network for encoding all tasks and a single neural network for decoding aggregated encodings for all diverse tasks. Examiner respectfully disagrees. Specifically, Examiner notes that claim 1 does not recite a single encoder. Instead, claim 1 recites “a single model comprised of multiple neural networks”. Examiner additionally, notes that even so, Kuga teaches “wherein a same encoder is used for all diverse tasks” and “wherein decoding is implemented by a single neural network decoder” through Figure 1 (shown below)
PNG
media_image1.png
580
835
media_image1.png
Greyscale
Examiner notes that the left box is the encoder and the right circle is the decoder of the model in Kuga.
On page 15, Applicant argues:
To the extent not addressed, the undersigned submits that it is not necessary to address the rejections of the dependent claims with respect to the elements therein in this response since the dependent claims are believed to be allowable in view of the allowability of the independent claims. The undersigned does not admit to the propriety of the Office's rejection.
Regarding the Applicant’s argument that the dependent claims are allowable at least due in part to their dependency on the independent claims, the Examiner respectfully disagrees and notes the instant rejections and response to arguments regarding the independent claims above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Thathachar et al. (US 2020/0090035 A1) also discusses an encoder-decoder structure using multi-task learning.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KAITLYN R LAU whose telephone number is (571)272-1429. The examiner can normally be reached Monday - Thursday: 7:15 am - 5:15 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/K.R.L./ Examiner, Art Unit 2148 /MICHELLE T BECHTOLD/Supervisory Patent Examiner, Art Unit 2148
Prosecution Insights