DETAILED ACTION
This nonfinal office action is responsive to claims filed on November 30, 2023. Claims 81-121 are pending. Claims 81, 105, and 108 are independent.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on May 22, 2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 81-95, 97-99, and 101-104 are rejected under 35 U.S.C. 103 as being unpatentable over Jiang et al. (Beyond Greedy Ranking: Slate Optimization via List-CVAE), hereinafter Jiang, in view of Zhou et al. (PLAS: Latent Action Space for Offline Reinforcement Learning), hereinafter Zhou, in view of Ie et al. (Reinforcement Learning for Slate-based Recommender Systems: A Tractable Decomposition and Practical Methodology), hereinafter Ie.
Jiang and Zhou were cited in Applicant’s IDS filed on 5/22/2024.
Regarding claim 81, Jiang teaches the method:
A method of training a recommender system implemented by a processor and memory (Jiang, page 2, paragraph 2: “Therefore, the model first learns which slates give which type of responses and then directly generates similar slates given a desired response vector as the conditioning at inference time.” And page 8, paragraph 1: “This technique speeds up the total training and inference time for 2 million documents to merely 4 minutes on 1 GPU for both the response model (with 40k training steps) and List-CVAE (with 5k training steps).”)
pretraining a neural network-based decoder to generate a slate of items from a representation in a continuous low-dimensional latent space; (Jiang, page 4, paragraph 1: “At inference time, the List-CVAE model attempts to generate an optimal slate by conditioning on the ideal user response r*.” and page 5, paragraph 2: “Since the number of documents in D can be large, we first embed the documents into a low dimensional space. Let
Ψ
:
D
→
S
q
-
1
be that normalized embedding where
S
q
-
1
denotes the unit sphere in
R
q
. Ψ can easily be pretrained using a standard supervised model that predicts user responses from documents or through a standard auto-encoder technique.” – Figure 2 shows that there is a training phase and an inference phase, thus the training of the decoder is analogous to the pretraining.)
wherein the recommender system comprises the pretrained decoder for generating the recommended slate of items... (Jiang, page 5, paragraph 4: “During inference, output slates are generated by first sampling z from the conditionally learned prior distribution N(µ*, σ*), concatenating with the ideal condition c* =Φ(r*), and passed into the decoder, generating
x
1
,
⋯
,
x
k
from the learned
P
θ
(
s
|
z
,
c
)
, and finally taking argmax over the dot-products with the full embedding matrix independently for each position i=1,...,k.” – Where
x
1
,
⋯
,
x
k
is the recommended slate of items being generated.)
Jiang does not explicitly teach:
training a reinforcement learning agent in the recommender system to determine an action in the latent space, the action representing a recommended slate of items from the collection based on a state;
However, Zhou teaches:
training a reinforcement learning agent in the recommender system to determine an action in the latent space, the action representing a recommended slate of items from the collection based on a state; (Zhou, page 2, paragraph 1: “Our insight is that we can learn a policy in the latent action space of the CVAE and then use its decoder to output an action in the original action space of the environment.” And paragraph 2: “We demonstrate that PLAS allows generalization within the dataset and can provide consistently good performance for datasets with diverse actions.” – The action being a recommended slate of items from the collection is taught by Ie below, the diverse actions being generalized from the decoder and Figure 2 on page 4 shows that the action is being generated based on a state.)
That the recommended slate of items is generated from the action determined by the agent (Zhou, page 2, Figure 1: “Instead of explicitly matching the action distribution of the agent policy with
the behavior policy using divergence metrics such as KL or MMD, we implicitly constrain the policy
to output actions within the support of the behavior policy through the latent action space.”)
Zhou is considered analogous to the claimed invention as it is in the same field of endeavor, machine learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to have modified Jiang, which already teaches a decoder to generate a slate of items but does not explicitly teach that the decoder is used in a reinforcement learning agent, to include the teachings of Zhou which does teach that the decoder is used in a reinforcement learning agent in order to implicitly constrain the policy which "can be naturally satisfied without affecting the optimization of the other components and without being restricted by the density of the behavior policy distribution." (Zhou, page 2, paragraph 1)
Jiang and Zhou do not explicitly teach:
the action representing a recommended slate of items from the collection based on a state;
However, Ie teaches:
the action representing a recommended slate of items from the collection based on a state; (Ie, page 7, paragraph 4: “Session optimization with slates can be modeled as a MDP with states S, actions A, reward function R and transition kernel P, with discount factor
0
≤
γ
≤
1
.” And last paragraph: “The states S typically reflect user state. This includes relatively static user features such as demographics, declared interests, and other user attributes, as well as more dynamic user features, such as user context (e.g., time of day).” And page 8, paragraph 2: “The action space A is simply the set of all possible recommendation slates.”)
Ie is considered analogous to the claimed invention as it is in the same field of endeavor, machine learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to have modified Jiang and Zhou, which already teaches training a recommender system using reinforcement learning to determine an action in the latent space but does not explicitly teach that the action is a recommended slate of items based on a state, to include the teachings of Ie which does teach that the action is a recommended slate of items based on a state in order to "find optimal slate recommendation as a function of the state." (Ie, page 8, paragraph 5)
Regarding claim 82, Jiang, Zhou, and Ie teach the method of claim 81, as cited above.
Jiang further teaches:
said pretraining of the decoder comprises pretraining a generative model comprising the decoder and a neural network-based encoder; the generative model comprises a variational autoencoder (VAE); (Jiang, Fig. 2: “Structure of List-CVAE for both (a) training and (b) inference. s = (d1,d2,...,dk) is the input slate.” – The CVAE is a variational autoencoder and therefore includes both the decoder and an encoder. Figure 2 shows that there is both training and inference and, therefore, the decoder and encoder are both pretrained.)
said pretraining of the generative model comprises training the decoder to reconstruct the slate of items from a representation in the latent space of a combination of the slate of items and a set of interactions associated with the slate of items; and (Jiang, page 4, paragraph 2: “The decoder learns these biases through reconstruction of the input slates from latent variables z with conditions.” – The conditions is analogous to the set of interactions.)
interactions in the set of interactions are respectively associated with items in the slate of items. (Jiang, page 5, section 4.1, paragraph 1: “The simulator generates a random matrix
W
~
N
μ
,
σ
k
×
n
×
k
×
n
where each element
W
i
,
d
i
,
j
,
d
j
represents the interaction between document
d
i
at position
i
and document
d
j
at position
j
, and
n
=
D
.”)
Regarding claim 83, Jiang, Zhou, and Ie teach the method of claim 82, as cited above.
Jiang further teaches:
wherein the interactions comprise selections among the items in the slate; wherein the set of interactions (i) comprises a vector indicating whether the items in the slate were selected or not selected, (ii) indicates that a plurality of items in the slate were selected or not selected, and (iii) comprise user clicks. (Jiang, page 3, section 3.1: “Then let
r
=
r
1
,
⋯
,
r
k
be the user response vector, where
r
i
∈
R
is the user’s response on document
d
i
. For example, if the problem is to maximize the number of clicks on a slate, then let
r
i
∈
0,1
denote whether the document
d
i
is clicked, and thus an optimal slate
s
=
d
1
,
d
2
,
⋯
,
d
k
where
d
i
∈
D
is such that
s
maximizes
E
∑
i
=
1
k
r
i
.” – The user response vector includes the user’s response on the document which is analogous to the set of interactions. Whether the user clicked the document also indicates that the vector comprises whether the items in the slate were selected or not.)
Regarding claim 84, Jiang, Zhou, and Ie teach the method of claim 82, as cited above.
Jiang further teaches:
said pretraining of the generative model (i) comprises learning a joint distribution over slates of items, associated sets of interactions, and latent representations; (Jiang, bottom of page 3 through top of page 4: “In this paper, we use a CVAE to model the joint distribution of all documents in the slate conditioned on the user responses
r
, i.e.
P
d
1
,
d
2
,
⋯
,
d
k
|
r
.”) (ii) uses a dataset comprising a plurality of logged interactions which comprise a plurality of data pairs, each data pair comprising a slate of items and an associated set of interactions. (Jiang, page 5, section 4.1, paragraph 1: “The simulator generates a random matrix
W
~
N
μ
,
σ
k
×
n
×
k
×
n
where each element
W
i
,
d
i
,
j
,
d
j
represents the interaction between document
d
i
at position
i
and document
d
j
at position
j
, and
n
=
D
.” – The matrix representing the interactions between the documents is analogous to the data pairs comprising the slate of items and the associated interactions.)
Regarding claim 85, Jiang, Zhou, and Ie teach the method of claim 82, as cited above.
Jiang further teaches:
the dataset is generated … (ii) based on prior interactions with the user, and (iii) based on prior interactions at least partially with users other than the user. (Jiang, page 7, paragraph 3: “This dataset consists of 9.2M user purchase sessions around 53K products. Each user session contains an ordered list of products on which the user clicked, and whether they decided to buy them.” – The ordered list of products and whether the user decided to buy them is analogous the prior interactions with the user while the dataset consisting of 9.2M user purchase sessions is analogous to the prior interactions at least partially with users other than the user.)
Jiang does not explicitly teach:
the dataset is generated (i) offline,
However, Zhou further teaches:
the dataset is generated (i) offline, (Zhou, page 3, paragraph 3: “In offline RL, we are given a fixed dataset
D
=
s
t
,
a
t
,
r
t
,
s
t
+
1
i
with a finite number of transitions.”)
Regarding claim 86, Jiang, Zhou, and Ie teach the method of claim 82, as cited above.
Jiang further teaches:
sampling a data pair from the dataset; and embedding the items in the slate from the sampled data pair; (Jiang, page 2, paragraph 2: “All documents in a slate along with their positional, contextual biases are jointly encoded into the latent space, which is then sampled and combined with desired conditioning for direct slate generation, i.e. sampling from the learned conditional joint distribution. Therefore, the model first learns which slates give which type of responses and then directly generates similar slates given a desired response vector as the conditioning at inference time.” – The document along with the positional, contextual biases being jointly encoded then sampled is analogous to sampling a data pair from the dataset and that the embedded items in the slate, which are generated from the sampled latent space items, are from the sampled data pair.)
wherein the latent representation of the slate comprises a vector having a dimension d, (Jiang, page 6, paragraph 3: “The model consists of an embedding layer, which encodes documents into 8-dimensional embeddings.” – the 8-dimensional embedding is a vector having a dimension d.) and wherein a combination of the embedded items from the slate comprises a vector having a dimension greater than d. (Jiang, page 6, paragraph 3: “It then concatenates the embeddings of all the documents that form a slate and follows this concatenation with two hidden layers and a final softmax layer that predicts the slate response amongst the 2k possible responses.” – Concatenating all the embeddings is analogous to the combination of the embedded items having a dimension greater than d.)
Regarding claim 87, Jiang, Zhou, and Ie teach the method of claim 86, as cited above.
Jiang further teaches:
the item embeddings are learnable. (Jiang, page 4, last paragraph: “The decoder learns these biases through reconstruction of the input slates from latent variables z with conditions.” – The reconstruction of the input slates from latent variables is analogous to the item embeddings.)
Regarding claim 88, Jiang, Zhou, and Ie teach the method of claim 86, as cited above.
Jiang further teaches:
combining the embedded items from the slate with the associated set of interactions; (Jiang, page 4, last paragraph: “When the encoder encodes the input slate s into the latent space, it learns the joint distribution of the k documents in a fixed order, and thus also encodes any contextual, positional biases between documents and their respective positions into the latent variable z.” – Contextual biases between documents is analogous to the associated set of interactions.)
said combining comprises concatenating the embedded items from the slate with the associated set of interactions; and (Jiang, page 4, paragraph 2: “It transforms a user response vector r into a vector in the conditioning space C that encodes the user engagement metric we wish to optimize for.” And Fig. 2: “The concatenation of s and c makes the input vector to the encoder.”)
the generative model generating the latent representation using the combined embedded items from the slate and associated set of interactions; wherein said generation of the latent representation further comprises an encoder neural network of the encoder encoding the combined embedded items from the slate and associated set of interactions; and (Jiang, page 2, paragraph 2: “All documents in a slate along with their positional, contextual biases are jointly encoded into the latent space, which is then sampled and combined with desired conditioning for direct slate generation, i.e. sampling from the learned conditional joint distribution.”)
Jiang does not explicitly teach:
wherein the encoder neural network is modeled by a first set of parameters.
However, Zhou further teaches:
wherein the encoder neural network is modeled by a first set of parameters. (Zhou, page 4, paragraph 3: “Converting Equation 3 into our problem formulation, the objective of the CVAE is to maximize
log
p
(
a
|
s
)
by maximizing its lower bound:
max
α
,
β
log
p
(
a
|
s
)
≥
max
α
,
β
E
z
~
q
α
log
p
β
a
|
s
,
z
-
D
K
L
q
α
z
|
a
,
s
|
|
P
z
|
s
where
z
is the latent variable,
α
and
β
are the parameters of the encoder and the decoder, respectively.” –
α
is analogous to the first set of parameters.)
Regarding claim 89, Jiang, Zhou, and Ie teach the method of claim 88, as cited above.
Jiang further teaches:
computing a posterior probability distribution from a prior probability distribution over the latent space, the posterior probability distribution corresponding to the embedded items from the slate and associated set of interactions; and (Jiang, page 4, last paragraph: “As usual with CVAEs, the decoder models a distribution
P
θ
s
|
z
,
c
that, conditioned on z, is easy to represent. In our case,
n
=
D
models an independent probability for each document on the slate, represented by a softmax distribution. Note that the documents are only independent to each other conditional on z. In fact, the marginalized posterior
P
θ
s
|
c
=
∫
z
a
P
θ
s
|
z
,
c
P
θ
z
|
c
d
z
can be arbitrarily complex.”)
generating the latent representation using the posterior probability distribution; and (Jiang, page 4, last paragraph: “When the encoder encodes the input slate s into the latent space, it learns the joint distribution of the k documents in a fixed order, and thus also encodes any contextual, positional biases between documents and their respective positions into the latent variable z.”)
wherein the prior probability distribution comprises a Gaussian distribution. (Jiang, page 5, paragraph 1: “To shed light on to what is encoded in the latent space, we simplify the prior distribution of
z
to be a fixed Gaussian distribution
N
(
0
,
I
)
in
R
2
.”)
Regarding claim 90, Jiang, Zhou, and Ie teach the method of claim 88, as cited above.
Jiang further teaches:
the decoder decoding the generated latent representation to reconstruct the slate of item; and the associated set of interactions; (Jiang, page 4, last paragraph: “The decoder learns these biases through reconstruction of the input slates from latent variables z with conditions. At inference time, the decoder reproduces the input slate distribution from the latent variable z with the ideal conditioning, taking into account all the biases learned during training time.” – The conditions is analogous to the associated set of interactions.)
Jiang does not explicitly teach:
wherein the decoder comprises a neural network having a second set of parameters.
However, Zhou further teaches:
wherein the decoder comprises a neural network having a second set of parameters. (Zhou, page 4, paragraph 3: “Converting Equation 3 into our problem formulation, the objective of the CVAE is to maximize
log
p
(
a
|
s
)
by maximizing its lower bound:
max
α
,
β
log
p
(
a
|
s
)
≥
max
α
,
β
E
z
~
q
α
log
p
β
a
|
s
,
z
-
D
K
L
q
α
z
|
a
,
s
|
|
P
z
|
s
where
z
is the latent variable,
α
and
β
are the parameters of the encoder and the decoder, respectively.” –
β
is analogous to the second set of parameters.)
Regarding claim 91, Jiang, Zhou, and Ie teach the method of claim 90, as cited above.
Jiang further teaches:
reconstructing the embedded items from the slate; and (Jiang, page 4, last paragraph: “The decoder learns these biases through reconstruction of the input slates from latent variables z with conditions.”)
generating the reconstructed slate of items from the reconstructed embeddings. (Jiang, page 4, last paragraph: “At inference time, the decoder reproduces the input slate distribution from the latent variable z with the ideal conditioning, taking into account all the biases learned during training time.”)
Regarding claim 92, Jiang, Zhou, and Ie teach the method of claim 91, as cited above.
Jiang further teaches:
deriving logits for a set of item probabilities from the reconstructed embeddings; and (Jiang, page 5, paragraph 2: “This operation produces k vectors of logits for k softmaxes, i.e. the k-head softmax.”)
generating the reconstructed slate of items from the derived logits. (Jiang, page 5, paragraph 2: “At training time, for large document corpora D, we uniformly randomly downsample negative documents and compute only a small subset of the logits for every training example, therefore efficiently scaling the nearest neighbor search to millions of documents with minimal model quality loss.” – See Fig. 2 where the downsampling of the logits is happening on the raw output prior to generating the recommended slate s*.)
Regarding claim 93, Jiang, Zhou, and Ie teach the method of claim 92, as cited above.
Jiang further teaches:
reconstructing the associated set of interactions; and (Jiang, page 4, last paragraph: “The decoder learns these biases through reconstruction of the input slates from latent variables z with conditions. At inference time, the decoder reproduces the input slate distribution from the latent variable z with the ideal conditioning, taking into account all the biases learned during training time.” – The ideal conditioning is the reconstruction of the associated set of interactions.)
Jiang does not explicitly teach:
wherein said training the generative model further comprises: updating the first and second sets of parameters using the reconstructed slate of items and the reconstructed associated set of interactions to optimize a loss function.
However, Zhou further teaches:
wherein said training the generative model further comprises: updating the first and second sets of parameters using the reconstructed slate of items and the reconstructed associated set of interactions to optimize a loss function. (Zhou, page 4, paragraph 3: “The CVAE is trained to reconstruct actions conditioned on the states. Converting Equation 3 into our problem formulation, the objective of the CVAE is to maximize
log
p
(
a
|
s
)
by maximizing its lower bound:
max
α
,
β
log
p
(
a
|
s
)
≥
max
α
,
β
E
z
~
q
α
log
p
β
a
|
s
,
z
-
D
K
L
q
α
z
|
a
,
s
|
|
P
z
|
s
where
z
is the latent variable,
α
and
β
are the parameters of the encoder and the decoder, respectively. This is similar to Equation 3, except that all terms are conditioned on the state
s
. A trained decoder
p
β
a
|
s
,
z
provides a mapping from the latent space to the action space, conditioned on the state.” – Maximizing α and β in the equation would indicate that the first and second parameters are being updated, while the equation is analogous to the loss function that is being optimized.)
Regarding claim 94, Jiang, Zhou, and Ie teach the method of claim 93, as cited above.
Jiang does not explicitly teach:
one of each sampled data pair; a batch of sampled data pairs, and all of the data pairs in the dataset.
However, Zhou further teaches:
one of each sampled data pair; a batch of sampled data pairs, and all of the data pairs in the dataset. (Zhou, page 5, Algorithm 1: “Sample a minibatch of k state-action pairs
s
t
,
a
t
from
D
” – The action state pair is analogous to the data pairs, where sampling a minibatch could mean sampling one, a batch, or all of the data pairs.)
Regarding claim 95, Jiang, Zhou, and Ie teach the method of claim 93, as cited above.
Jiang further teaches:
optimizing the loss function comprises maximizing an evidence lower bound (ELBO) on a task of reconstructing slates of items and associated sets of interactions. (Jiang, page 3, section 3.2: “For this, a variational posterior density
Q
ϕ
z
|
x
parametrized by a vector φ is introduced and we optimize the variational Evidence Lower-Bound (ELBO) on the data log likelihood:”)
Regarding claim 97, Jiang, Zhou, and Ie teach the method of claim 81, as cited above.
Jiang further teaches:
said items comprise identifiers respectively associated with items in the collection; and (Jiang, page 2, paragraph 2: “All documents in a slate along with their positional, contextual biases are jointly encoded into the latent space, which is then sampled and combined with desired conditioning for direct slate generation, i.e. sampling from the learned conditional joint distribution.” – The positional, contextual biases jointly encoded alongside the document is analogous to the identifiers.)
wherein the items in the collection comprise media items, documents, terms, tokens, news articles, e-commerce products, or a combination. (Jiang, page 2, paragraph 1: “Another example considers news articles, the optimal slate has k ordered articles such that every article is read by the user. In general, optimality can be defined as a desired user response vector on the slate and the proposed model should be agnostic to these problem-specific definitions. Solving the slate recommendation problem by direct slate generation differs from ranking in that first, the entire slate is used as a training example instead of single documents, preserving numerous biases encoded into the slate that might influence user responses. Secondly, it does not assume that more relevant documents should necessarily be put in earlier positions in the slate at serving time. Our model directly generates slates, taking into account all the relevant biases learned through training.”)
Regarding claim 98, Jiang, Zhou, and Ie teach the method of claim 81 as cited above.
Jiang does not explicitly teach:
said training of the reinforcement learning agent takes place while the pretrained decoder is integrated into the recommender system, and while the pretrained decoder is frozen.
However, Zhou further teaches:
said training of the reinforcement learning agent takes place while the pretrained decoder is integrated into the recommender system, and while the pretrained decoder is frozen. (Zhou, algorithm 1 has a trained encoder and decoder being initialized prior to the policy training of the reinforcement learning agent which indicates that the training of the reinforcement learning agent takes place while the pretrained decoder is frozen.)
Regarding claim 99, Jiang, Zhou, and Ie teach the method of claim 81, as cited above.
Jiang and Zhou do not explicitly teach:
the recommender system outputs the recommended slate to the user; and
wherein the reinforcement learning agent determines the state based on observed interactions from the user.
However, Ie further teaches:
the recommender system outputs the recommended slate to the user; and (Ie, page 7, paragraph 3: “We consider a setting in which a recommender system is charged with presenting a slate to a user, from which the user selects zero or more items for consumption (e.g., listening to selected music tracks, reading content, watching video content).”)
wherein the reinforcement learning agent determines the state based on observed interactions from the user. (Ie, page 8, paragraph 4: “Transition probability P(s|s,A) reflects the probability that the user transitions to state s when action A is taken at user state s.”)
Regarding claim 101, Jiang, Zhou, and Ie teach the method of claim 81, as cited above.
Jiang does not explicitly teach:
wherein the reinforcement learning agent is defined by a policy; and
wherein said training of the reinforcement learning agent updates the policy to improve a return.
However, Zhou further teaches:
wherein the reinforcement learning agent is defined by a policy; and (Zhou, Fig. 1: “Instead of explicitly matching the action distribution of the agent policy with the behavior policy using divergence metrics such as KL or MMD, we implicitly constrain the policy to output actions within the support of the behavior policy through the latent action space.”)
wherein said training of the reinforcement learning agent updates the policy to improve a return. (Zhou, page 3, paragraph 1: “The general objective of RL is to find a policy that maximizes the expectation of the return
G
t
=
∑
k
=
0
∞
γ
k
r
s
t
+
k
,
a
t
+
k
,
s
t
+
k
+
1
.” And paragraph 3: “The policy
π
θ
is updated following the Deterministic Policy Gradient [23]:
∇
θ
J
θ
=
E
s
~
p
π
∇
θ
π
θ
s
∇
a
Q
π
s
s
,
a
|
a
=
π
θ
(
s
)
”)
Regarding claim 102, Jiang, Zhou, and Ie teach the method of claim 101, as cited above.
Jiang does not explicitly teach:
wherein the recommender system is configured to interact with the user throughout an episode of turns;
wherein in each turn the recommender system recommends a slate of items and the user interacts with one or more of the recommended slate of items;
wherein the return is determined based on a reward over the episode of turns;
wherein the reward is based on a cumulative number of interactions by the user over the episode; and
wherein said training of a reinforcement learning agent comprises a plurality of policy evaluation and policy improvement steps.
However, Zhou further teaches:
wherein the return is determined based on a reward over the episode of turns; (Zhou, page 7, last paragraph: “The true return is calculated by the cumulative discounted reward until termination or up to rt+1000,”)
wherein the reward is based on a cumulative number of interactions by the user over the episode; and (Zhou, page 3, paragraph 5: “Reconsidering this problem, another reasonable objective for offline RL is to maximize the cumulative reward of the MDP under the transitions that have been visited in the dataset.” – The transitions that have been visited being analogous to the interactions by the user over the episode.)
wherein said training of a reinforcement learning agent comprises a plurality of policy evaluation and policy improvement steps. (Zhou, page 6, paragraph 3: “We train the policy for 500 epochs and each epoch has 1000 training steps. The policy is evaluated every 1 epoch over 10 episodes.”)
Jiang and Zhou do not explicitly teach:
wherein the recommender system is configured to interact with the user throughout an episode of turns;
wherein in each turn the recommender system recommends a slate of items and the user interacts with one or more of the recommended slate of items;
However, Ie further teaches:
wherein the recommender system is configured to interact with the user throughout an episode of turns; (Ie, page 22, paragraph 3: “At each stage of interaction with a user, m candidate documents are drawn from PD, from which a slate of size k must be selected for recommendation.” – each stage of interaction with a user indicates multiple stages and, therefore, an episode of turns.)
wherein in each turn the recommender system recommends a slate of items and the user interacts with one or more of the recommended slate of items; (Ie, page 2, last paragraph: “In this work, we develop a new slate decomposition technique called SLATEQ that estimates the long-term value (LTV) of a slate of items by directly using the estimated LTV of the individual items on the slate. This decomposition exploits certain assumptions about the specifics of user choice behavior—i.e., the process by which user preferences dictate selection and/or engagement with items on a slate—but these assumptions are minimal and, we argue below, very natural in many recommender settings.” – The selection and/or engagement with items on a slate are analogous to the user being recommended and interacting with a slate of items during each stage of interaction, e.g., each turn.)
Regarding claim 103, Jiang, Zhou, and Ie teach the method of claim 102, as cited above.
Jiang does not explicitly teach:
wherein the policy evaluation step comprises evaluating a Q-function of an optimal policy; and
wherein the policy improvement step comprises E-greedily maximizing the estimated Q-function.
However, Zhou further teaches:
wherein the policy evaluation step comprises evaluating a Q-function of an optimal policy; and (Zhou, page 3, paragraph 2: “Given a policy π, the action-value function, or Q-function, is defined as
Q
π
s
,
a
=
E
π
G
t
|
S
t
=
s
,
A
t
=
a
. Our method builds on top of the commonly used off-policy actor-critic procedure with a deterministic policy [21, 22]. The Q-function of the deterministic policy π is estimated based on the Bellman Operator:
T
Q
^
π
s
t
,
a
t
=
E
s
t
+
1
r
t
+
γ
Q
^
π
(
s
t
+
1
,
π
s
t
+
1
The policy
π
θ
is updated following the Deterministic Policy Gradient [23]:
∇
θ
J
θ
=
E
s
~
p
π
∇
θ
π
θ
s
∇
a
Q
π
s
s
,
a
|
a
=
π
θ
(
s
)
”)
Jiang and Zhou do not explicitly teach:
wherein the policy improvement step comprises E-greedily maximizing the estimated Q-function.
However, Ie further teaches:
wherein the policy improvement step comprises E-greedily maximizing the estimated Q-function. (Ie, page 9, paragraph 2: “With repetition—i.e., if the updated
Q
π
is used to make recommendations (with some exploration), from which new training data is generated—the process will converge to the optimal Q function. Note that acting greedily w.r.t.
Q
π
requires the ability to compute optimal slates at serving time.”)
Regarding claim 104, Jiang, Zhou, and Ie teach the method of claim 102, as cited above.
Jiang does not explicitly teach:
wherein the policy evaluation step comprises estimating an expected return of a current policy; and
wherein the policy improvement step comprises using gradient ascent on the estimated expected return.
However, Zhou further teaches:
wherein the policy evaluation step comprises estimating an expected return of a current policy; and (Zhou, page 7, last paragraph: “By definition, the Q-value
Q
π
s
t
,
a
t
is equal to the expected return starting from state
s
t
following action
a
t
; our learned Q-function attempts to estimate this value.”)
wherein the policy improvement step comprises using gradient ascent on the estimated expected return. (Zhou, page 3, paragraph 1: “The general objective of RL is to find a policy that maximizes the expectation of the return
G
t
=
∑
k
=
0
∞
γ
k
r
s
t
+
k
,
a
t
+
k
,
s
t
+
k
+
1
.” – Maximizing the expectation of the return is analogous to gradient ascent.)
Claim 96 is rejected under 35 U.S.C. 103 as being unpatentable over Jiang in view of Zhou in view of Ie in view of Liu et al. (Variation Control and Evaluation for Generative Slate Recommendations), hereinafter Liu.
Liu was cited in Applicant’s IDS filed on May 22, 2024.
Regarding claim 96, Jiang, Zhou, and Ie teach the method of claim 93, as cited above.
Jiang further teaches:
a slate reconstruction loss; a prior matching loss; and wherein the prior matching loss comprises a Kullback-Leibler divergence. (Jiang, page 5, paragraph 3: “We train this model as a CVAE by minimizing the sum of the reconstruction loss and the KL divergence term:
L
=
β
KL
Q
ϕ
z
|
s
,
c
|
|
P
θ
z
|
c
-
E
Q
ϕ
z
|
s
,
c
log
P
θ
s
|
z
,
c
where
β
is a function of the training step (Higgins et al., 2017).” – Where the KL term is the prior matching loss and the sum of the reconstruction loss is the slate reconstruction loss.)
Jiang, Zhou, and Ie do not explicitly teach:
a set of interactions reconstruction loss; and
wherein the slate reconstruction loss, the set of interactions reconstruction loss, and/or the prior matching loss are weighted by a hyperparameter; and
However, Liu teaches:
a set of interactions reconstruction loss; and (Liu, page 3, column 1, paragraph 2: “During training, in order to avoid overfitting, the reconstruction loss is calculated by the cross entropy over down-sampled items instead of the entire D. At inference time, the slate is generated by passing the ideal condition 𝒄∗ into the decoder along with a randomly sampled encoding 𝒛 (e.g., from random Gaussian) based on the variational information provided by the conditional prior.” – The reconstruction loss being calculated on the downsampled items using the ideal condition is analogous to the reconstruction loss of the set of interactions.)
wherein the slate reconstruction loss, the set of interactions reconstruction loss, and/or the prior matching loss are weighted by a hyperparameter; and (Liu, page 1, column 2, paragraph 3: “For example, in the case of VAE-based models, the performance depends on a trade-off coefficient 𝛽 [23]—the larger the 𝛽-value during training, the more the model is focused on encoding variation control against the data reconstruction accuracy” – where 𝛽 is the hyperparameter that weights the loss functions.)
Liu is considered analogous to the claimed invention as it is in the same field of endeavor, machine learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to have modified Jiang, Zhou, and Ie, which already teaches optimizing a loss function but does not explicitly teach that the loss function includes a set of interactions reconstruction loss or that the loss is/are weighted by a hyperparameter, to include the teachings of Liu which does teach that the loss function includes a set of interactions reconstruction loss or that the loss is/are weighted by a hyperparameter in order to "push the model to favor one of the terms over the
other." (Liu, page 3, column 1, last paragraph)
Claim 100 is rejected under 35 U.S.C. 103 as being unpatentable over Jiang in view of Zhou in view of Ie in view of Huang et al. (State Encoders in Reinforcement Learning for Recommendation: A Reproducibility Study), hereinafter Huang.
Regarding claim 100, Jiang, Zhou, and Ie teach the method of claim 81, as cited above.
Jiang does not explicitly teach:
(i) a belief encoder for determining the state based on observed interactions from the user;
(ii) the belief encoder is modeled by a gated recurrent unit (GRU); and
(iii) the reinforcement learning agent comprises an actor-critic algorithm.
However, Zhou further teaches:
(iii) the reinforcement learning agent comprises an actor-critic algorithm. (Zhou, page 3, paragraph 2: “Our method builds on top of the commonly used off-policy actor-critic procedure with a deterministic policy [21, 22].”)
Jiang, Zhou, and Ie do not explicitly teach:
(i) a belief encoder for determining the state based on observed interactions from the user;
(ii) the belief encoder is modeled by a gated recurrent unit (GRU); and
However, Huang teaches:
(i) a belief encoder for determining the state based on observed interactions from the user; (Huang, page 2738, column 2, paragraph 2: “the state encoder that encodes the state– a user’s historical interactions–into a dense representation that is used to estimate the user’s preference and the value of state-action pairs;” – The historical interactions is analogous to the observed interactions from the user.)
(ii) the belief encoder is modeled by a gated recurrent unit (GRU); and (Huang, page 2741, column 2, paragraph 1: “Besides the four state encoders proposed by Liu et al. [35], we expand the comparison by adding three more state encoders based on typical neural network architectures: MLP, GRU and CNN, which are widely used in recommendation methods to generate representations according to historical user interactions.”)
Huang is considered analogous to the claimed invention as it is in the same field of endeavor, machine learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to have modified Jiang, Zhou, and Ie, which already teaches the recommender system having an encoder but does not explicitly teach that the encoder is a belief encoder that is modeled by a GRU and is based on observed interactions from the user, to include the teachings of Huang which does teach that the encoder is a belief encoder that is modeled by a GRU and is based on observed interactions from the user in order to provide significantly higher performance. (Huang, page 2, column 1, paragraph 2)
Claims 105-107 are rejected under 35 U.S.C. 103 as being unpatentable over Ie in view of Zhou in view of Jiang in view of Liu.
Regarding claim 105, Ie teaches the method:
determining, by an agent comprising a neural network based reinforcement learning model, … the action representing a slate of items from the collection being recommended based on a state, (Ie, page 8, paragraph 1: “However, our methods apply readily when certain items cannot be recommended at particular states by specifying
I
s
for each
s
∈
S
and restricting
A
s
to subsets of
I
s
.”) the state being based on observed interactions received from the user; (Ie, page 7, last paragraph: “The states
S
typically reflect user state. This includes relatively static user features such as demographics, declared interests, and other user attributes, as well as more dynamic user features, such as user context (e.g., time of day).” – The user features and user attributes is analogous to the observed interactions received from the user.)
outputting the recommended slate of items to the user; (Ie, page 7, paragraph 3: “We consider a setting in which a recommender system is charged with presenting a slate to a user, from which the user selects zero or more items for consumption (e.g., listening to selected music tracks, reading content, watching video content).”)
Ie does not explicitly teach:
an action in a continuous low-dimensional latent space
receiving the action by a ranker comprising a neural network-based decoder;
generating, by the ranker, a recommended slate of items from the received action; and
wherein the decoder comprises a decoder of a pretrained variational autoencoder, the autoencoder being pretrained to optimize a loss function comprising a slate reconstruction loss for slates of items, a reconstruction loss for sets of interactions associated with the reconstruction loss, and a prior matching loss;
wherein the agent is trained to improve a return based on a reward function while the decoder of the trained autoencoder is frozen.
However, Zhou teaches:
an action in a continuous low-dimensional latent space, (Zhou, page 2, paragraph 1: “ Our insight is that we can learn a policy in the latent action space of the CVAE and then use its decoder to output an action in the original action space of the environment. The latent action space implicitly constrains the policy by construction” – The action being a recommended slate of items from the collection is taught by Ie above, the diverse actions being generalized from the decoder and Figure 2 on page 4 shows that the action is being generated based on a state.)
receiving the action by a ranker comprising a neural network-based decoder; (Zhou, page 2, paragraph 1: “Our insight is that we can learn a policy in the latent action space of the CVAE and then use its decoder to output an action in the original action space of the environment.” And paragraph 2: “We demonstrate that PLAS allows generalization within the dataset and can provide consistently good performance for datasets with diverse actions.”)
wherein the decoder comprises a decoder of a pretrained variational autoencoder, (Zhou, page 4, paragraph 3: “Given a static dataset, we use a conditional variational autoencoder (CVAE) to model the behavior policy
p
(
a
|
s
)
,”) the autoencoder being pretrained to optimize a loss function (Zhou, page 4, paragraph 3: “The CVAE is trained to reconstruct actions conditioned on the states. Converting Equation 3 into our problem formulation, the objective of the CVAE is to maximize
log
p
(
a
|
s
)
by maximizing its lower bound:
max
α
,
β
log
p
(
a
|
s
)
≥
max
α
,
β
E
z
~
q
α
log
p
β
a
|
s
,
z
-
D
K
L
q
α
z
|
a
,
s
|
|
P
z
|
s
where
z
is the latent variable,
α
and
β
are the parameters of the encoder and the decoder, respectively. This is similar to Equation 3, except that all terms are conditioned on the state
s
. A trained decoder
p
β
a
|
s
,
z
provides a mapping from the latent space to the action space, conditioned on the state.” – Maximizing α and β in the equation would indicate that the first and second parameters are being updated, while the equation is analogous to the loss function that is being optimized.) comprising a slate reconstruction loss for slates of items, (Zhou, page 4, paragraph 1: “The term
log
p
θ
(
x
|
z
)
(where z is sampled from
q
ϕ
(
z
|
x
)
) represents the reconstruction loss.”) … and a prior matching loss; (Zhou, page 4, paragraph 1: “The second term is the KL-divergence between the encoder output and the prior of z, which is usually set to be N(0; 1).”)
wherein the agent is trained to improve a return based on a reward function while the decoder of the trained autoencoder is frozen. (Zhou, algorithm 1 has a trained encoder and decoder being initialized prior to the policy training of the reinforcement learning agent which indicates that the training of the reinforcement learning agent takes place while the pretrained decoder is frozen.)
Zhou is considered analogous to the claimed invention as it is in the same field of endeavor, machine learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to have modified Ie, which already teaches a reinforcement learning agent to provide a slate of item recommendations to a user but does not explicitly teach the action is in a continuous low-dimensional space or the action is received by a decoder, to include the teachings of Zhou which does teach the action is in a continuous low-dimensional latent space or the action is received by a decoder as the latent space implicitly constrains the policy and “it can be naturally satisfied without affecting the optimization of the other components and without being restricted by the density of the behavior policy distribution.” (Zhou, page 2, paragraph 1)
Ie and Zhou do not explicitly teach:
generating, by the ranker, a recommended slate of items from the received action; and
a loss function comprising … a reconstruction loss for sets of interactions associated with the reconstruction loss…
However, Jiang teaches:
generating, by the ranker, a recommended slate of items from the received action; and (Jiang, page 5, paragraph 4: “During inference, output slates are generated by first sampling z from the conditionally learned prior distribution N(µ*, σ*), concatenating with the ideal condition c* =Φ(r*), and passed into the decoder, generating
x
1
,
⋯
,
x
k
from the learned
P
θ
(
s
|
z
,
c
)
, and finally taking argmax over the dot-products with the full embedding matrix independently for each position i=1,...,k.” – Where
x
1
,
⋯
,
x
k
is the recommended slate of items being generated.)
Jiang is considered analogous to the claimed invention as it is in the same field of endeavor, machine learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to have modified Ie and Zhou, which already teaches the recommended slate of items being generated and provided to the user but does not explicitly teach that the recommended slate of items is generated by the ranker, to include the teachings of Jiang which does teach that the recommended slate of items is generated by the ranker since this method “outperforms popular comparable ranking methods consistently on various scales of documents corpora.” (Jiang, abstract)
Ie, Zhou, and Jiang do not explicitly teach:
a loss function comprising … a reconstruction loss for sets of interactions associated with the reconstruction loss…
However, Liu teaches:
a loss function comprising … a reconstruction loss for sets of interactions associated with the reconstruction loss… (Liu, page 3, column 1, paragraph 2: “During training, in order to avoid overfitting, the reconstruction loss is calculated by the cross entropy over down-sampled items instead of the entire D. At inference time, the slate is generated by passing the ideal condition 𝒄∗ into the decoder along with a randomly sampled encoding 𝒛 (e.g., from random Gaussian) based on the variational information provided by the conditional prior.” – The reconstruction loss being calculated on the downsampled items using the ideal condition is analogous to the reconstruction loss of the set of interactions.)
Liu is considered analogous to the claimed invention as it is in the same field of endeavor, machine learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to have modified Ie, Zhou, and Jiang, which already teaches a loss function but does not explicitly teach the loss function comprises a reconstruction loss for sets of interactions associated with the reconstruction loss, to include the teachings of Liu which does teach the loss function comprises a reconstruction loss for sets of interactions associated with the reconstruction loss in order to avoid overfitting. (Liu, page 3, column 1, paragraph 3)
Regarding claim 106, Ie, Zhou, Jiang, and Huang teach the method of claim 105, as cited above.
Ie and Zhou do not explicitly teach:
wherein the interactions comprise selections among the items in the slate; and wherein the set of interactions indicate whether the items in the slate were selected or not selected;
wherein the autoencoder is trained using a dataset comprising slates of items and associated sets of interactions.
However, Jiang further teaches:
wherein the interactions comprise selections among the items in the slate; and wherein the set of interactions indicate whether the items in the slate were selected or not selected; (Jiang, page 3, section 3.1: “Then let
r
=
r
1
,
⋯
,
r
k
be the user response vector, where
r
i
∈
R
is the user’s response on document
d
i
. For example, if the problem is to maximize the number of clicks on a slate, then let
r
i
∈
0,1
denote whether the document
d
i
is clicked, and thus an optimal slate
s
=
d
1
,
d
2
,
⋯
,
d
k
where
d
i
∈
D
is such that
s
maximizes
E
∑
i
=
1
k
r
i
.” – The user response vector includes the user’s response on the document which is analogous to the set of interactions. Whether the user clicked the document also indicates that the vector comprises whether the items in the slate were selected or not.)
wherein the autoencoder is trained using a dataset comprising slates of items and associated sets of interactions. (Jiang, page 4, paragraph 2: “The decoder learns these biases through reconstruction of the input slates from latent variables z with conditions.” And page 5, section 4.1, paragraph 1: “The simulator generates a random matrix
W
~
N
μ
,
σ
k
×
n
×
k
×
n
where each element
W
i
,
d
i
,
j
,
d
j
represents the interaction between document
d
i
at position
i
and document
d
j
at position
j
, and
n
=
D
.” – Where the conditions is analogous to the associated sets of interactions.)
Regarding claim 107, Ie, Zhou, Jiang, and Liu teach the method of claim 106, as cited above.
Ie and Zhou do not explicitly teach:
the decoder reconstructs embedded items from the slate and generates the reconstructed slate of items from the reconstructed item embeddings.
However, Jiang further teaches:
the decoder reconstructs embedded items from the slate (Jiang, page 4, last paragraph: “The decoder learns these biases through reconstruction of the input slates from latent variables z with conditions.” – The reconstruction of the input slates from latent variables is analogous to the item embeddings.) and generates the reconstructed slate of items from the reconstructed item embeddings. (Jiang, page 4, last paragraph: “At inference time, the decoder reproduces the input slate distribution from the latent variable z with the ideal conditioning, taking into account all the biases learned during training time.”)
Claims 108, 116, and 118-121 are rejected under 35 U.S.C. 103 as being unpatentable over Ie in view of Zhou in view of Ie, Jian, et al. (US20210081753), hereinafter Jain.
Regarding claim 108, Ie teaches the system:
a reinforcement learning agent trained to determine an action… the action representing a slate of items from the collection being recommended based on a state, (Ie, page 8, paragraph 1: “However, our methods apply readily when certain items cannot be recommended at particular states by specifying
I
s
for each
s
∈
S
and restricting
A
s
to subsets of
I
s
.”) the state being based on observed interactions received from the user; and (Ie, page 7, last paragraph: “The states S typically reflect user state. This includes relatively static user features such as demographics, declared interests, and other user attributes, as well as more dynamic user features, such as user context (e.g., time of day).” – The user features and user attributes is analogous to the observed interactions received from the user.)
…output the recommended slate of items to the user; (Ie, page 7, paragraph 3: “We consider a setting in which a recommender system is charged with presenting a slate to a user, from which the user selects zero or more items for consumption (e.g., listening to selected music tracks, reading content, watching video content).”)
Ie does not explicitly teach:
A recommender system implemented by a processor and memory
a reinforcement learning agent trained to determine an action in a low-dimensional latent space,
a ranker comprising a neural network based decoder pretrained to generate a recommended slate of items from the determined action
wherein the decoder comprises a decoder of a pretrained variational autoencoder. (Zhou, page 4, paragraph 3: “Given a static dataset, we use a conditional variational autoencoder (CVAE) to model the behavior policy
p
(
a
|
s
)
,”)
However, Zhou teaches:
a reinforcement learning agent trained to determine an action in a low-dimensional latent space, (Zhou, page 2, paragraph 1: “ Our insight is that we can learn a policy in the latent action space of the CVAE and then use its decoder to output an action in the original action space of the environment. The latent action space implicitly constrains the policy by construction” – The action being a recommended slate of items from the collection is taught by Ie above, the diverse actions being generalized from the decoder and Figure 2 on page 4 shows that the action is being generated based on a state.)
a ranker comprising a neural network based decoder pretrained to generate a recommended slate of items from the determined action (Zhou, page 2, paragraph 1: “Our insight is that we can learn a policy in the latent action space of the CVAE and then use its decoder to output an action in the original action space of the environment.” And paragraph 2: “We demonstrate that PLAS allows generalization within the dataset and can provide consistently good performance for datasets with diverse actions.”)
wherein the decoder comprises a decoder of a pretrained variational autoencoder. (Zhou, page 4, paragraph 3: “Given a static dataset, we use a conditional variational autoencoder (CVAE) to model the behavior policy
p
(
a
|
s
)
,”)
Zhou is considered analogous to the claimed invention as it is in the same field of endeavor, machine learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to have modified Ie, which already teaches a reinforcement learning agent to provide a slate of item recommendations to a user but does not explicitly teach the action is in a continuous low-dimensional space or the action is received by a decoder, to include the teachings of Zhou which does teach the action is in a continuous low-dimensional latent space or the action is received by a decoder as the latent space implicitly constrains the policy and “it can be naturally satisfied without affecting the optimization of the other components and without being restricted by the density of the behavior policy distribution.” (Zhou, page 2, paragraph 1)
Ie and Zhou do not explicitly teach:
A recommender system implemented by a processor and memory
However, Jain teaches:
A recommender system implemented by a processor and memory (Jian, paragraphs 0015-0016 describes a recommender system and paragraph 0119: “Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both.”)
Jain is considered analogous to the claimed invention as it is in the same field of endeavor, machine learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to have modified Ie and Zhou, which already teaches a recommender system but does not explicitly teach the recommender system is implemented by a processor and memory, to include the teachings of Jain which does teach the recommender system is implemented by a processor and memory as implementing a reinforcement learning system and an autoencoder on a computer is well-known in the art.
Regarding claim 116, Ie, Zhou, and Jain teach the system of claim 108, as cited above.
Ie does not explicitly teach:
the agent is trained while the trained decoder is integrated into the recommender system, and while the trained decoder is frozen.
However, Zhou further teaches:
the agent is trained while the trained decoder is integrated into the recommender system, and while the trained decoder is frozen. (Zhou, algorithm 1 has a trained encoder and decoder being initialized prior to the policy training of the reinforcement learning agent which indicates that the training of the reinforcement learning agent takes place while the pretrained decoder is frozen.)
Regarding claim 118, Ie, Zhou, and Jain teach the system of claim 108, as cited above.
Ie does not explicitly teach:
the agent is modeled by an actor-critic algorithm.
However, Zhou further teaches:
the agent is modeled by an actor-critic algorithm. (Zhou, page 3, paragraph 2: “Our method builds on top of the commonly used off-policy actor-critic procedure with a deterministic policy [21, 22].”)
Regarding claim 119, Ie, Zhou, and Jain teach the method of claim 108, as cited above.
Ie further teaches:
wherein the recommender system is configured to interact with the user throughout an episode of turns; (Ie, page 22, paragraph 3: “At each stage of interaction with a user, m candidate documents are drawn from PD, from which a slate of size k must be selected for recommendation.” – each stage of interaction with a user indicates multiple stages and, therefore, an episode of turns.)
wherein in each turn the recommender system recommends a slate of items and the user interacts with one or more of the recommended slate of items; and (Ie, page 2, last paragraph: “In this work, we develop a new slate decomposition technique called SLATEQ that estimates the long-term value (LTV) of a slate of items by directly using the estimated LTV of the individual items on the slate. This decomposition exploits certain assumptions about the specifics of user choice behavior—i.e., the process by which user preferences dictate selection and/or engagement with items on a slate—but these assumptions are minimal and, we argue below, very natural in many recommender settings.” – The selection and/or engagement with items on a slate are analogous to the user being recommended and interacting with a slate of items during each stage of interaction, e.g., each turn.)
Ie does not explicitly teach:
wherein the reinforcement learning agent is defined by a policy; and
wherein said training of the reinforcement learning agent updates the policy to improve a return;
wherein the return is determined based on a reward over the episode of turns.
However, Zhou further teaches:
wherein the reinforcement learning agent is defined by a policy; and (Zhou, Fig. 1: “Instead of explicitly matching the action distribution of the agent policy with the behavior policy using divergence metrics such as KL or MMD, we implicitly constrain the policy to output actions within the support of the behavior policy through the latent action space.”)
wherein said training of the reinforcement learning agent updates the policy to improve a return; (Zhou, page 3, paragraph 1: “The general objective of RL is to find a policy that maximizes the expectation of the return
G
t
=
∑
k
=
0
∞
γ
k
r
s
t
+
k
,
a
t
+
k
,
s
t
+
k
+
1
” And paragraph 3: “The policy πθ is updated following the Deterministic Policy Gradient [23]:
∇
θ
J
θ
=
E
s
~
p
π
∇
θ
π
θ
s
∇
a
Q
π
s
s
,
a
|
a
=
π
θ
(
s
)
”)
wherein the return is determined based on a reward over the episode of turns. (Zhou, page 7, last paragraph: “The true return is calculated by the cumulative discounted reward until termination or up to rt+1000,”)
Regarding claim 120, Ie, Zhou, and Jain teach the system of claim 108, as cited above.
Ie does not explicitly teach:
reward is based on a cumulative number of interactions by the user over the episode.
However, Zhou further teaches:
the reward is based on a cumulative number of interactions by the user over the episode. (Zhou, page 3, paragraph 5: “Reconsidering this problem, another reasonable objective for offline RL is to maximize the cumulative reward of the MDP under the transitions that have been visited in the dataset.” – The transitions that have been visited being analogous to the interactions by the user over the episode.)
Regarding claim 121, Ie, Zhou, and Jain teach the system of claim 108, as cited above.
Ie further teaches:
wherein the recommender system is embodied in a second stage of a two-stage information retrieval system; and wherein the information retrieval system further comprises a first stage for retrieving a collection of items in response to a request. (Ie, page 28, paragraph 1: “The system is typical of many practical recommender systems with two main components. A candidate generator retrieves a small subset (hundreds) of items from a large corpus that best match a user context. The ranker scores/ranks candidates using a DNN with both user context and item features as input. It optimizes a combination of several objectives (e.g., clicks, expected engagement, several other factors).” – Where the candidate generator is analogous to the information retrieval system and the ranker is analogous to the recommender system.)
Claims 109-113, and 115 are rejected under 35 U.S.C. 103 as being unpatentable over Ie in view of Zhou in view of Jain in view of Jiang.
Regarding claim 109, Ie, Zhou, and Jain teach the system of claim 108, as cited above.
Ie, Zhou, and Jain do not explicitly teach:
wherein the decoder is pretrained to reconstruct a slate of items from a representation in the latent space of a combination of the slate of items and a set of interactions associated with the slate of items;
wherein interactions in the set of interactions are respectively associated with items in the slate of items.
However, Jiang teaches:
wherein the decoder is pretrained to reconstruct a slate of items from a representation in the latent space of a combination of the slate of items and a set of interactions associated with the slate of items; (Jiang, page 4, paragraph 2: “The decoder learns these biases through reconstruction of the input slates from latent variables z with conditions.” – Where the conditions is analogous to the associated sets of interactions.)
wherein interactions in the set of interactions are respectively associated with items in the slate of items. (Jiang, page 5, section 4.1, paragraph 1: “The simulator generates a random matrix
W
~
N
μ
,
σ
k
×
n
×
k
×
n
where each element
W
i
,
d
i
,
j
,
d
j
represents the interaction between document
d
i
at position
i
and document
d
j
at position
j
, and
n
=
D
.”)
Jiang is considered analogous to the claimed invention as it is in the same field of endeavor, machine learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to have modified Ie, Zhou, and Jain, which already teaches a reinforcement learning system with a decoder to provide recommendations to users but does not explicitly teach the decoder being pretrained to reconstruct a slate of items, to include the teachings of Jiang which does teach the decoder being pretrained to reconstruct a slate of items since this method “outperforms popular comparable ranking methods consistently on various scales of documents corpora.” (Jiang, abstract)
Regarding claim 110, Ie, Zhou, Jain, and Jiang teach the system of claim 109, as cited above.
Ie, Zhou, and Jain do not explicitly teach:
wherein the interactions comprise selections among the items in the slate; wherein the set of interactions comprises a vector indicating whether the items in the slate were selected or not selected; wherein the set of interactions indicates that a plurality of items in the slate was selected; and wherein the interactions comprise user clicks.
However, Jiang further teaches:
wherein the interactions comprise selections among the items in the slate; wherein the set of interactions comprises a vector indicating whether the items in the slate were selected or not selected; wherein the set of interactions indicates that a plurality of items in the slate was selected; and wherein the interactions comprise user clicks. (Jiang, page 3, section 3.1: “Then let
r
=
r
1
,
⋯
,
r
k
be the user response vector, where
r
i
∈
R
is the user’s response on document
d
i
. For example, if the problem is to maximize the number of clicks on a slate, then let
r
i
∈
0,1
denote whether the document
d
i
is clicked, and thus an optimal slate
s
=
d
1
,
d
2
,
⋯
,
d
k
where
d
i
∈
D
is such that
s
maximizes
E
∑
i
=
1
k
r
i
.” – The user response vector includes the user’s response on the document which is analogous to the set of interactions. Whether the user clicked the document also indicates that the vector comprises whether the items in the slate were selected or not.)
Regarding claim 111, Ie, Zhou, and Jain teach the system of claim 108, as cited above.
Ie, Zhou, and Jain do not explicitly teach:
wherein the autoencoder is trained using a dataset comprising a plurality of logged interactions; wherein the plurality of logged interactions comprise a plurality of data pairs, each data pair comprising a slate of items and an associated set of interactions.
However, Jiang teaches:
wherein the autoencoder is trained using a dataset comprising a plurality of logged interactions; wherein the plurality of logged interactions comprise a plurality of data pairs, each data pair comprising a slate of items and an associated set of interactions. (Jiang, page 5, section 4.1, paragraph 1: “The simulator generates a random matrix
W
~
N
μ
,
σ
k
×
n
×
k
×
n
where each element
W
i
,
d
i
,
j
,
d
j
represents the interaction between document
d
i
at position
i
and document
d
j
at position
j
, and
n
=
D
.”) – The matrix representing the interactions between the documents is analogous to the data pairs comprising the slate of items and the associated interactions.)
Jiang is considered analogous to the claimed invention as it is in the same field of endeavor, machine learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to have modified Ie, Zhou, and Jain, which already teaches a reinforcement learning system with an autoencoder to provide recommendations for users but does not explicitly teach that the autoencoder is trained with a dataset comprising logged interactions of data pairs with a slate of items and associated set of interactions, to include the teachings of Jiang which does teach that the autoencoder is trained with a dataset comprising logged interactions of data pairs with a slate of items and associated set of interactions since this method “outperforms popular comparable ranking methods consistently on various scales of documents corpora.” (Jiang, abstract)
Regarding claim 112, Ie, Zhou, and Jain teach the system of claim 108, as cited above.
Ie does not explicitly teach:
wherein the autoencoder comprises the decoder and a neural network-based encoder;
wherein the encoder is modeled by a first set of parameters; and
wherein the decoder is modeled by a second set of parameters;
wherein the autoencoder is trained by updating the first and second sets of parameters to optimize a loss function;
wherein the decoder decodes the action to reconstruct embedded items from the slate;
wherein the decoder generates the reconstructed slate of items from the reconstructed item embeddings; and
wherein the decoder derives logits for a set of item probabilities from the reconstructed embeddings and generates the reconstructed slate of items from the derived logits.
However, Zhou further teaches:
wherein the encoder is modeled by a first set of parameters; and wherein the decoder is modeled by a second set of parameters; (Zhou, page 4, paragraph 3: “Converting Equation 3 into our problem formulation, the objective of the CVAE is to maximize
log
p
(
a
|
s
)
by maximizing its lower bound:
max
α
,
β
log
p
(
a
|
s
)
≥
max
α
,
β
E
z
~
q
α
log
p
β
a
|
s
,
z
-
D
K
L
q
α
z
|
a
,
s
|
|
P
z
|
s
where
z
is the latent variable,
α
and
β
are the parameters of the encoder and the decoder, respectively. This is similar to Equation 3, except that all terms are conditioned on the state
s
. A trained decoder
p
β
a
|
s
,
z
provides a mapping from the latent space to the action space, conditioned on the state.” – α and β are analogous to the first and second parameters, respectively.)
wherein the autoencoder is trained by updating the first and second sets of parameters to optimize a loss function; (Zhou, page 4, paragraph 3: “The CVAE is trained to reconstruct actions conditioned on the states. Converting Equation 3 into our problem formulation, the objective of the CVAE is to maximize
log
p
(
a
|
s
)
by maximizing its lower bound:
max
α
,
β
log
p
(
a
|
s
)
≥
max
α
,
β
E
z
~
q
α
log
p
β
a
|
s
,
z
-
D
K
L
q
α
z
|
a
,
s
|
|
P
z
|
s
where
z
is the latent variable,
α
and
β
are the parameters of the encoder and the decoder, respectively. This is similar to Equation 3, except that all terms are conditioned on the state
s
. A trained decoder
p
β
a
|
s
,
z
provides a mapping from the latent space to the action space, conditioned on the state.” – Maximizing α and β in the equation would indicate that the first and second parameters are being updated, while the equation is analogous to the loss function that is being optimized.)
Ie, Zhou, and Jain do not explicitly teach:
wherein the autoencoder comprises the decoder and a neural network-based encoder;
wherein the decoder decodes the action to reconstruct embedded items from the slate;
wherein the decoder generates the reconstructed slate of items from the reconstructed item embeddings; and
wherein the decoder derives logits for a set of item probabilities from the reconstructed embeddings and generates the reconstructed slate of items from the derived logits.
However, Jiang teaches:
wherein the autoencoder comprises the decoder and a neural network-based encoder; (Jiang, Fig. 2: “Structure of List-CVAE for both (a) training and (b) inference. s = (d1,d2,...,dk) is the input slate.”)
wherein the decoder decodes the action to reconstruct embedded items from the slate; (Jiang, page 4, last paragraph: “The decoder learns these biases through reconstruction of the input slates from latent variables z with conditions.” – The reconstruction of the input slates from latent variables is analogous to the item embeddings.)
wherein the decoder generates the reconstructed slate of items from the reconstructed item embeddings; and (Jiang, page 4, last paragraph: “The decoder learns these biases through reconstruction of the input slates from latent variables z with conditions. At inference time, the decoder reproduces the input slate distribution from the latent variable z with the ideal conditioning, taking into account all the biases learned during training time.” – The conditions is analogous to the associated set of interactions.)
wherein the decoder derives logits for a set of item probabilities from the reconstructed embeddings (Jiang, page 5, paragraph 2: “This operation produces k vectors of logits for k softmaxes, i.e. the k-head softmax.”) and generates the reconstructed slate of items from the derived logits. (Jiang, page 5, paragraph 2: “At training time, for large document corpora D, we uniformly randomly downsample negative documents and compute only a small subset of the logits for every training example, therefore efficiently scaling the nearest neighbor search to millions of documents with minimal model quality loss.” – See Fig. 2 where the downsampling of the logits is happening on the raw output prior to generating the recommended slate s*.)
Jiang is considered analogous to the claimed invention as it is in the same field of endeavor, machine learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to have modified Ie, Zhou, and Jain, which already teaches a reinforcement learning system with an autoencoder to provide recommendations for users but does not explicitly teach the decoder reconstructs the slate of items from reconstructed embeddings or that the decoder derives logits to generate the slate of items, to include the teachings of Jiang which does teach the decoder reconstructs the slate of items from reconstructed embeddings or that the decoder derives logits to generate the slate of items since this method “outperforms popular comparable ranking methods consistently on various scales of documents corpora.” (Jiang, abstract)
Regarding claim 113, Ie, Zhou, and Jain teach the system of claim 108, as cited above.
Ie, Zhou, and Jain do not explicitly teach:
optimizing the loss function comprises maximizing an evidence lower bound (ELBO) on a task of reconstructing slates of items and associated sets of interactions.
However, Jiang teaches:
optimizing the loss function comprises maximizing an evidence lower bound (ELBO) on a task of reconstructing slates of items and associated sets of interactions. (Jiang, page 3, section 3.2: “For this, a variational posterior density
Q
ϕ
z
|
x
parametrized by a vector φ is introduced and we optimize the variational Evidence Lower-Bound (ELBO) on the data log likelihood:”)
Jiang is considered analogous to the claimed invention as it is in the same field of endeavor, machine learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to have modified Ie, Zhou, and Jain, which already teaches optimizing a loss function but does not explicitly teach maximizing an ELBO on a tasks of reconstructing slates of items and associated sets of interactions, to include the teachings of Jiang which does teach maximizing the ELBO since this method “outperforms popular comparable ranking methods consistently on various scales of documents corpora.” (Jiang, abstract)
Claim 114 is rejected under 35 U.S.C. 103 as being unpatentable over Ie in view of Zhou in view of Jain in view of Liu.
Regarding claim 114, Ie, Zhou, and Jain teach the system of claim 108, as cited above.
Ie does not explicitly teach:
a slate reconstruction loss;
a set of interactions reconstruction loss; and
a prior matching loss; and
wherein the slate reconstruction loss, the set of interactions reconstruction loss, and/or the prior matching loss are weighted by a hyperparameter.
However, Zhou further teaches:
a slate reconstruction loss; (Zhou, page 4, paragraph 1: “The term
log
p
θ
(
x
|
z
)
(where z is sampled from
q
ϕ
(
z
|
x
)
) represents the reconstruction loss.”)
a prior matching loss; and (Zhou, page 4, paragraph 1: “The second term is the KL-divergence between the encoder output and the prior of z, which is usually set to be N(0; 1).”)
Ie, Zhou, and Jain do not explicitly teach:
a set of interactions reconstruction loss; and
wherein the slate reconstruction loss, the set of interactions reconstruction loss, and/or the prior matching loss are weighted by a hyperparameter.
However, Liu teaches:
a set of interactions reconstruction loss; and (Liu, page 3, column 1, paragraph 2: “During training, in order to avoid overfitting, the reconstruction loss is calculated by the cross entropy over down-sampled items instead of the entire D. At inference time, the slate is generated by passing the ideal condition 𝒄∗ into the decoder along with a randomly sampled encoding 𝒛 (e.g., from random Gaussian) based on the variational information provided by the conditional prior.” – The reconstruction loss being calculated on the downsampled items using the ideal condition is analogous to the reconstruction loss of the set of interactions.)
wherein the slate reconstruction loss, the set of interactions reconstruction loss, and/or the prior matching loss are weighted by a hyperparameter. (Liu, page 1, column 2, paragraph 3: “For example, in the case of VAE-based models, the performance depends on a trade-off coefficient 𝛽 [23]—the larger the 𝛽-value during training, the more the model is focused on encoding variation control against the data reconstruction accuracy” – where 𝛽 is the hyperparameter that weights the loss functions.)
Liu is considered analogous to the claimed invention as it is in the same field of endeavor, machine learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to have modified Ie, Zhou, and Jain, which already teaches optimizing a loss function but does not explicitly teach that the loss function includes a set of interactions reconstruction loss or that the loss is/are weighted by a hyperparameter, to include the teachings of Liu which does teach that the loss function includes a set of interactions reconstruction loss or that the loss is/are weighted by a hyperparameter in order to "push the model to favor one of the terms over the
other." (Liu, page 3, column 1, last paragraph)
Claim 117 is rejected under 35 U.S.C. 103 as being unpatentable over Ie in view of Zhou in view of Jain in view of Huang.
Regarding claim 117, Ie, Zhou, and Jain teach the system of claim 108, as cited above.
Ie, Zhou, and Jain do not explicitly teach:
a belief encoder configured to determine the state based on observed interactions from the user; and
wherein the belief encoder is modeled by a gated recurrent unit (GRU).
However, Huang teaches:
a belief encoder configured to determine the state based on observed interactions from the user; and (Huang, page 2738, column 2, paragraph 2: “the state encoder that encodes the state– a user’s historical interactions–into a dense representation that is used to estimate the user’s preference and the value of state-action pairs;” – The historical interactions is analogous to the observed interactions from the user.)
wherein the belief encoder is modeled by a gated recurrent unit (GRU). (Huang, page 2741, column 2, paragraph 1: “Besides the four state encoders proposed by Liu et al. [35], we expand the comparison by adding three more state encoders based on typical neural network architectures: MLP, GRU and CNN, which are widely used in recommendation methods to generate representations according to historical user interactions.”)
Huang is considered analogous to the claimed invention as it is in the same field of endeavor, machine learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to have modified Ie, Zhou, and Jain, which already teaches the recommender system having an encoder but does not explicitly teach that the encoder is a belief encoder that is modeled by a GRU and is based on observed interactions from the user, to include the teachings of Huang which does teach that the encoder is a belief encoder that is modeled by a GRU and is based on observed interactions from the user in order to provide significantly higher performance. (Huang, page 2, column 1, paragraph 2)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Sunehag (US20200279162)
Xin et al. (Self-supervised Reinforcement Learning for Recommender Systems)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JACQUELINE MEYER whose telephone number is (703)756-5676. The examiner can normally be reached M-F 8:00 am - 4:30 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tamara Kyle can be reached at 571-272-4241. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.C.M./Examiner, Art Unit 2144
/TAMARA T KYLE/Supervisory Patent Examiner, Art Unit 2144
Prosecution Insights