DETAILED ACTION
Status of the Claims
The following is a Final Office Action in response to amendments and remarks filed 24 August 2026.
Claims 1, 3-4, 12, 14-15, and 19 have been amended.
Claims 1-20 are pending and have been examined.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicants argue that the 35 U.S.C. 101 rejection under the Alice Corp. vs. CLS Bank Int’l be withdrawn; however the Examiner respectfully disagrees. As an initial note, the arguments are not compliant under 37 CFR 1.111(b) as they amount to a mere allegation of patent eligibility based upon a bare assertion of improvement. The Examiner respectfully does not find the assertion persuasive because a bare assertion of an improvement without the detail necessary to be apparent is not sufficient to show an improvement (MPEP 2106.04(d)(1) (discussing MPEP 2106.05(a)). That is, the Examiner does not find any evidence that the claimed aspects are any improvement over conventional systems. This argument also appears to be that the claims are patent eligible due to the arguments regarding the §103 rejections; however, the Examiner asserts that subject matter eligibility and novelty/non-obviousness are two separate inquires, neither being a benchmark for the other. See Amdocs Ltd. v. Openet Telecom, Inc., 841 F.3d 1288, 1311 (Fed. Cir. 2016) (“Novelty is the question of whether the claimed invention is new. Inventiveness is the question of whether the claimed matter is invention at all, new or otherwise. The inventiveness inquiry of § 101 should therefore not be confused with the separate novelty inquiry of § 102 or the obviousness inquiry of § 103.”).
The Examiner notes that in order to be patent eligible under 35 U.S.C. 101, the claims must be directed towards a patent eligible concept, which, the instant claims are not directed. Contrary to Applicants’ assertion that the claims are not a mental process, the Examiner notes that methodologies to control artificial intelligence is a function that one of ordinary skill in the art have traditionally performed/provided (i.e. not actual artificial intelligence, but the general thought process behind how to control artificial intelligence). Next, the claims are not directed to a practical application of the concept. The claims do not result in improvements to the functioning of a computer or to any other technology or technical field. They do not effect a particular treatment for a disease. They are not applied with or by a particular machine. They do not effect a transformation or reduction of a particular article to a different state or thing. And they are not applied in some other meaningful way beyond generally linking the use of the judicial exception (i.e., using rewarding functions to improve recommendations) to a particular technological environment (i.e., artificial intelligence). Here, again as noted in the previous rejection, mere instructions to apply an exception using a generic computer component cannot provide an inventive concept - MPEP 2016.05(f). The claims recitation of the “user interface” “one or more processors” and “database” only generally linking the use of the judicial exception to a particular technological environment or field of use – see MPEP 2106.05(h). The claim(s) is/are not patent eligible.
Applicant’s next argue that the claims are eligible as the claims cannot be “practically performed in the human mind;”(due to the recitation of artificial intelligence) however the Examiner respectfully disagrees. As an intimal note, the general theory behind artificial intelligence is to model a computer around the human brain, such that the computer learns as our brains do. Secondly, while the specification may discuss sophisticated techniques and advanced functions such as artificial intelligence, it is the claims that are deemed eligible or ineligible under §101. Here, the claims are more of a generalized guideline of how to arrange a software model to implement the overarching abstract idea. Thirdly, this argument appears to be whether or not the use of computer or computing components for increased speed and efficiency makes the claims eligible; however the Examiner respectfully disagrees. Nor, in addressing the second step of Alice, does claiming the improved speed or efficiency inherent with applying the abstract idea on a computer provide a sufficient inventive concept. See Bancorp Servs., LLC v. Sun Life Assurance Co. of Can., 687 F.3d 1266, 1278 (Fed. Cir. 2012) (“[T]he fact that the required calculations could be performed more efficiently via a computer does not materially alter the patent eligibility of the claimed subject matter.”); CLS Bank, Int’l v. Alice Corp., 717 F.3d 1269, 1286 (Fed. Cir. 2013) (en banc) aff’d, 134 S. Ct. 2347 (2014) (“[S]imply appending generic computer functionality to lend speed or efficiency to the performance of an otherwise abstract concept does not meaningfully limit claim scope for purposes of patent eligibility.” (citations omitted)). The claim(s) is/are not patent eligible and the rejection not withdrawn.
Applicant’s arguments with respect to the prior art have been fully considered but are moot on grounds of new rejection, as necessitated by amendments.
In response to arguments in reference to any depending claims that have not been individually addressed, all rejections made towards these dependent claims are maintained due to a lack of reply by the Applicants in regards to distinctly and specifically pointing out the supposed errors in the Examiner's prior office action (37 CFR 1.111). The Examiner asserts that the Applicants only argue that the dependent claims should be allowable because the independent claims are unobvious and patentable over the prior art.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1, 12, and 19 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for pre-AIA the inventor(s), at the time the application was filed, had possession of the claimed invention. The claims recite the newly amended limitation “that continuously discounts older rewards using a forgetfulness hyperparameter without resetting the weighted average reward” however there is no discussion, throughout the entirety of the specification and drawings, how the hyperparameter is a forgetfulness hyperparameter which can discount older rewards without resetting the weighted average reward. Paragraph [0051] states “Model parameters refer to parameters determined through learning and include a weight value of synaptic connection and deflection of neurons. A hyperparameter means a parameter to be set in the machine learning algorithm before learning, and includes a learning rate, a repetition number, a mini batch size, and an initialization function.” While a “hyperparameter” is mentioned, there is no mention or discussion as to how a hyperparameter is able to be a forgetfulness hyperparameter that is able to continuously discount older reward without resetting the weighted average reward (the words “forget” “forgetful” and “discount” are not present in the specification whatsoever). As such, the Examiner asserts this as evidence that the newly amended claims are new matter.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 is/are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claims are directed to a process (an act, or series of acts or steps), a machine (a concrete thing, consisting of parts, or of certain devices and combination of devices), and a manufacture (an article produced from raw or prepared materials by giving these materials new forms, qualities, properties, or combinations, whether by hand labor or by machinery). Thus, each of the claims falls within one of the four statutory categories (Step 1). The claims recite a method (process) and a device (apparatus), however, the claim(s) recite(s) outputting recommendations based upon the weighted average of a reward within a group of items to be recommended which is an abstract idea of a mental process as well as the abstract idea of performing computations in accordance with a mathematical formula on that data.
The limitations of ...
In claims 1 and 12: “during an exploration stage, recommending each item within the group of items at least one time for outputting a first plurality of recommendations and receiving rewards for the first plurality of recommendations, storing a weighted average reward for each item within the group of items in the memory, and storing a number of times each item within the group has been recommended in the memory; and during an exploration-exploitation stage subsequent to the exploration stage, determining a maximum expected reward value for each item within the group of items based on the weighted average reward received for each item within the group of items that continuously discounts older rewards using a forgetfulness hyperparameter without resetting the weighted average reward, an upper confidence bound value for each item within the group of items and a user preference value for each item within the group of items that prioritizes or entirely excludes the item from consideration,, and outputting a recommendation identifying an item having a highest maximum expected reward among the group of items,”
In claim 19 “in response to receiving a first recommendation request, recommending a first item within the group of items for outputting a first recommendation; receiving a first reward for first recommendation; storing weighted average reward for the first item in the memory, the weighted average reward being based on the first reward; storing a number of times the first item has been recommended in the memory; and in response to receiving a second recommendation request, determining a maximum expected reward value for the first item based on a sum of the weighted average reward that continuously discounts older rewards using a forgetfulness hyperparameter without resetting the weighted average reward, an upper confidence bound value for the first item and a user preference value for the first item that prioritizes or entirely excludes the item from consideration, and outputting a second recommendation based on a highest maximum expected reward for the first item”
...as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process—concepts performed in the human mind (including an observation, evaluation, judgment, opinion) or mathematical concepts—mathematical relationships, mathematical formulas or equations, mathematical calculations but for the recitation of generic computer components (Step 2A Prong 1). That is, other than reciting “...an artificial intelligence (AI) device...via a processor,” (or “An artificial intelligence (AI) device for providing recommendations, the Al device comprising: a memory configured to store item information, user preferences and reward feedback information; and a controller configured to:” in claim 12) nothing in the claim element precludes the step from practically being performed in the mind or from the mathematical concept grouping. For example, but for the “...an artificial intelligence (AI) device...via a processor,” (or “An artificial intelligence (AI) device for providing recommendations, the Al device comprising: a memory configured to store item information, user preferences and reward feedback information; and a controller configured to:” in claim 12) language, “determining” in the context of this claim encompasses the user manually mapping, computing centroids, distances and comparing to the threshold which is a mental process/judgement of how or when to provide a recommendation of an item based upon a reward system for the process (i.e. a learning process) or the mathematical concept of using weighted averages to track the feedback with respect to the recommendations to reinforce the recommendations. However, if possible, the Examiner should consider the limitations together as a single abstract idea rather than as a plurality of separate abstract ideas to be analyzed individually. “For example, in a claim that includes a series of steps that recite mental steps as well as a mathematical calculation, an examiner should identify the claim as reciting both a mental process and a mathematical concept for Step 2A, Prong One to make the analysis clear on the record.” MPEP 2106.04, subsection II.B. Under such circumstances, however, the Supreme Court has treated such claims in the same manner as claims reciting a single judicial exception. Id. (discussing Bilski v. Kappos, 561 U.S. 593 (2010)). Here, the limitations are considered together as a single abstract idea for further analysis. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitations as a mathematical concept, while some of the limitations may be performed in the mind after certain limitations are performed, but for the recitation of generic computer components, then it falls within the grouping of abstract ideas. (Step 2A, Prong One: YES). Accordingly, the claim(s) recite(s) an abstract idea.
This judicial exception is not integrated into a practical application (Step 2A Prong Two). The “storing...” “obtaining...” and “receiving...” steps are all insignificant extrasolution data gathering activities. In particular, the claims only recite one additional element – using a processor or controller to perform the steps. The processor or controller in the steps is recited at a high-level of generality such that it amounts no more than mere instructions to apply the exception using a generic computer component. Specifically the claims amount to nothing more than an instruction to apply the abstract idea using a generic computer or invoking computers as tools by adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.04(d)(I) discussing MPEP 2106.05(f). Accordingly, the combination of these additional elements does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea, even when considered as a whole (Step 2A Prong Two: NO).
The claim does not include a combination of additional elements that are sufficient to amount to significantly more than the judicial exception (Step 2B). As discussed above with respect to integration of the abstract idea into a practical application (Step 2A Prong 2), the combination of additional elements of using a processor or controller to perform the steps amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Reevaluating here in step 2B, the “storing...” “obtaining...” and “receiving...” step(s) which are insignificant extrasolution activities are also determined to be well-understood, routine and conventional activity in the field. The Symantec, TLI, and OIP Techs court decisions in MPEP 2106.05(d)(II) indicate that the mere receipt or transmission of data over a network is well-understood, routine, and conventional function when it is claimed in a merely generic manner (as is here). Therefore, when considering the additional elements alone, and in combination, there is no inventive concept in the claim. As such, the claim(s) is/are not patent eligible, even when considered as a whole (Step 2B: NO).
Claims 2-6, 8, 13-18 and 20 recite(s) the additional limitation(s) further including additional mathematical concepts, which is still directed towards the abstract idea previously identified and is not an inventive concept that meaningfully limits the abstract idea. Again, as discussed with respect to claims 1, 12, and 19, the claims are simply limitations which are no more than mere instructions to apply the exception using a computer or with computing components. Accordingly, the additional element(s) does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. Even when considered as a whole, the claims do not integrate the judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Claim 7 recite(s) the additional limitation(s) further including displaying an output which is not an inventive concept that meaningfully limits the abstract idea. Again, as discussed with respect to claims 1, 12, and 19, the claims are simply limitations which are no more than mere instructions to apply the exception using a computer or with computing components. Accordingly, the additional element(s) does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. Even when considered as a whole, the claims do not integrate the judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Claims 9-11 recite(s) the additional limitation(s) further limiting the user preference and the group of items which is still directed towards the abstract idea previously identified and is not an inventive concept that meaningfully limits the abstract idea. Again, as discussed with respect to claims 1, 12, and 19, the claims are simply limitations which are no more than mere instructions to apply the exception using a computer or with computing components. Accordingly, the additional element(s) does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. Even when considered as a whole, the claims do not integrate the judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Claims 1-20 are therefore not eligible subject matter, even when considered as a whole.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Cao et al. (US PG Pub. 2020/0074504) further in view of Upadhyay et al. (US PG Pub. 2021/0065897) and Sohane (US PG Pub. 2023/0230141).
As per claims 1 and 12, Cao discloses a method for controlling a...device, and device for providing recommendations, the Al device comprising: a memory configured to store item information, user preferences and reward feedback information; and a controller configured to: the method comprising (method, Cao ¶3; computing device, memory, processor, ¶42):
storing, in a memory of the device, item information, user preferences and reward feedback information (The digital medium environment 100 may be implemented in a variety of ways to leverage recommendations 124 generated by the recommendation system 104. The digital medium environment 100, for instance, may utilize the recommendation system 104 as an information filtering system that seeks to predict user preferences with respect to an item, e.g., as a rating or “reward.” The rating then serves as a basis to form a recommendation for an item chosen from a plurality of items. As such, recommendations generated by the digital media environment 100 may take a variety of forms, such as to recommend items of digital content (e.g., digital movies, digital articles), news articles (e.g., in a digital newsfeed), search, social media (e.g., friends in a social network service), locations (e.g., restaurants, events, shows), as part of online dating, financial services, social media websites, humor, and so forth, Cao ¶35; user interaction data, ¶8) (Examiner notes the ability to receive and ascertain user interaction data as the ability to receive and store user preferences and feedback with respect to content);
obtaining, via a processor in the device, item information for a group of items (A “multi-armed bandit” refers to a technique is configured to address allocation of a fixed limited set of resources (e.g., digital content) through use of recommendations in a way to maximize the reward. In this technique, the recommendation system is modeled to make a decision regarding which option in a collection of options to recommend, how many times to recommend each option, in which order to recommend the options, and whether to continue with a current option or to try a different option. Each option is modeled as providing a random reward based on a probability distribution specific to that machine, referred to as a reward distribution. Thus, the objective of the recommendation system is to maximize a sum of rewards earned through a series of recommendations by identifying which options (i.e., items of digital content) are to be subject of the user interaction, Cao ¶28; digital content, ¶29; for recommendations, ¶30);
receiving, via the processor, at least one user preference (user interaction data, Cao ¶8; processing user interactions, ¶37);
during an exploration stage, recommending each item within the group of items at least one time for outputting a first plurality of recommendations and receiving rewards for the first plurality of recommendations, storing a weighted average reward for each item within the group of items in the memory, and storing a number of times each item within the group has been recommended in the memory (An Implementation Example is described in a respective section regarding a technique referred to as Monitored-Upper Confidence Bound (M-UCB), in which, a change point detection component is incorporated into an upper confidence bound technique. M-UCB, for instance, may be leveraged by the recommendation system to monitor a mean of the reward distribution for a currently selected item of digital content, i.e., “arm.” Once a change is detected, the M-UCB technique is configured to reset the statistical model to adapt to the change and generate a recommendation for exploration to learn a new optimal arm. M-UCB is shown in the following to be nearly optimal for the considered scenario, in the sense that an O(√{square root over (MKT log T)}) regret bound may be established where T is the number of time steps, K is the number of arms, and M is the number of stationary segments. In a real-world scenario, M-UCB has been observed to achieve an approximate regret reduction of fifty percent with respect to the best performing state-of-the-art conventional techniques that are currently available. A variety of other examples are also contemplated, Cao ¶26; exploration techniques, ¶3, ¶8, and ¶23) (Examiner interprets the mean reward as the weighted average reward); and
during an exploration-exploitation stage subsequent to the exploration stage, determining a maximum expected reward value for each item within the group of items based on the weighted average reward received for each item within the group of items, an upper confidence bound value for each item within the group of items and a user preference value for each item within the group of items that prioritizes or entirely excludes the item from consideration, and outputting a recommendation identifying an item having a highest maximum expected reward among the group of items (In exploitation as implemented by the exploitation module 210, the recommendation 124 of digital content 118 is generated to maximize effectiveness in achieving the desired user interaction, e.g., based on the reward distribution of the statistical model 128 for respective items of digital content 118. An example of an exploitation technique includes an upper confidence bound (UCB) technique, Thus, in exploitation as implemented by the exploitation module 210 a digital content recommendation 124 is generated to maximize a reward at an individual time step. Exploration as implemented by the exploration module 208 is used to increase knowledge and accuracy of the reward distributions 206, e.g., a confidence interval for each distribution. This increased knowledge and accuracy may then be used to further exploit collection of rewards for future time steps and thus maximize the reward for a series of times steps “T.” Cao ¶61-¶62; exploitation techniques, ¶23).
While Cao discloses as shown above, Cao does not expressly disclose an artificial intelligence (AI) device.
However, Upadhyay teaches an artificial intelligence (AI) device (AI device, Upadhyay ¶42).
Both the Upadhyay and the Cao references are analogous in that both are directed towards/concerned with rewarded learning. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to use Upadhyay’s ability to utilize rewarded learning with artificial intelligence in Cao’s system to improve the system and method with reasonable expectation that this would result in an improved recommendation system with rewarded learning that is also applicable to artificial intelligence.
The motivation being that in many machine learning problems, data is naturally collected over time and systems are required to make predictions (take an action) before they are allowed to observe any response from the environment. Oftentimes, there is no response available, e.g. there is a missing label and/or the environment is not responding to the system's action. Furthermore, in many practical systems, one can only hope to observe feedback indicating whether a given action is “good” or “bad” (1 or 0 reward); the latter case obscuring the true label for learning (Upadhyay ¶2).
The combination of Cao and Upadhyay do not expressly disclose that continuously discounts older rewards using a forgetfulness hyperparameter without resetting the weighted average reward.
However, Sohane teaches that continuously discounts older rewards using a forgetfulness hyperparameter without resetting the weighted average reward (forgetting factor, Sohane ¶6 and ¶40-¶41).
The Sohane, Upadhyay, and the Cao references are analogous in that both are directed towards/concerned with rewarded learning. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to use Sohane’s ability to include a forgettinfg factor in Upadhyay’s and Cao’s system to improve the system and method with reasonable expectation that this would result in an improved recommendation system with rewarded learning that is also applicable to automated machine learning with respect to product recommendations for a user.
The motivation being that collaborative filtering methods face challenges associated with inability to adapt to new users. These methods hence cannot be used in a session-based environment. Most collaborative algorithms require computing recommendations in some offline batch processes, which cannot be adapted with user interactions. Also, the optimization based on metrices such as purchase, add to cart and view is not very straightforward for collaborative filtering methods. Deploying a collaborative filtering-based solution hence requires careful fine tuning and data analysis (Sohane ¶3).
As per claims 2 and 13, Cao, Upadhyay, and Sohane disclose as shown above with respect to claims 1 and 12. Cao further discloses wherein the maximum expected reward value for a specific item within the group of items is a sum of the weighted average reward received for the specific item, the upper confidence bound value for the specific item and the user preference value assigned to the specific item (To do so, the recommendation system employs exploration techniques in which recommendations are generated to gather data regarding the reward distributions for respective items of digital content. The recommendation system also employs exploitation techniques to maximize a reward at a respective time step for a recommendation. In this way, the recommendation system may “learn” about the reward distributions of respective items of digital content through exploration and maximize the amount of rewards collected at the individual time steps through exploitation. Both of these techniques are thus employed to maximize an overall amount of reward over the series of time steps through learning about the rewards and collect the rewards based on this learned knowledge as described by respective reward distributions, Cao ¶23).
As per claims 3 and 14, Cao, Upadhyay, and Sohane disclose as shown above with respect to claims 1 and 12. Cao further discloses wherein a sum of the weighted average reward and the upper confidence bound value represents a highest possible expected mean value of a reward distribution for a corresponding item within the group of items (Detection of this change point may then be used by the recommendation system to reset the statistical model 128 to address this change. This is also used to cause generation of a subsequent recommendation configured for exploration of reward distributions of the items of digital marketing content. In this way, the recommendation system 104 may detect a change and, based on this, generate recommendations to learn about effects of the change on reward distributions overall in order to maximize overall rewards over a series of time steps. An implementation example is described in a respective section regarding an example of such a technique, which is referred to as Monitored-Upper Confidence Bound (M-UCB), Cao ¶55).
As per claims 4 and 15, Cao, Upadhyay, and Sohane disclose as shown above with respect to claims 1 and 12. Cao further discloses wherein the exploration stage is carried out K times to generate the first plurality of recommendations, K being equal to a number of items within the group of items (number of arms, k arms, as the plurality of options, Cao ¶4).
As per claims 5 and 16, Cao, Upadhyay, and Sohane disclose as shown above with respect to claims 1 and 12. While Upadhyay does disclose the ability to normalize value ranges to 1 and 0 (Upadhyay ¶2), the combination of Cao and Upadhyay do not expressly disclose wherein the user preference value is a rating received from a user that is normalized to be a value in a range of -1 to 0.
However, before the effective filing date of the invention, one of ordinary skill in the art would have recognized the numerical ranges for a user preference as a scale of -1 to 0 is only a slight variation from 1 to 0 and thus would have found it obvious to modify the teachings of Cao and Upadhyay to include such a range. The claim would have been obvious because the design incentives or market forces provided a reason to make an adaptation, and the invention resulted from application of the prior knowledge in a predictable manner.
As per claims 6 and 17, Cao, Upadhyay, and Sohane disclose as shown above with respect to claims 1 and 12. Cao further discloses updating a weighted sum for each item within the group of items, the weighted sum being based on a number of times a corresponding item has been recommended, wherein the upper confidence bound value becomes larger as the weighted sum becomes smaller (Detection of the change point may be performed in a variety of ways by the recommendation system. In one example, the change point is detected by the recommendation system based at on a determination that a reward distribution for the item of digital content that is subject to a recommendation departs from a reward distribution of the statistical model for the item of digital content by at least a threshold amount. The change point, for instance, may be detected by the recommendation system by monitoring a mean of a respective reward distribution for an item of digital content that corresponds to the digital content recommendation. The recommendation system, for instance, may obtain data that describes that user interaction resulting from exposure to a recommendation may cause a change to a reward distribution for the item in the statistical model by greater than a threshold amount. Such a change is thus used by the recommendation system to determine that the statistical model is no longer accurate, and in response regenerate the model. In this way, the recommendation system may respond to changes in real world scenarios that is not possible using conventional techniques, Cao ¶25).
As per claim 7, Cao, Upadhyay, and Sohane disclose as shown above with respect to claim 1. Cao further discloses displaying, on a display of the AI device, the recommendation identifying the item having the highest maximum expected reward among the group of items (recommendation result, Cao ¶30; output to client device, ¶37).
As per claims 8 and 18, Cao, Upadhyay, and Sohane disclose as shown above with respect to claims 1 and 12. Cao further discloses wherein the upper confidence bound value is based on an iterative logarithm function (logarithmic factors for upper bounds, Cao ¶92-¶94).
As per claim 9, Cao, Upadhyay, and Sohane disclose as shown above with respect to claim 1. Upadhyay further teaches wherein the receiving the at least one user preference includes: receiving a chat message or a text message from the user including a rating or a score (feedback, to dialog agent, ¶69) (Examiner interprets the feedback from the user as including a message or chat with a rating or score).
As per claim 10, Cao, Upadhyay, and Sohane disclose as shown above with respect to claim 1. Cao further discloses wherein the group of items is a group of movies, a group of advertisements, a group of restaurants, a group of available routes to a destination, or a group of videos (The digital medium environment 100 may be implemented in a variety of ways to leverage recommendations 124 generated by the recommendation system 104. The digital medium environment 100, for instance, may utilize the recommendation system 104 as an information filtering system that seeks to predict user preferences with respect to an item, e.g., as a rating or “reward.” The rating then serves as a basis to form a recommendation for an item chosen from a plurality of items. As such, recommendations generated by the digital media environment 100 may take a variety of forms, such as to recommend items of digital content (e.g., digital movies, digital articles), news articles (e.g., in a digital newsfeed), search, social media (e.g., friends in a social network service), locations (e.g., restaurants, events, shows), as part of online dating, financial services, social media websites, humor, and so forth, Cao ¶35).
As per claim 11, Cao, Upadhyay, and Sohane disclose as shown above with respect to claim 1. Cao further discloses wherein the AI device includes at least one of a smart television, a mobile phone, and a robot (The digital medium environment 100 may be implemented in a variety of ways to leverage recommendations 124 generated by the recommendation system 104. The digital medium environment 100, for instance, may utilize the recommendation system 104 as an information filtering system that seeks to predict user preferences with respect to an item, e.g., as a rating or “reward.” The rating then serves as a basis to form a recommendation for an item chosen from a plurality of items. As such, recommendations generated by the digital media environment 100 may take a variety of forms, such as to recommend items of digital content (e.g., digital movies, digital articles), news articles (e.g., in a digital newsfeed), search, social media (e.g., friends in a social network service), locations (e.g., restaurants, events, shows), as part of online dating, financial services, social media websites, humor, and so forth, Cao ¶45).
As per claim 19, Cao discloses method for controlling a...device, the method comprising (method, Cao ¶3; computing device, memory, processor, ¶42):
storing, in a memory of the device, item information, user preferences and reward feedback information (The digital medium environment 100 may be implemented in a variety of ways to leverage recommendations 124 generated by the recommendation system 104. The digital medium environment 100, for instance, may utilize the recommendation system 104 as an information filtering system that seeks to predict user preferences with respect to an item, e.g., as a rating or “reward.” The rating then serves as a basis to form a recommendation for an item chosen from a plurality of items. As such, recommendations generated by the digital media environment 100 may take a variety of forms, such as to recommend items of digital content (e.g., digital movies, digital articles), news articles (e.g., in a digital newsfeed), search, social media (e.g., friends in a social network service), locations (e.g., restaurants, events, shows), as part of online dating, financial services, social media websites, humor, and so forth, Cao ¶35; user interaction data, ¶8) (Examiner notes the ability to receive and ascertain user interaction data as the ability to receive and store user preferences and feedback with respect to content);
obtaining, via a processor in the device, item information for a group of items (A “multi-armed bandit” refers to a technique is configured to address allocation of a fixed limited set of resources (e.g., digital content) through use of recommendations in a way to maximize the reward. In this technique, the recommendation system is modeled to make a decision regarding which option in a collection of options to recommend, how many times to recommend each option, in which order to recommend the options, and whether to continue with a current option or to try a different option. Each option is modeled as providing a random reward based on a probability distribution specific to that machine, referred to as a reward distribution. Thus, the objective of the recommendation system is to maximize a sum of rewards earned through a series of recommendations by identifying which options (i.e., items of digital content) are to be subject of the user interaction, Cao ¶28; digital content, ¶29; for recommendations, ¶30);
receiving, via the processor, at least one user preference (user interaction data, Cao ¶8; processing user interactions, ¶37);
in response to receiving a first recommendation request, recommending a first item within the group of items for outputting a first recommendation (first step, first recommendation, Cao ¶51; An Implementation Example is described in a respective section regarding a technique referred to as Monitored-Upper Confidence Bound (M-UCB), in which, a change point detection component is incorporated into an upper confidence bound technique. M-UCB, for instance, may be leveraged by the recommendation system to monitor a mean of the reward distribution for a currently selected item of digital content, i.e., “arm.” Once a change is detected, the M-UCB technique is configured to reset the statistical model to adapt to the change and generate a recommendation for exploration to learn a new optimal arm. M-UCB is shown in the following to be nearly optimal for the considered scenario, in the sense that an O(√{square root over (MKT log T)}) regret bound may be established where T is the number of time steps, K is the number of arms, and M is the number of stationary segments. In a real-world scenario, M-UCB has been observed to achieve an approximate regret reduction of fifty percent with respect to the best performing state-of-the-art conventional techniques that are currently available. A variety of other examples are also contemplated, ¶26; exploration techniques, ¶3, ¶8, and ¶23) (Examiner interprets the mean reward as the weighted average reward);
receiving a first reward for first recommendation (reward received, Cao ¶48; update with reward, ¶51-¶53);
storing weighted average reward for the first item in the memory, the weighted average reward being based on the first reward (mean of the reward distribution, Cao ¶10 and ¶26);
storing a number of times the first item has been recommended in the memory (number of arms, k arms, as the plurality of options, Cao ¶4); and
in response to receiving a second recommendation request, determining a maximum expected reward value for the first item based on a sum of the weighted average reward, an upper confidence bound value for the first item that prioritizes or entirely excludes the item from consideration and a user preference value for the first item, and outputting a second recommendation based on a highest maximum expected reward for the first item (In exploitation as implemented by the exploitation module 210, the recommendation 124 of digital content 118 is generated to maximize effectiveness in achieving the desired user interaction, e.g., based on the reward distribution of the statistical model 128 for respective items of digital content 118. An example of an exploitation technique includes an upper confidence bound (UCB) technique, Thus, in exploitation as implemented by the exploitation module 210 a digital content recommendation 124 is generated to maximize a reward at an individual time step. Exploration as implemented by the exploration module 208 is used to increase knowledge and accuracy of the reward distributions 206, e.g., a confidence interval for each distribution. This increased knowledge and accuracy may then be used to further exploit collection of rewards for future time steps and thus maximize the reward for a series of times steps “T.” Cao ¶61-¶62; exploitation techniques, ¶23).
While Cao discloses as shown above, Cao does not expressly disclose an artificial intelligence (AI) device.
However, Upadhyay teaches an artificial intelligence (AI) device (AI device, Upadhyay ¶42).
Both the Upadhyay and the Cao references are analogous in that both are directed towards/concerned with rewarded learning. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to use Upadhyay’s ability to utilize rewarded learning with artificial intelligence in Cao’s system to improve the system and method with reasonable expectation that this would result in an improved recommendation system with rewarded learning that is also applicable to artificial intelligence.
The motivation being that in many machine learning problems, data is naturally collected over time and systems are required to make predictions (take an action) before they are allowed to observe any response from the environment. Oftentimes, there is no response available, e.g. there is a missing label and/or the environment is not responding to the system's action. Furthermore, in many practical systems, one can only hope to observe feedback indicating whether a given action is “good” or “bad” (1 or 0 reward); the latter case obscuring the true label for learning (Upadhyay ¶2).
The combination of Cao and Upadhyay do not expressly disclose that continuously discounts older rewards using a forgetfulness hyperparameter without resetting the weighted average reward.
However, Sohane teaches that continuously discounts older rewards using a forgetfulness hyperparameter without resetting the weighted average reward (forgetting factor, Sohane ¶6 and ¶40-¶41).
The Sohane, Upadhyay, and the Cao references are analogous in that both are directed towards/concerned with rewarded learning. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to use Sohane’s ability to include a forgettinfg factor in Upadhyay’s and Cao’s system to improve the system and method with reasonable expectation that this would result in an improved recommendation system with rewarded learning that is also applicable to automated machine learning with respect to product recommendations for a user.
The motivation being that collaborative filtering methods face challenges associated with inability to adapt to new users. These methods hence cannot be used in a session-based environment. Most collaborative algorithms require computing recommendations in some offline batch processes, which cannot be adapted with user interactions. Also, the optimization based on metrices such as purchase, add to cart and view is not very straightforward for collaborative filtering methods. Deploying a collaborative filtering-based solution hence requires careful fine tuning and data analysis (Sohane ¶3).
As per claim 20, Cao, Upadhyay, and Sohane disclose as shown above with respect to claim 19. Cao further discloses wherein the determining the maximum expected reward value for the first item includes: taking a sum of the weighted average reward for the first item, the upper confidence bound value for the first item and the user preference value for the first item (To do so, the recommendation system employs exploration techniques in which recommendations are generated to gather data regarding the reward distributions for respective items of digital content. The recommendation system also employs exploitation techniques to maximize a reward at a respective time step for a recommendation. In this way, the recommendation system may “learn” about the reward distributions of respective items of digital content through exploration and maximize the amount of rewards collected at the individual time steps through exploitation. Both of these techniques are thus employed to maximize an overall amount of reward over the series of time steps through learning about the rewards and collect the rewards based on this learned knowledge as described by respective reward distributions, Cao ¶23).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the Examiner should be directed to ANDREW B WHITAKER whose telephone number is (571)270-7563. The examiner can normally be reached on M-F, 8am-5pm, EST.
If attempts to reach the examiner by telephone are unsuccessful, the Examiner’s supervisor, Lynda Jasmin can be reached on (571) 272-6782. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center. Status information for unpublished applications is available through Patent Center for authorized users only. Should you have questions about access to Patent Center, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-
automated- interview-request-air-form
/ANDREW B WHITAKER/Primary Examiner, Art Unit 3629
Prosecution Insights