Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Continued Examination Under 37 CFR 1.114
2. A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 2/12/2026 has been entered. Claims 1, 17, and 18 have been amended. Claim 19 has been added. Claims 1-19 remain pending in the application.
Response to Amendments
3. Applicant’s amendments to claim 1 have been fully considered and are persuasive. The objection to claims 6, 8-12 is respectfully withdrawn.
Response to Arguments
Applicant argues that claim 1 does not recite matter that falls within the enumerated grouping of abstract ideas should not be treated as reciting abstract ideas. However, the Examiner respectfully disagrees and notes that claim1 is not treated as abstract merely because it uses a machine learning system. Rather, claim 1 recites specific limitations that fall within the mathematical concept grouping of abstract ideas. In particular, claim 1 recites obtaining a value function by evaluating state information in association with environment information, performing reinforcement learning on the value function, selecting environment information when the value function is highest, training a correlation between target behavior and environment information, and calculating an achievement difficulty level based on the value function.
Applicant argues that claim 1 is directed to an improvement to computer functionality itself. However, the Examiner respectfully disagrees and notes that claim 1 is directed to applying mathematical value function/reinforcement learning using generic computer components, rather than to an improvement in computer functionality itself. Furthermore, the additional elements include the machine learning system, circuitry, acquiring state information, environment information, target behavior information, and machine learning classifier. These elements do not add significantly more because they merely provide the generic computer/ML environment and input data for applying the mathematical concepts. The acquisition of state information is mere data gathering. The use of environment information is contextual input data. The use of circuitry and a machine learning classifier amounts to applying the mathematical value function/reinforcement learning/correlation analysis using generic computer components.
Applicant argues that He and Lee do not disclose amended claim 1. However, the argument is moot since this is a newly presented limitation, thus changes the scope of the claim. However, newly found references, Arar and Aghdaie, are applied.
Claim Rejections - 35 USC § 101
4. 35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-19 are rejected under 35 U.S.C. 101 because the claimed invention is directed to the abstract idea without significantly more.
Step 1, the claims are directed to a process and manufacture.
Step 2A Prong 1, Claims 1, 17, and 18 recite, in part
obtain a value function by evaluating the state information in association with environment information (Mathematical concepts, mathematical scoring/valuation relationship).
perform reinforcement learning on the value function to select the environment information associated with the state information when the value function is highest according to a trained correlation between a target behavior indicated by the state information and the environment information (Mathematical concepts, mathematical relationships).
repeatedly performs the reinforcement learning to train the correlation between the environment information and the value function obtained by evaluating the state information associated with the environment information, and wherein an achievement difficulty level for the target behavior is calculated based on the value function obtained by evaluating the state information (Mathematical concepts, mathematical calculations/operations).
Step 2A Prong 2, this judicial exception is not integrated into a practical application.
The additional elements:
the circuitry and processor (mere instructions to apply the exception using a generic computer component).
acquire information including at least state information regarding a behavior of a person (mere data gathering and recited at a high level of generality, and thus are insignificant extra-solution activity).
an environment around the person when acquiring the state information (mere data gathering and recited at a high level of generality, and thus are insignificant extra-solution activity).
wherein the environment information is selected by a machine learning classifier of the circuitry (mere instructions to apply the exception using a generic computer component).
Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception, either alone or in combination.
The additional elements:
the circuitry and processor (mere instructions to apply the exception using a generic computer component).
acquire information including at least state information regarding a behavior of a person (mere data gathering and recited at a high level of generality, and thus are insignificant extra-solution activity).
an environment around the person when acquiring the state information (mere data gathering and recited at a high level of generality, and thus are insignificant extra-solution activity).
wherein the environment information is selected by a machine learning classifier of the circuitry (mere instructions to apply the exception using a generic computer component).
Claims 2-16 and 19 provide further limitations to the abstract idea (Mathematical concepts and/or Mental processes) as rejected in claims 1, 17, 18, however, they do not disclose any additional elements that would amount to a practical application or significantly more than an abstract idea (data gathering/insignificant extra-solution activity and/or generic computer component).
Claim Rejections – 35 USC § 103
5. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
6. Claims 1-4, 13-15, and 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over He (U.S. Patent Application Pub. No. US 20210049632 A1) in view of Arar et al. (U.S. Patent Application Pub. No. US 20180113586 A1), and further in view of Aghdaie et al. (U.S. Patent Application Pub. No. US 20170259177 A1).
Claim 1: He teaches a machine learning system (i.e. a deep reinforcement learning apparatus for referral marketing; para. [0016]), comprising:
circuitry configured to (i.e. a computing device is provided, including a memory and a processor, where the memory stores executable code, and the processor executes the executable code to implement the method; para. [0017])
acquire information including at least state information regarding a behavior of a person (i.e. a state acquisition unit 71, configured to obtain state information of an execution environment, where the state information includes at least user information of a current user; para. [0037-0041, 0075]),
obtain a value function by evaluating the state information in association with environment information regarding an environment around the person when acquiring the state information (i.e. In the scenario of deep reinforcement learning for referral marketing in some previous embodiments, the return value can be defined as a weighted sum that includes the current reward score and at least one future reward score, where each reward score is a feedback of the execution environment for the current marketing behavior; para. [0040, 0041, 0044, 0062, 0063]), and
perform reinforcement learning on the value function (i.e. FIG. 2 is a schematic diagram illustrating a deep reinforcement learning system. Generally, the deep reinforcement learning system includes an agent and an execution environment. The agent continuously learns and optimizes its strategies through interaction and feedback with the execution environment; para. [0034, 0035, 0067]) to select the environment information associated with the state information (i.e. the agent determines the marketing channel, the marketing content, and the marketing time period that are applicable to the current environment state based on the learned and trained marketing strategy, and determines, based on a combination of these three factors, the next marketing behavior to be taken; para. [0054]) when the value function is highest (i.e. For the previous definition of the return value, the marketing strategy is optimized by solving the Bellman equation. The optimization goal is to maximize the return value. For example, the process of solving the Bellman equation can include value iteration, strategy iteration, Q-learning, Sarsa, etc; para. [0067, 0068]) according to a trained correlation between a target behavior indicated by the state information and the environment information (i.e. the current reward score is determined based on the user's response to the marketing behavior, for example, whether the user receives the marketing information (i.e., whether the marketing information reaches the user), whether the user taps the marketing information, whether the user signs a contract … multiple factors involved in marketing are comprehensively considered during decision making, so as to fully learn, train, and optimize the marketing strategy; para. [0056, 0061]),
wherein the environment information by a machine learning of the circuitry that repeatedly performs the reinforcement learning to train the correlation between the environment information and the value function obtained by evaluating the state information associated with the environment information (i.e. The agent determines, based on the obtained reward score, whether the previous behavior is correct and whether the strategy needs to be adjusted, and then updates its strategy. By repeatedly observing the state, determining the behavior, and receiving feedback, the agent can continuously update the strategy … The agent can then adjust and update its marketing strategy based on the reward score; para. [0034, 0035, 0057]).
He does not explicitly teach wherein the environment information is selected by a machine learning classifier, wherein an achievement difficulty level for the target behavior is calculated based on the state information.
However, Arar teaches wherein the environment information is selected by a machine learning classifier (i.e. classifier may be trained to identify the occurrence of the event. For example, the classifier may be trained with the historical contextual data and associated event identifier. The classifier may then analyze input real-time contextual data in order to identify the occurrence of the event in response to detecting data associated with the event identifier; para. [0059, 0067, 0080]) of the circuitry (i.e. The processor, e.g., processing circuit(s); para. [0055]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of He to include the feature of Arar. One would have been motivated to make this modification because it provides the ability to recognize the user’s current environment before selecting the marketing behavior, thereby improving the relevance and accuracy of the reinforcement learning decision.
However, Aghdaie teaches wherein an achievement difficulty level for the target behavior is calculated based on the state information (i.e. based at least in part on the set of input data, the system can determine a retention probability associated with the probability that the user ceases to play the video game. Moreover, based at least in part on the retention probability for the user, the system can identify an adjustment value to a variable in the video game. The variable may be associated with a level of difficulty of the video game. In addition, the system may modify execution of the video game based at least in part on the adjustment value; para. [0008, 0082]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of He and Arar to include the feature of Aghdaie. One would have been motivated to make this modification because it provides an interpretable measure of how difficult it is for a use to achieve the target behavior.
Claim 2: He, Arar, and Aghdaie teach the machine learning system according to claim 1. He further teaches wherein the circuitry is further configured to calculate a reward based on a difference between the state information and target state information regarding the target behavior, and calculate the value function based on the reward, the environment information, and the state information (i.e. the current reward score is determined based on the user's response to the marketing behavior, for example, whether the user receives the marketing information (i.e., whether the marketing information reaches the user), whether the user taps the marketing information, whether the user signs a contract, and whether the user sets recommended content in the marketing information as a homepage or as a preferred option, and so on. As such, the reward score reflects multiple result goals to be optimized; para. [0034, 0035, 0056, 0064-0066]).
Claim 3: He, Arar, and Aghdaie teach the machine learning system according to claim 1. He further teaches wherein the system holds target state-related information including a plurality of pieces of target behavior information (i.e. the current reward score is determined based on the user's response to the marketing behavior, for example, whether the user receives the marketing information (i.e., whether the marketing information reaches the user), whether the user taps the marketing information, whether the user signs a contract, and whether the user sets recommended content in the marketing information as a homepage or as a preferred option, and so on. As such, the reward score reflects multiple result goals to be optimized; para. [0056, 0068]).
Claim 4: He, Arar, and Aghdaie teach the machine learning system according to claim 3. He further teaches wherein the target state-related information includes at least one of time-specific target state information or stage-specific target state information (i.e. determining the marketing behavior further includes determining the marketing time period. It can be understood that, different users have different APP use habits, and these use habits can be reflected in that frequency of using the APP in different periods and duration for maintaining attention are different. The same user has different sensitivity and attention to the marketing information in different periods. Therefore, according to some implementations, the marketing time period is divided in terms of multiple dimensions. For example, in an example, in terms of time dimension for every days of a week, the marketing time period is divided into working days (such as Monday to Friday) and non-working days (such as Saturday and Sunday) based on working time. In an example, in terms of hours within a day, the marketing time period in a day is divided into working hours (e.g., 9 am to 6 pm) and non-working hours based on working hours. In another example, a day is divided into dining periods and other periods based on the average dining time; para. [0028, 0030, 0052, 0053, 0068]).
Claim 13: He, Arar, and Aghdaie teach the machine learning system according to claim 1. He does not explicitly teach wherein the circuitry is further configured to calculate the achievement difficulty level for the target behavior based on the acquired state information.
However, Aghdaie further teaches wherein the circuitry is further configured to calculate the achievement difficulty level for the target behavior based on the acquired state information (i.e. based at least in part on the set of input data, the system can determine a retention probability associated with the probability that the user ceases to play the video game. Moreover, based at least in part on the retention probability for the user, the system can identify an adjustment value to a variable in the video game. The variable may be associated with a level of difficulty of the video game. In addition, the system may modify execution of the video game based at least in part on the adjustment value; para. [0008, 0010, 0082]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of He and Arar to include the feature of Aghdaie. One would have been motivated to make this modification because it provides an interpretable measure of how difficult it is for a use to achieve the target behavior.
Claim 14: He, Arar, and Aghdaie teach the machine learning system according to claim 13. He further teaches an achievement rate indicating a degree to which the target behavior is prompted (i.e. the current reward score is determined based on the user's response to the marketing behavior, for example, whether the user receives the marketing information (i.e., whether the marketing information reaches the user), whether the user taps the marketing information, whether the user signs a contract, and whether the user sets recommended content in the marketing information as a homepage or as a preferred option, and so on. As such, the reward score reflects multiple result goals to be optimized; para. [0056, 0068]).
He does not explicitly teach wherein the achievement difficulty level includes an achievement rate.
However, Aghdaie further teaches wherein the achievement difficulty level includes an achievement rate indicating a degree to which the target behavior is prompted (i.e. based on the predicted churn rate for the user determined at block 304, the difficulty configuration system 132 selects a seed associated with a particular level of difficulty for a portion of the video game. Selecting the seed may include accessing mapping data at the mapping data repository 144. This mapping data may indicate a mapping between one or more churn rates and one or more difficulty levels for the video game 112. In some cases, the churn rate above a particular threshold level may be mapped to one or more seed values that may reduce the difficulty of the video game 112; para. [0052, 0080, 0082]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of He and Arar to include the feature of Aghdaie. One would have been motivated to make this modification because it provides an interpretable measure of how difficult it is for a use to achieve the target behavior.
Claim 15: He, Arar, and Aghdaie teach the machine learning system according to claim 13. He does not explicitly teach wherein the achievement difficulty level includes a standard achievement time indicating a standard time for which the target behavior is prompted.
However, Aghdaie further teaches wherein the achievement difficulty level includes a standard achievement time indicating a standard time for which the target behavior is prompted (i.e. istorical user information is fed into a machine learning system to generate a prediction model that predicts an expected duration of game play, such as for example, an expected churn rate, a retention rate, the length of time a user is expected to play the game, or an indication of the user's expected game play time relative to a historical set of users who have previously played the game. Before or during game play, the prediction model is applied to information about the user to predict the user's expected duration of game play. Based on the expected duration, the system may then utilize a mapping data repository to determine how to dynamically adjust the difficulty of the game, such as, for example, changing the values of one or more knobs to make portions of the game less difficult; para. [0029, 0041]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of He and Arar to include the feature of Aghdaie. One would have been motivated to make this modification because it provides an interpretable measure of how difficult it is for a use to achieve the target behavior.
Claim 17 is similar in scope to Claims 1 and is rejected under a similar rationale.
He teaches a non-transitory computer-readable storage medium having embodied thereon a method (i.e. A non-transitory, computer-readable medium storing one or more instructions executable by a computer system to perform operations; claim 10, para. [0085]), which when executed by a computer causes the computer to execute a machine learning method (i.e. a deep reinforcement learning apparatus for referral marketing; para. [0016]).
Claim 18 is similar in scope to Claims 1 and is rejected under a similar rationale.
He teaches a machine learning method (i.e. a deep reinforcement learning apparatus for referral marketing; para. [0016]), executed by at least one processor (i.e. a computing device is provided, including a memory and a processor, where the memory stores executable code, and the processor executes the executable code to implement the method; para. [0017]).
Claim 19: He, Arar, and Aghdaie teach the machine learning system according to claim 1. He further teaches wherein the circuitry is further configured to update the obtained value function based on a change in the state information associated with the environment information as the reinforcement learning is repeatedly performed (i.e. By repeatedly observing the state, determining the behavior, and receiving feedback, the agent can continuously update the strategy. The final goal is to be able to obtain a strategy through learning, so that an accumulation of obtained reward scores is maximized. This is a typical reinforcement learning process. In the process of learning and adjusting strategies, if the agent uses some deep learning algorithms including a neural network, such a system is referred to as a deep reinforcement learning system; para. [0034, 0038]).
7. Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over He, in view of Arar, Aghdaie, and further in view of Lee et al. (U.S. Patent Application Pub. No. US 20210031000 A1).
Claim 5: He, Arar, and Aghdaie teach the machine learning system according to claim 1. He does not explicitly teach wherein the environment information includes information regarding at least one of scents, lighting, temperature, humidity, video, or sound in the environment around the person.
However, Lee teaches wherein the environment information includes information regarding at least one of scents, lighting, temperature, humidity, video, or sound in the environment around the person (i.e. when the sleeping environment control device 100 using reinforcement learning wants to lower a body temperature of the user and increase humidity of a space in which the user is located, the control signal may be the signal that causes the operation unit 120 to perform the cold air supply operation and the humidification operation, and the operation unit 120 may perform the cold air supply operation and the humidification operation to lower the body temperature of the user and increase humidity of the space in which the user is located. That is, the control signal may control the operation unit so as to change various conditions of the sleeping environment to influence the quality of sleep of the user and enable the user to take sleep in the improved environment; para. [0061]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of He, Arar, and Aghdaie to include the feature of Lee. One would have been motivated to make this modification because it makes the system adaptable to user preferences or physiological conditions
8. Claims 6 and 8-11 are rejected under 35 U.S.C. 103 as being unpatentable over He in view of Arar, Aghdaie, and further in view of Ohno et al. (U.S. Patent Application Pub. No. US 20220351853 A1).
Claim 6: He, Arar, and Aghdaie teach the machine learning system according to claim 1. He does not explicitly teach wherein the circuitry is further configured to control generated scent based on the environment information selected by the machine learning classifier.
However, Ohno teaches wherein the circuitry is configured to control generated scent based on the environment information selected by the machine learning classifier (i.e. Examples of the above environment control include projecting a specific video image onto a display, playing specific music from a speaker, emitting a specific scent from a diffuser, having a patient talk with a specific person (e.g., a family member and a responder) and a virtual person (e.g., idle Artificial Intelligence (hereinafter referred to as AI), family AI, and responder AI) via an environment management system, changing a condition of a bed (the angle of the bed or the sheet thereof), providing vibration or electrical stimulation to a patient, moving a bed to a specific location, changing a temperature or humidity, controlling a lighting apparatus (changing its brightness and color temperature), and automatically opening and closing a curtain; para. [0041]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of He, Arar, and Aghdaie to include the feature of Ohno. One would have been motivated to make this modification because it enables systems to tailor olfactory output on individual behavioral data and environmental context.
Claim 8: He, Arar, and Aghdaie teach the machine learning system according to claim 1. He does not explicitly teach wherein the circuitry is further configured to control light to be emitted based on the environment information selected by the machine learning classifier.
However, Ohno teaches wherein the circuitry is further configured to control light to be emitted based on the environment information selected by the machine learning classifier (i.e. Examples of the above environment control include projecting a specific video image onto a display, playing specific music from a speaker, emitting a specific scent from a diffuser, having a patient talk with a specific person (e.g., a family member and a responder) and a virtual person (e.g., idle Artificial Intelligence (hereinafter referred to as AI), family AI, and responder AI) via an environment management system, changing a condition of a bed (the angle of the bed or the sheet thereof), providing vibration or electrical stimulation to a patient, moving a bed to a specific location, changing a temperature or humidity, controlling a lighting apparatus (changing its brightness and color temperature), and automatically opening and closing a curtain; para. [0041]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of He, Arar, and Aghdaie to include the feature of Ohno. One would have been motivated to make this modification because it enables systems to tailor lighting output on individual behavioral data and environmental context.
Claim 9: He, Arar, and Aghdaie teach the machine learning system according to claim 1. He does not explicitly teach wherein the circuitry is further configured to control at least one of a temperature or humidity based on the environment information selected by the machine learning classifier.
However, Ohno teaches wherein the circuitry is further configured to control a temperature or humidity based on the environment information selected by the machine learning classifier (i.e. Examples of the above environment control include projecting a specific video image onto a display, playing specific music from a speaker, emitting a specific scent from a diffuser, having a patient talk with a specific person (e.g., a family member and a responder) and a virtual person (e.g., idle Artificial Intelligence (hereinafter referred to as AI), family AI, and responder AI) via an environment management system, changing a condition of a bed (the angle of the bed or the sheet thereof), providing vibration or electrical stimulation to a patient, moving a bed to a specific location, changing a temperature or humidity, controlling a lighting apparatus (changing its brightness and color temperature), and automatically opening and closing a curtain; para. [0041, 0048]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of He, Arar, and Aghdaie to include the feature of Ohno. One would have been motivated to make this modification because it enables systems to tailor temperature output on individual behavioral data and environmental context.
Claim 10: He, Arar, and Aghdaie teach the machine learning system according to claim 1. He does not explicitly teach wherein the circuitry is further configured to control a video to be displayed based on the environment information selected by the machine learning classifier.
However, Ohno teaches wherein the circuitry is further configured to control a video to be displayed based on the environment information selected by the machine learning classifier (i.e. Examples of the above environment control include projecting a specific video image onto a display, playing specific music from a speaker, emitting a specific scent from a diffuser, having a patient talk with a specific person (e.g., a family member and a responder) and a virtual person (e.g., idle Artificial Intelligence (hereinafter referred to as AI), family AI, and responder AI) via an environment management system, changing a condition of a bed (the angle of the bed or the sheet thereof), providing vibration or electrical stimulation to a patient, moving a bed to a specific location, changing a temperature or humidity, controlling a lighting apparatus (changing its brightness and color temperature), and automatically opening and closing a curtain; para. [0041]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of He, Arar, and Aghdaie to include the feature of Ohno. One would have been motivated to make this modification because it enables systems to tailor video output on individual behavioral data and environmental context.
Claim 11: He, Arar, and Aghdaie teach the machine learning system according to claim 1. He does not explicitly teach wherein the circuitry is further configured to control a sound to be played based on the environment information selected by the machine learning classifier.
However, Ohno teaches wherein the circuitry is further configured to control a sound to be played based on the environment information selected by the machine learning classifier (i.e. Examples of the above environment control include projecting a specific video image onto a display, playing specific music from a speaker, emitting a specific scent from a diffuser, having a patient talk with a specific person (e.g., a family member and a responder) and a virtual person (e.g., idle Artificial Intelligence (hereinafter referred to as AI), family AI, and responder AI) via an environment management system, changing a condition of a bed (the angle of the bed or the sheet thereof), providing vibration or electrical stimulation to a patient, moving a bed to a specific location, changing a temperature or humidity, controlling a lighting apparatus (changing its brightness and color temperature), and automatically opening and closing a curtain. The display may be a television set located in an area around the bed, or a projector may be used to project a video image onto a surface near the bed such as the ceiling. When specific music is played from a speaker, in consideration of the influence of the music on the surrounding patients, a sound reproduction technique for producing a directional sound may be used for the speaker so that only a target patient can hear the sound. For example, a speaker built into a pillow of a target patient or a speaker near the pillow may be used. Further, the sound may be reproduced by combining a plurality of ultrasonic waves; para. [0041]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of He, Arar, and Aghdaie to include the feature of Ohno. One would have been motivated to make this modification because it enables systems to tailor sound output on individual behavioral data and environmental context.
9. Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over He in view of Arar, Aghdaie, Ohno, and further in view of Deveaux et al. (U.S. Patent Application Pub. No. US 20210090156 A1).
Claim 7: He, Arar, Aghdaie, and Ohno teach the machine learning system according to claim 6. He does not explicitly teach wherein the circuitry is further configured to make items have scent based on the environment information selected by the machine learning classifier, and the machine learning classifier determines which of scent is generated based on the environment information.
Deveaux teaches wherein the circuitry is further configured to make items have scent based on the environment information selected by the machine learning classifier, and the machine learning classifier determines which of scent is generated based on the environment information (i.e. the one or more environment changing devices comprises an aroma generator for providing a scent or smell and the device controller is arranged to cause the aroma generator to generate an aroma or small associated with the item or a profile associated with the item; para. [0197]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of He, Arar, Aghdaie, and Ohno to include the feature of Deveaux. One would have been motivated to make this modification because it enables systems to localize scent application.
10. Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over He in view of Arar, Aghdaie, and further in view of Dietterich (Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition; published 2000, pages 227-303).
Claim 12: He, Arar, and Aghdaie teach the machine learning system according to claim 1. He does not explicitly teach wherein the value function is divided into a plurality of value groups, and wherein the machine learning classifier uses the value function held by each of the plurality of value groups.
However, Dietterich teaches wherein the value function is divided into a plurality of value groups, and wherein the machine learning classifier uses the value function held by each of the plurality of value groups (i.e. This paper presents a new approach to hierarchical reinforcement learning based on decomposing the target Markov decision process (MDP) into a hierarchy of smaller MDPs and decomposing the value function of the target MDP into an additive combination of the value functions of the smaller MDPs. The decomposition, known as the MAXQ decomposition; pages 227-233).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of He, Arar, and Aghdaie to include the feature of Dietterich. One would have been motivated to make this modification because it supports multi-objective reinforcement learning.
11. Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over He in view of Arar, Aghdaie, and further in view of Arora et al. (U.S. Patent Pub. No. US 11030535 B1).
Claim 16: He, Arar, and Aghdaie teach the machine learning system according to claim 13. He does not explicitly teach wherein the achievement difficulty level includes a number of key variables indicating an average number of items in the environment information when the target behavior is prompted.
However, Aghdaie further teaches wherein the achievement difficulty level includes a number of key variables (i.e. The prediction models 160A, 160B, 160N may generally include a set of one or more parameters 162A, 162B, 162N, respectively (which may be referred to collectively as “parameters 162”). Each set of parameters 162 (such as parameters 162A) may be combined using one or more mathematical functions to obtain a parameter function; para. [0069, 0071]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of He and Arar to include the feature of Aghdaie. One would have been motivated to make this modification because it provides an interpretable measure of how difficult it is for a use to achieve the target behavior.
However, Arora teaches a number of key variables indicating an average number of items in the environment information when the target behavior is prompted (i.e. Other metrics calculated by the data analyzer 130 using the behavior data 122 and/or the merchant data 112 may include, without limitation, an average amount of time spent (average duration of a visit) at the merchant location or on the merchant's site, a frequency of purchasing a particular item and/or a frequency of purchasing items from a particular merchant 112, a number of times, or frequency at which, the customer 102 has visited a particular merchant 112, an average transaction amount at a particular type of merchant 112, an average quantity of items ordered/purchased, number of likes or posts on a social media site regarding a particular merchant 112, a number of positive or negative emotive words recognized in text or audio data, statistics on merchant ratings (e.g., average number, total number, mode, median, etc.), and so on; col. 5 line 52 to col. 6 line 23).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of He, Arar, and Aghdaie to include the feature of Arora. One would have been motivated to make this modification because it improves the difficulty calculation with an additional quantitative contextual variable indicating the average number of items associated with the user’s environment.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
Han et al. (Pub. No. US 20210287128 A1), The reinforcement learning may refer to a learning method in which an agent defined in a certain environment learns to select a behavior or a behavior sequence that maximizes cumulative compensation in each state.
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33, 216 U.S.P.Q. 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 U.S.P.Q. 275, 277 (C.C.P.A. 1968)).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAN TRAN whose telephone number is (303)297-4266. The examiner can normally be reached on Monday - Thursday - 8:00 am - 5:00 pm MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Ell can be reached on 571-270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TAN H TRAN/Primary Examiner, Art Unit 2141
Prosecution Insights