CTNF 18/517,931 CTNF 87274 DETAILED ACTION This action is responsive to the following communications: Original Application filed on November 22, 2023. All references to this application refer to the U.S. Patent Application Publication No. 2024/0256885 A1. 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Claims 1-12 are pending in this case. Claims 1 and 7 are the independent claims. Claims 1-4 and 7-10 are rejected. Claims 5, 6, 11, and 12 are objected to. Priority Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55. Applicants have perfected priority to Korean Patent Application No. 10-2023-0010071, filed on January 26, 2023. Claim Rejections - 35 USC § 112 07-30-02 AIA The following is a quotation of 35 U.S.C. 112(b): (b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention. The following is a quotation of 35 U.S.C. 112 (pre-AIA), second paragraph: The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention. 07-34-01 Claims 2, 3, 8, and 9 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Dependent claims 2 and 8 recite “to generate the positive reward estimation value.” There is a lack of antecedent basis for “the positive reward estimation value.” Accordingly, dependent claims 2 and 8 is rendered indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. To overcome these rejections, the Examiner recommends amending the claims to recite “to generate a positive reward estimation value.” Dependent claims 3 and 9 are rejected solely due to their dependence from a rejected parent claim. To expedite a complete examination of the instant application, the claims rejected above under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA), second paragraph, for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention, are further rejected as set forth below in anticipation of amendments to these claims to correct the failure. Claim Rejections - 35 USC § 101 07-04-01 AIA 07-04 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claims 1-12 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. With regard to independent claim 1 , Step 2A, Prong 1 This part of the eligibility analysis evaluates whether the claim recites a judicial exception. As explained in MPEP 2106.04, subsection II, a claim “recites” a judicial exception when the judicial exception is “set forth” or “described” in the claim. Claim 1 recites: An exploration method based on reward decomposition in multi-agent reinforcement learning, the exploration method comprising: generating a positive reward estimation model through neural network training based on training data including states of all agents, actions of all the agents, and a global reward true value ; generating, for each of the agents, a first individual utility function based on the global reward true value and generating a second individual utility function using the positive reward estimation model ; and determining an action of each of the agents using the first individual utility function and the second individual utility function based on the state of each of the agents . The broadest reasonable interpretation of the bolded limitations above are directed to a mental process able to be performed in the human mind or by a human using pen and paper. A human can generate a reward model through NN training based on training data, generate first and second utility functions based on reward values, and determine actions for each agent using the utility functions mentally or with pen and paper. Step 2A, Prong 1 (Yes). Step 2A, Prong 2 This part of the eligibility analysis evaluates whether the claim as a whole integrates the recited judicial exception into a practical application of the exception or whether the claim is “directed to” the judicial exception. This evaluation is performed by (1) identifying whether there are any additional elements recited in the claim beyond the judicial exception, and (2) evaluating those additional elements individually and in combination to determine whether the claim as a whole integrates the exception into a practical application. See MPEP 2106.04(d). There are no additional elements in this claim. Step 2A, Prong 2 (Yes). Step 2B This part of the eligibility analysis evaluates whether the claim as a whole amounts to significantly more than the recited exception i.e., whether any additional element, or combination of additional elements, adds an inventive concept to the claim. See MPEP 2106.05. As explained with respect to Step 2A, there are no additional elements. Step 2B (Yes). Claim 1 is ineligible. With respect to independent claim 7 , These claims are similar in scope to Claim 1 and are rejected under a similar rationale. The computer system comprising a memory and at least one processor recited in the claim is also a generic computing component. Claim 7 is ineligible. Dependent Claims: Claims 2-6 and 8-12 : These claims only recite further abstract ideas (mental processes) and thus are ineligible . To expedite a complete examination of the instant application, the claims rejected above under 35 U.S.C. 101, as relating to judicial exceptions without significantly more, are further rejected as set forth below in anticipation of amendments to these claims to place them within the four statutory categories of invention. Examiner’s Note 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. Claim Rejections - 35 USC § 103 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-23-aia AIA The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows: 1. Determining the scope and contents of the prior art. 2. Ascertaining the differences between the prior art and the claims at issue. 3. Resolving the level of ordinary skill in the pertinent art. 4. Considering objective evidence present in the application indicating obviousness or nonobviousness. 07-20-02-aia AIA This application currently names joint inventors. In considering patentability of the claims the Examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicants are advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the Examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention. 07-21-aia AIA Claim s 1-4 and 7-10 are rejected under 35 U.S.C. 103 as being unpatentable over Non-Patent Literature Reference entitled “Locality Matters: A Scalable Value Decomposition Approach for Cooperative Artificial Intelligence,” by Zohar et al., 36 th AAAI Conference on Artificial Intelligence, 2022 (hereinafter Zohar), in view of Non-Patent Literature Reference entitled “RD2: Reward Decomposition with Representation Disentanglement,” 34 th Conference on Neural Information Processing Systems, 2020 (hereinafter Lin) . With respect to independent claim 1 , Zohar discloses an exploration method based on reward decomposition in multi-agent reinforcement learning, the exploration method comprising : Generating a positive reward estimation model through neural network training based on training data including states of all agents, actions of all the agents, and a global reward true value ; Zohar discloses generating a positive reward estimation model based on training data comprising the entire state space of multi agents and a global reward (see Zohar, pg. 9278, right col., [utilize local rewards which are critical for learning in scalable settings], 9279, left col. [scalable value decomposition that leverages local agent rewards while maintaining a cooperative objective], 9279, right col. [Assumption 1, the reward function is additively decomposable], 9282, left col. [section 4.2, describing how the global reward is computed by minimizing the loss while using training data of the tuple (state, action, global true)]). Generating, for each of the agents, a first individual utility function based on the global reward true value…; Zohar discloses generating a first utility function of each agent based on the global reward true value (see Zohar, pgs. 9281-9282 (section 4.1) [the neural network is trained by minimizing the loss where the local agent states are submitted to a feed-forward utility network architecture in order to output an encoded vector of each agent’s states, which is used to approximate the function]; see also, Zohar, Algorithm 2; see also, Zohar, Fig. 2) Zohar fails to expressly disclose generating a second individual utility function using the positive reward estimation model . However, Lin teaches generating a second utility function by decomposing the goal into sub-tasks and basing the agent’s utility function on that sub-task (see Lin, pages 3-4 [each sub-reward should be unique to relevant features exclusive to the sub-task] and page 5 [each agent’s reward is directly dependent of the sub-state and the sub-state reward]). Accordingly, it would have been obvious to one of ordinary skill in the art, having the teachings of Zohar and Lin before him before the effective filing date of the claimed invention, to modify the method of Zohar to incorporate a second individual agent utility function as taught by Lin. One would have been motivated to make such a combination because this relates sub-actions and their reward to the relevant utility features for each agent as taught by Lin (see Lin, Introduction). Zohar, as modified by Lin, further teaches determining an action of each of the agents using the first individual utility function and the second individual utility function based on the state of each of the agents . Zohar further teaches determining each agent’s action based on the optimization of the global task objective (e.g., first utility function) (see Zohar, pgs. 9281-9282 (section 4.1); see also, Zohar, Algorithm 2; see also, Zohar, Fig. 2, described supra ). Additionally, Lin further teaches that the agent’s actions may be decomposed into relevant sub-tasks, such that each agent’s actions are determined based on optimization of completion of the sub-task (see Lin, pages 3-5, described supra ). With respect to dependent claim 2 , Zohar, as modified by Lin, teaches the exploration method of claim 1, as described above. Zohar further teaches the method wherein the generating of the positive reward estimation model includes : Inputting the state of the agent included in the training data into an encoding neural network to generate a state encoding vector, and inputting the action of the agent included in the training data into the encoding neural network to generate an action encoding vector ; Zohar further teaches generating state encoding vectors and action encoding vectors (see Zohar, pgs. 9281-9282 (section 4.1); see also, Zohar, Algorithm 2; see also, Zohar, Fig. 2, described supra , claim 1). Inputting the state encoding vector and the action encoding vector into a global reward neural network to generate a global reward estimation value ; Zohar further teaches inputting the vectors into a reward NN to generate the reward estimation (see Zohar, pgs. 9281-9282 (section 4.1); see also, Zohar, Algorithm 2; see also, Zohar, Fig. 2, described supra , claim 1). Training a positive local reward neural network included in the global reward neural network using a loss function based on the global reward estimation value and the global reward true value, to generate [a] positive reward estimation value ; Zohar further teaches training a local reward NN using an optimized loss function based on both the reward estimation value and the global reward true value which outputs the estimation value (see Zohar, pgs. 9281-9282 (section 4.1); see also, Zohar, Algorithm 2; see also, Zohar, Fig. 2, described supra , claim 1). With respect to dependent claim 3 , Zohar, as modified by Lin, teaches the exploration method of claim 2, as described above. Zohar further teaches the method wherein the generating of the positive reward estimation model includes : Inputting the global reward estimation value and the global reward true value into the loss function ; Zohar further teaches inputting the estimation and true reward values into the loss function (see Zohar, pgs. 9281-9282 (section 4.1); see also, Zohar, Algorithm 2; see also, Zohar, Fig. 2, described supra , claim 1). Training the positive local reward neural network such that a function value of the loss function is minimized, to generate the positive reward estimation model ; Zohar further teaches training the reward NN by optimizing the loss function (see Zohar, pgs. 9281-9282 (section 4.1); see also, Zohar, Algorithm 2; see also, Zohar, Fig. 2, described supra , claim 1). With respect to dependent claim 4 , Zohar, as modified by Lin, teaches the exploration method of claim 1, as described above. Zohar and Lin further teach the method wherein the determining of the action includes : Selecting any one of the first individual utility function and the second individual utility function according to a predetermined criterion ; Zohar further teaches selecting the first utility function according to the global task (criterion) (see Zohar, pgs. 9281-9282 (section 4.1); see also, Zohar, Algorithm 2; see also, Zohar, Fig. 2, described supra , claim 1). Additionally, Lin further teaches selecting the second utility function based on the sub-task (criterion) (see Lin, pages 3-5, described supra , claim 1). Selecting any one of a random action and an action that maximizes a value of the selected individual utility function according to a predetermined criterion based on the state of each of the agents ; Zohar further teaches selecting an action the maximizes the first utility function according to the state of each agent (see Zohar, pgs. 9281-9282 (section 4.1); see also, Zohar, Algorithm 2; see also, Zohar, Fig. 2, described supra , claim 1). Additionally or alternatively, Lin further teaches selecting an action that maximizes the second utility function based on the state of each agent (see Lin, pages 3-5, described supra , claim 1). Independent claim 7 , and its respective dependent claims 8-10 , recite a computer system comprising: a memory in which instructions readable by a computer are stored; and at least one processor implemented to execute the instructions, wherein the at least one processor is configured to execute the instructions to perform the method of independent claim 1 , and its respective dependent claims 2-4 . Accordingly, independent claim 7 , and its respective dependent claims 8-10 , are rejected under the same rationales used to reject independent claim 1 , and its respective dependent claims 2-4 , which are incorporated herein. Allowable Subject Matter 15. Claims 5, 6, 11, and 12 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims, a well as the rejections under 35 USC 101. Conclusion 07-96 The prior art made of record and not relied upon is considered pertinent to Applicants’ disclosure. See PTO-892. It is noted that any citation to specific pages, columns, figures, or lines in the prior art references any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck , 699 F.2d 1331-33, 216 USPQ 1038-39 (Fed. Cir. 1983) ( quoting In re Lemelson , 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)). Any inquiry concerning this communication or earlier communications from the Examiner should be directed to ERIC J. BYCER whose telephone number is (571) 270-3741. The Examiner can normally be reached Monday - Thursday 9am-6pm, and alternate Fridays 9am-5pm. Examiner interviews are available via a variety of formats. See MPEP § 713.01. To schedule an interview, Applicants are encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/InterviewPractice. If attempts to reach the Examiner by telephone are unsuccessful, the Examiner’s supervisor, MATT ELL can be reached on (571) 270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of an application may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center. Status information for unpublished applications is available through Patent Center to authorized users only. Should you have questions about access to the USPTO patent electronic filing system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). /ERIC J. BYCER/ Primary Examiner Art Unit 2141 Application/Control Number: 18/517,931 Page 2 Art Unit: 2141 Application/Control Number: 18/517,931 Page 3 Art Unit: 2141 Application/Control Number: 18/517,931 Page 4 Art Unit: 2141 Application/Control Number: 18/517,931 Page 5 Art Unit: 2141 Application/Control Number: 18/517,931 Page 6 Art Unit: 2141 Application/Control Number: 18/517,931 Page 7 Art Unit: 2141 Application/Control Number: 18/517,931 Page 8 Art Unit: 2141 Application/Control Number: 18/517,931 Page 9 Art Unit: 2141 Application/Control Number: 18/517,931 Page 10 Art Unit: 2141 Application/Control Number: 18/517,931 Page 11 Art Unit: 2141 Application/Control Number: 18/517,931 Page 12 Art Unit: 2141 Application/Control Number: 18/517,931 Page 13 Art Unit: 2141
Prosecution Insights