DETAILED ACTION
This action is responsive to the following communications: Response to Restriction Requirement filed on July 21, 2026. All references to this application refer to the U.S. Patent Application Publication No. 2023/0394758 A1.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-6, 9-14, and 16-20 are pending in this case. Claims 1, 2, 4, 6, 9, 10, 14, 16, and 17 were amended. Claims 7, 8, and 15 are withdrawn as directed to a Non-Elected Invention. Claims 1, 9, 16, and 20 are the independent claims. Claims 1-6, 9-14, and 16-20 are rejected.
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55. Applicants have perfected priority to CN 202210621933.7, filed on June 1, 2022.
Election/Restrictions
Applicants’ election of Invention I (claims 1-6, 9-14, and 16-20) in the reply filed on July 21, 2026, is acknowledged. Because the Applicants did not distinctly and specifically point out the supposed errors in the restriction requirement, the election has been treated as an election without traverse (MPEP § 818.01(a)).
Specification
The disclosure is objected to because of the following informalities:
The equation in paragraph 0117 (describing the probability of selecting one role type b in the candidate set C) is neither legible nor reproducible. Applicants are required to provide a Specification amendment with a more legible equation.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 6 and 14 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Dependent claims 6 and 14 recite the following:
…determining a first winning probability between a virtual object corresponding to the training role type and a virtual object corresponding to each candidate role type among the role types except the training role type;
…and the second reward weight parameter is each reward weight parameter among a plurality of reward weight parameters except the first reward weight parameter;
This language is unclear, and appears to be based, at least in part, on the equation in paragraph 0117 (indicated above), which cannot be understood due to illegibility. This is further complicated because the final limitation uses the winning probabilities and reward weight parameters to determine the role type and the training reward parameters. Because there is so much uncertainty in this claim language, and furthermore, because the Specification fails to provide insight as to how these aspects are to be determined, the claims are rendered indefinite.
Furthermore, because there is such uncertainty in the meaning of the limitations, claims 6 and 14 are not examined under 35 USC 102/103.
Second, where there is a great deal of confusion and uncertainty as to the proper interpretation of the limitations of a claim, it would not be proper to reject such a claim on the basis of prior art. As stated in In re Steele, 305 F.2d 859, 134 USPQ 292 (CCPA 1962), a rejection under 35 U.S.C. 103 should not be based on considerable speculation about the meaning of terms employed in a claim or assumptions that must be made as to the scope of the claims.
MPEP 2173.06(II) (emphasis added).
To expedite a complete examination of the instant application, the claims rejected above under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention, are further rejected as set forth below in anticipation of amendments to these claims to correct the failure.
Examiner’s Note
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the Examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicants are advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the Examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-5, 9-13, and 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Chinese Patent Application Publication No. CN112926628, published on June 8, 2021, by Fan et al. (hereinafter Fan)1, in view of Chinese Patent Application Publication No CN109513215B, published on April 12, 2022, by Liu et al. (hereinafter Liu).
With respect to independent claim 1, Fan discloses a method for training a deep reinforcement learning model, comprising:
Acquiring an interaction sequence generated by an interaction between a first virtual object and a second virtual object in a virtual environment, wherein the interaction sequence comprises multiple sampled data; Fan discloses acquiring an interaction sequence of a first virtual object and a second virtual object in a virtual environment comprising multiple sampled data (see Fan, pages 3-4, beginning at the bottom of the page [describing Step 11, in which an interaction sequence is obtained based on multiple sampled datapoints based on state-action pairs, such as in a game scene between players and the environment]).
Each of the sampled data comprises a return value obtained by the first virtual object executing a decision action under a state feature sampled in the virtual environment, the first virtual object is controlled based on a training deep reinforcement learning model, and the second virtual object is controlled based on a historical deep reinforcement learning model corresponding to the training deep reinforcement learning model; Fan discloses each sample data point comprises a return value obtained as a result of executing a decision based on DRL model corresponding to the training DRL model (see Fan, pages 3-4 [describing step 12, in which the advantage function is determined based on the decision strategy and entropy (loss)], page 4 [describing step 13, in which the action value is determined based on the dominance function, dominance expectation, and the state value function], and page 6 [which provides the equations which describe the return values for each action and the equation that defines the expected cumulative return]).
Fan does not appear to expressly disclose acquiring a training reward weight parameter corresponding to each interaction sequence, wherein the training reward weight parameter corresponds to a decision style type of the training deep reinforcement learning model.
However, Liu teaches determining positive and negative outputs (e.g., rewards) for each of the parties and for each of the different potential outcomes (see Liu, page 10, center of the page [describing the prediction combination for the different queues for the battle each with particular weights assigned to the reward/penalty] and page 11, top of the page [describing how different roles have different skill sets and those different skills/roles determine various decision style types for the actions taken]).
Accordingly, it would have been obvious to one of ordinary skill in the art, having the teachings of Fan and Liu before him before the effective filing date of the claimed invention, to modify the method of Fan to incorporate acquiring training reward weights based on decision style types as taught by Liu. One would have been motivated to make such a combination because this increases accuracy and achieves greater balance as compared to typical ELO style integration, as taught by Liu (see Liu, page 2, center of the page [“Therefore, the matching system based on ELO integration algorithm only based on ELO to locate the player level, but the location is not accurate enough, after matching the strength difference is too large, affecting the balance of the game, reducing the player game experience.”]).
Fan, as modified by Liu, further teaches
Determining a first return value corresponding to each of the sampled data, according to the training reward weight parameter corresponding to the interaction sequence and the return value in the interaction sequence; Fan further teaches determining a first return value corresponding to the sampled data according to the training reward weight parameters (see Fan, pages 3-4, described steps 12 and 13, described supra).
Determining a loss of the training deep reinforcement learning model, according to an action-value predicted value determined based on a state feature and a decision action of each of the sampled data and the first return value corresponding to the sampled data; Liu further teaches determining a loss function for the DRL model according to the state-action pair and the return value (see Liu, pages 10-11, describing the weight parameters and the loss function, described supra).
Training the training deep reinforcement learning model based on the loss; Fan further teaches training the DRL model based on the loss function (see Fan, page 9, starting at the top of the page [describing the determination of the loss function which is then used to train/update the DRL model via gradient updates].
With respect to dependent claim 2, Fan, as modified by Liu, teaches the method according to claim 1, as described above.
Fan and Liu further teach the method
Wherein an output layer of the training deep reinforcement learning model comprises a plurality of output forks, and the plurality of output forks are in one-to-one correspondence with a plurality of reward weight parameters; Fan further teaches the output layer of the DRL model comprising multiple output paths (e.g., forks) (see Fan, pages 3-4, describing steps 12 and 13, described supra, claim 1). Additionally, Liu further teaches the output forks are in a one-to-one correspondence with the reward weight parameters (see Liu, page 10, described supra, claim 1).
The training the training deep reinforcement learning model based on the loss, comprises:
Updating a parameter corresponding to an output fork corresponding to the training reward weight parameter in the training deep reinforcement learning model based on the loss, wherein the training reward weight parameter is one of the plurality of reward weight parameters; Fan further teaches updating the DRL model output path (e.g., fork) corresponding to the determined advantage and loss functions (see Fan, pages 3-4 and 9, described supra, claim 1).
With respect to dependent claim 3, Fan, as modified by Liu, teaches the method according to claim 1, as described above.
Fan and Liu further teach the method wherein the interaction sequence is generated by:
Determining the training reward weight parameter for the training deep reinforcement learning model; Liu further teaches determining the reward weight parameter for training the DRL model (see Liu, pages 10-11, described supra, claim 1).
Sampling the interaction between the first virtual object and the second virtual object in the virtual environment, in order to obtain a state feature corresponding to the first virtual object; Fan further teaches sampling the interaction sequence between the first and second objects in the environment to obtain a state feature for each virtual object (see Fan, page 6, described supra, claim 1).
Determining a decision action for the first virtual object under the state feature, based on the state feature, the training reward weight parameter, and the training deep reinforcement learning model, in order to control an operation of the first virtual object based on the decision action, and returning to the step of sampling interaction between the first virtual object and the second virtual object in the virtual environment in order to obtain the state feature corresponding to the first virtual object, until an end of an interaction round; Fan further teaches determining the decision action for the first virtual object based on the state feature and the expected return values to control the object to transition to a second state based on the policy and resuming sampling data for the next epoch (see Fan, pages 3-4 and 6, described supra, claim 1). Additionally, Liu further teaches the decision action is further based on the training reward parameter (see Liu, page 10, described supra, claim 1).
Sorting the sampled data acquired by sampling from the interaction round in an order of sampling time, in order to obtain the interaction sequence; and associating the interaction sequence with the training reward weight parameter, wherein each of the sampled data comprises the return value obtained by executing the decision action under the state feature in the virtual environment; Fan further teaches sorting the sample data based on the acquisition time of the sample data (e.g., timestamps) (see Fan, pages 3-4 and 6, described supra, claim 1). Additionally, Liu further teaches associating the decisions with the training reward weights for executing the decisions (see Liu, page 10, described supra, claim 1).
With respect to dependent claim 4, Fan, as modified by Liu, teaches the method according to claim 3, as described above.
Fan and Liu further teach the method
Wherein the output layer of the training deep reinforcement learning model comprises a plurality of output forks; Fan further teaches the output layer of the DRL model comprising multiple output paths (e.g., forks) (see Fan, pages 3-4, describing steps 12 and 13, described supra, claim 1).
The plurality of output forks are in one-to-one correspondence with a plurality of reward weight parameters; Liu further teaches the output forks are in a one-to-one correspondence with the reward weight parameters (see Liu, page 10, described supra, claim 1)..
The determining a decision action of the first virtual object under the state feature, based on the state feature, the training reward weight parameter, and the training deep reinforcement learning model, comprises:
Inputting the state feature and the training reward weight parameter into the training deep reinforcement learning model, in order to perform a feature extraction on the state feature and the training reward weight parameter by a feature layer of the training deep reinforcement learning model, obtaining sub-outputs corresponding to the plurality of output forks of the output layer based on an extracted feature, and determining the decision action according to the sub-output corresponding to the output fork corresponding to the training reward weight parameter in the training deep reinforcement learning model; Fan further teaches inputting the state feature into the DRL model to perform feature extraction on the state feature and obtain outputs and sub-outputs corresponding to the output paths (e.g., forks) and determining the decision action based on the expected return, loss, and value function (see Fan, pages 3-4 and 6, described supra, claim 1). Additionally, Liu further teaches determining the decision action based in part on the training reward weight parameters (see Liu, page 10, described supra, claim 1).
With respect to dependent claim 5, Fan, as modified by Liu, teaches the method according to claim 3, as described above.
Fan and Liu further teach the method wherein the determining a decision action for the first virtual object under the state feature, based on the state feature, the training reward weight parameter, and the training deep reinforcement learning model, comprises:
Inputting the state feature and the training reward weight parameter into the training deep reinforcement learning model, and determining the decision action according to the output of the training deep reinforcement learning model; Fan further teaches inputting the state feature into the training DRL model and determining the decision action based on the output of the DRL model (see Fan, pages 3-4 and 6, described supra, claim 1). Additionally, Liu further teaches determining the decision action based in part on the training reward weight parameters (see Liu, page 10, described supra, claim 1).
Wherein the training deep reinforcement learning model comprises a neural network feature layer, a type feature layer, and an attention layer; Fan further teaches the DRL model comprising a plurality of layers, including network feature layer, type feature layer, and attention (see Fan, page 10, describing the structure of the DRL model). Additionally, Liu further teaches a DRL model comprising network feature layer, type feature layer, and attention (see Liu, pages 11-12, starting at the bottom of page 11 [describing the structure of the model]).
The neural network feature layer is used for determining a state feature vector corresponding to the state feature and a parameter feature vector corresponding to the training reward weight parameter; Fan further teaches the network feature layer is used for determining the state vector and parameter vector (see Fan, pages 3-4 and 6, described supra, claim 1 and page 10, described supra).
The type feature layer is used for determining a type feature vector corresponding to each candidate type under the type feature layer based on the state feature vector and the parameter feature vector; Liu further teaches the type feature layer to determine each candidate type (see Liu, page 11, described supra, claim 1).
Wherein the candidate type is in one-to-one correspondence with a hidden-layer decision style; Liu further teaches the candidate type is in a one-to-one correspondence with hidden layer decision style (see Liu, pages 10-11, described supra, claim 1).
The attention layer is used for fusing a type feature vector corresponding to each type based on an attention mechanism according to the parameter feature vector, and determining the output of the training deep reinforcement learning model according to a result of the fusing; Fan further teaches the attention layer fuses type vector with the parameter vector to determine the DRL model output (see Fan, page 6, described supra, claim 1).
Independent claim 9, and its respective dependent claims 10-13, recite an apparatus for training a deep reinforcement learning model that performs the method recited in independent claim 1, and its respective dependent claims 2-5. Accordingly, independent claim 9, and its respective dependent claims 10-13, are rejected under the same rationales used to reject independent claim 1, and its respective dependent claims 2-5, which are incorporated herein.
Independent claim 16, and its respective dependent claims 17-19, recite a computer readable medium, having a computer program stored thereon, wherein when executed by a processing apparatus, the computer program causes the processing apparatus to perform the method recited in independent claim 1, and its respective dependent claims 2-4. Accordingly, independent claim 16, and its respective dependent claims 17-19, are rejected under the same rationales used to reject independent claim 1, and its respective dependent claims 2-4, which are incorporated herein.
Independent claim 20 recites an electronic device, comprising, a storage apparatus, having a computer program stored thereon, and a processing apparatus, configured to execute the computer program in the storage apparatus to implement the method recited in independent claim 1. Accordingly, independent claim 20 is rejected under the same rationales used to reject independent claim 1, which are incorporated herein.
Conclusion
The prior art made of record and not relied upon is considered pertinent to Applicants’ disclosure. See PTO-892.
It is noted that any citation to specific pages, columns, figures, or lines in the prior art references any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331-33, 216 USPQ 1038-39 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)).
Any inquiry concerning this communication or earlier communications from the Examiner should be directed to ERIC J. BYCER whose telephone number is (571) 270-3741. The Examiner can normally be reached Monday - Thursday 9am-6pm, and alternate Fridays 9am-5pm.
Examiner interviews are available via a variety of formats. See MPEP § 713.01. To schedule an interview, Applicants are encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/InterviewPractice.
If attempts to reach the Examiner by telephone are unsuccessful, the Examiner’s supervisor, MATT ELL can be reached on (571) 270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center. Status information for unpublished applications is available through Patent Center to authorized users only. Should you have questions about access to the USPTO patent electronic filing system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/ERIC J. BYCER/
Primary Examiner
Art Unit 2141
1 The rejection uses the provided translation for CN112926628 and CN 109513215B. Note: these translations do not include paragraph numbers, so page numbers and approximate locations will be provided.
Prosecution Insights