DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim 1, 5, 8 and 9 is rejected under 35 U.S.C. 103 as being unpatentable over Jiahao Li et al. [US 20250203098 A1] in view of Kevin Gatimu et al. [qMDP: DASH Adaptation using Queueing Theory within a Markov Decision Process].
Regarding claim 1, Jiahao teaches:
1. An information processing apparatus (i.e. Implementations of the subject matter described herein provide a solution for rate control based on reinforcement learning- Abstract) comprising:
at least one hardware processor (i.e. FIG. 1 illustrates a block diagram of a computing device 100 in which various implementations of the subject matter described herein can be implemented. It would be appreciated that the computing device 100 shown in FIG. 1 is merely for purpose of illustration, without suggesting any limitation to the functions and scopes of the implementations of the subject matter described herein in any manner. As shown in FIG. 1, the computing device 100 includes a general-purpose computing device 100. Components of the computing device 100 may include, but are not limited to, one or more processors or processing units 110, a memory 120, a storage device 130, one or more communication units 140, one or more input devices 150, and one or more output devices 160- ¶0014) configured to:
the experience data including
a state st-1 that is a state of encoding of a (t-1)-th image included in a moving image (i.e. At each time step t, the agent 202 observes the encoding state st of the encoder 204. The encoding state st at time step t may be determined based on the encoding of at least a video unit at time step t−1. Based on this input information, the agent 202 makes inference and outputs an action at. The action at indicates how fine the encoder 204 should compress a video unit at time step t- ¶0027) and includes the buffer occupancy rate bt-1(i.e. st may include a state representing an outcome for encoding at least the frame at time step t−1, a state of the transmission buffer at time step t, and a state associated with a status of a network at time step t for transmitting the encoded frames- ¶0029… the state of the transmission buffer may include the usage of the buffer, for example, a ratio of an occupied space to maximum space of the buffer, remaining space of the buffer measured in frames, or its combination. The remaining space of the buffer measured in frames may be calculated by dividing the remaining space of the buffer by the average target frame size. This value describes the buffer usage from another aspect, where the influence of frame rate is considered- ¶0031),
a reward rt that represents an evaluation value of encoding of the (t-1)-th image (i.e. a reward rt is provided after the encoder 204 finishes encoding each video unit with the action at- ¶0043),
an action at-1 obtained by a model that receives an input of the state st-1 (i.e. The action at output by the agent 202 can then be mapped into lambda that the encoder can understand- ¶0041) and outputs an action used for determining parameters (i.e. In some further implementations, the encoding parameter may be implemented as lambda- ¶0041) of encoding (i.e. At each time step t, the agent 202 observes the encoding state st of the encoder 204. The encoding state st at time step t may be determined based on the encoding of at least a video unit at time step t−1. Based on this input information, the agent 202 makes inference and outputs an action at. The action at indicates how fine the encoder 204 should compress a video unit at time step t- ¶0027), and
a state st that is a state of encoding of a t-th image included in the moving image and includes a buffer occupancy rate bt (i.e. The encoding of the video unit at time step t will then be used to update the encoding state st+1 for the agent 202 at time step t+1- ¶0027); and
learn the model using the experience data and the augmented experience data (i.e. Still referring to FIG. 2, in training the reinforcement learning module 200, it is necessary to evaluate how good the action made by agent 202 is. To this end, a reward rt is provided after the encoder 204 finishes encoding each video unit with the action at. The agent 202 may update its policy based on the reward rt when the agent 202 obtains an amount of training samples. The agent 202 can be trained to converge towards the direction that can maximize the accumulated reward. In order to obtain a better QOE, one or more factors reflecting the QOE can be incorporated into the reward. For example, the reward rt is configured to penalize buffer overshooting and to increase as the encoding parameter results in a higher visual quality. For example, the visual quality increases as the quantization parameter or lambda decreases- ¶0043).
However, Jiahao does not teach explicitly:
generate one or more pieces of augmented experience data in which at least a buffer occupancy rate bt-1 included in experience data is changed to a different value on a basis of the experience data.
In the same field of endeavor, Kevin teaches:
generate one or more pieces of augmented experience data in which at least a buffer occupancy rate bt-1 included in experience data is changed to a different value on a basis of the experience data (i.e. we propose a target range normally distributed around this B*-target, i.e. f(B*|μ, σ2), where μ = K-B is the mean and σ2 the variance- page 3, ¶6, fig. 2).
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention, to modify the teachings of Jiahao with the teachings of Kevin to result in higher QoE and faster convergence compared to a QoE-only model-free version (Abstract).
Regarding claim 5, Jiahao and Kevin teach all the limitations of claim 1 and Jiahao further teaches:
wherein the reward rt is calculated to be smaller as a distortion generated by encoding increases and to be smaller as an amount of a buffer occupancy rate exceeding a threshold increases (i.e. where a is a constant factor, b is a negative number, rbase represents a base reward, Bandwidthcur represents the bandwidth of the channel for transmitting the bitstream at time step t, Bandwidthmax represents the maximum bandwidth, and rfinal represents the final reward- ¶0045… The base reward rbase is calculated by Equation (3)… Therefore, the reward is set as a negative number (namely b) for buffer overshooting. Setting the negative number as a penalty is used to train the agent 202 to avoid the buffer overshooting- ¶0046).
Regarding claim 8, method claim 8 corresponds to apparatus claim 1, and therefore is also rejected for the same reasons of obviousness as listed above.
Regarding claim 9, computer-readable medium storing instructions claim 9 corresponds to apparatus claim 1, and therefore is also rejected for the same reasons of obviousness as listed above.
Claims 6-7 are rejected under 35 U.S.C. 103 as being unpatentable over Jiahao Li et al. [US 20250203098 A1] in view of Kevin Gatimu et al. [qMDP: DASH Adaptation using Queueing Theory within a Markov Decision Process] and further in view of Peeyush Kumar et al. [US 20230129665 A1].
Regarding claim 6, Jiahao and Kevin teach all the limitations of claim 1.
However, Jiahao and Kevin do not teach explicitly:
wherein the hardware processor is configured to: output a designation screen for designating a setting value to be used for generation of the augmented experience data, and generate the augmented experience data using the setting value designated by using the designation screen.
In the same field of endeavor, Peeyush teaches:
wherein the hardware processor is configured to: output a designation screen for designating a setting value to be used for generation of the augmented experience data, and generate the augmented experience data using the setting value designated by using the designation screen (i.e. The sharing settings window 76 may further include respective interface elements that may be selected by the user to display estimated rewards, and variance in those rewards, for corresponding information sharing settings. These rewards and variances may be estimated for the runtime agent 122 based on rewards estimated for previously simulated runtime agents 122 before the runtime agent 122 shares its runtime forecast states 124 with the reinforcement learning model 30. Thus, the user interacting with the GUI 72 may view an estimated change in reward that would occur if state sharing were activated or deactivated. For example, the GUI 72 may display an absolute number estimated for the reward or may alternatively display a percentage change in the reward. The estimated reward for an information sharing setting may be estimated, for example, by averaging the simulated rewards obtained by a plurality of other runtime agents 122 that have positions in the runtime supply chain 150 corresponding to that of the runtime agent 122 for which the GUI 72 is displayed. In such examples, the estimated rewards for the runtime agent 122 and the other runtime agents 122 may be normalized by the runtime agents' total quantities of raw materials purchased or products sold in order to account for differences in scale between different runtime agents 122. The variance in the rewards of the runtime agents 122 may also be estimated and displayed in the sharing settings window 76 in some examples. In such examples, the displayed variance may be computed based on the normalized rewards. Additionally or alternatively, the sharing settings window 76 may further include a confidence interval (e.g. a 90% confidence interval or a 95% confidence interval) for the estimated reward.).
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention, to modify the teachings of Jiahao and Keving with the teachings of Peeyush
Regarding claim 7, Jiahao, Kevin and Peeyush teach all the limitations of claim 6 and Jiahao further teaches:
However, Jiahao does not teach explicitly:
wherein the setting value includes a number of pieces of the augmented experience data to be generated and a method of changing the buffer occupancy rate bt-1 to a different value.
In the same field of endeavor, Kevin teaches:
wherein the setting value includes a number of pieces of the augmented experience data to be generated and a method of changing the buffer occupancy rate bt-1 to a different value (i.e. σ2 the variance- page 3, ¶6).
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention, to modify the teachings of Jiahao with the teachings of Kevin to result in higher QoE and faster convergence compared to a QoE-only model-free version (Abstract).
Allowable Subject Matter
Claims 2-4 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CLIFFORD HILAIRE whose telephone number is (571)272-8397. The examiner can normally be reached 5:30-1400.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, SATH V PERUNGAVOOR can be reached at (571)272-7455. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
CLIFFORD HILAIRE
Primary Examiner
Art Unit 2488
/CLIFFORD HILAIRE/Primary Examiner, Art Unit 2488
Prosecution Insights