DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 2-9, 11-16, and 18-20 are objected to because of the following informalities:
Claims 2-9, multiple of instances of “A data processing apparatus” should read “The data processing apparatus”.
Claims 11-16, multiple of instances of “A data processing method” should read “The data processing method”.
Claims 18-20, multiple of instances of “A non-transitory computer-readable storage medium” should read “The non-transitory computer-readable storage medium”.
Claim 5, line 3, “a first output video frame of the second sequence of video frames” should read “the first output video frame of the second sequence of video frames”.
Claim 7, line 3, “past in-game actions” should read “the past in-game actions”.
Claim 14, line 3, “a first output video frame of the second sequence of video frames” should read “the first output video frame of the second sequence of video frames”.
Claim 16, line 3, “past in-game actions” should read “the past in-game actions”.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5, 6, 10-12, 14, 15, and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Steven Osman (US 20200289937 A1; hereinafter Osman) in view of LIU et al. (US 20200394752 A1; hereinafter Liu).
Regarding claims 1, 10, and 17, Osman discloses a data processing apparatus comprising circuitry (the system 100 includes multiple apparatus used for processing video game data; ¶37 and Fig 1) configured to: obtain video game data from a video game application (the system 100 is dealing with video game data; ¶36) indicating an in-game situation (in-game situations naturally happen when a player plays a videogame as shown in examples in ¶36); obtain data indicating a predicted in-game action of a user in response to the in-game situation (obtaining data for a prediction model, such as predicting a subsequent video frame based on one or more previously rendered video frames; ¶26); begin output of a first sequence of video frames depicting the predicted in-game action (generating the one or more frames for a predicted state; ¶13); receive data indicating an instructed in-game action of the user (player input received; ¶16); if the predicted in-game action is the same as the instructed in-game action, continue output of the first sequence of video frames (if user inputs 2 and 3 match the predicted user inputs, then display the frames for this predicted outcome; ¶83); and if the predicted in-game action is not the same as the instructed in-game action: stop output of the first sequence of video frames (if user input 1 is different from predicted inputs 2 and 3, the system stops generating frames for images or audio; ¶89-90); and begin output of a second sequence of video frames depicting the instructed in-game action (immediately starts generating the state 1 from the user input 1; ¶90). For additional limitations of claim 17, Osman discloses a non-transitory computer-readable storage medium storing a program for controlling a computer to perform a data processing method (belongs to the type of client devices; ¶37).
Osman does not explicitly disclose output one or more interpolated video frames. However, Liu focuses on video frame interpolation that is a basic video processing technique that is used to generate intermediate frames between any two consecutive original frames, which relates to Osman because they are both focused on generating frames. Liu teaches output one or more interpolated video frames (video frame interpolation technique used to generate intermediate frames between 2 consecutive original frames; ¶19).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Osman to implement the teachings of Liu because it would allow the system to mask the misprediction and correct it. If the predicted action is not the same as the input from the user, it will avoid abruptly switching from the predicted action to the real action from the user by adding a visual from the interpolated frames.
Regarding claims 2, 11, and 18, Osman does not explicitly disclose wherein the one or more interpolated video frames are generated using a final output video frame of the first sequence of video frames and a first output video frame of the second sequence of video frames.
However, Liu teaches wherein the one or more interpolated video frames are generated using a final output video frame of the first sequence of video frames and a first output video frame of the second sequence of video frames (this is how video frame interpolation works, the interpolated frames will be generated between two consecutive original frames, in this case these two original frames will be a final output video frame of the first sequence of video frames and a first output video frame of the second sequence of video frames; ¶19).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Osman to implement the teachings of Liu because it would allow the system to mask the misprediction and correct it. If the predicted action is not the same as the input from the user, it will avoid abruptly switching from the predicted action to the real action from the user by adding a visual from the interpolated frames.
Regarding claims 3, 12, and 19, Osman discloses wherein the final output video frame of the first sequence of video frames is the video frame being output when the data indicating the instructed in-game action is received (inputs 2 and 3 received by the user are considered part of the final output video frame when they match the predicted user inputs 2 and 3; ¶83).
Regarding claims 5 and 14, Osman does not disclose wherein the one or more interpolated frames are generated using motion interpolation between the final output video frame of the first sequence of video frames and a first output video frame of the second sequence of video frames.
However, Liu teaches wherein the one or more interpolated frames are generated using motion interpolation between the final output video frame of the first sequence of video frames and a first output video frame of the second sequence of video frames (this is how video frame interpolation works, the interpolated frames will be generated between two consecutive original frames. For example the first consecutive original frame is the final output video frame of the first sequence of video frames and the second consecutive original frame is the first output video frame of the second sequence of video frames; ¶19).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Osman to implement the teachings of Liu because it would allow the system to mask the misprediction and correct it. If the predicted action is not the same as the input from the user, it will avoid abruptly switching from the predicted action to the real action from the user by adding a visual from the interpolated frames.
Regarding claims 6, 15, and 20, Osman discloses wherein the data indicating the predicted in-game action of the user is based on past video game data and past in-game actions of one or more second users (determining the prediction based on history of the user or users; ¶110).
Claims 4 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Osman in view of Liu as applied to claims 1-3 and 10-12 above, and further in view of LOBETE et al. (US 20210400170 A1; hereinafter Lobete).
Regarding claims 4 and 13, Osman does not disclose the first output video frame of the second sequence of video frames is the video frame output at two or more frame display times after a frame display time of the final output video frame of the first sequence of video frames; and the one or more interpolated video frames are output at respective frame display times between the frame display time of the final output video frame of the first sequence of video frames and the frame display time of the first output video frame of the second sequence of video frames.
However, Lobete focuses on making moving images look smoother through frame interpolation, which relates to Osman because Osman tries to reduce latency which leads to smoother transitions during gameplay. Lobete teaches the first output video frame of the second sequence of video frames is the video frame output at two or more frame display times after a frame display time of the final output video frame of the first sequence of video frames; and the one or more interpolated video frames are output at respective frame display times between the frame display time of the final output video frame of the first sequence of video frames and the frame display time of the first output video frame of the second sequence of video frames (this structure happens naturally through the use of video frame interpolation as shown in Fig 5 where rendered frames are 1, 2, 3, 4, and 5 and interpolated frames are at 1.5, 2.5, 3.5, and 4.5 as shown in Fig 5 and the dashes represent the timestamps of rendered frames in ¶36, for example a first output video frame can be represented by 1 and final output frame can be represented by 2 and their interpolated frame will be 1.5 as shown in Fig 5 will multiple frame display times as timestamps between them).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Osman to implement the teachings of Lobete because by adding frame display times in the form of timestamps allows the predictive engine to synchronize the frames with the interpolated frames perfectly. It ensures they all these frames are synchronized properly in the same exact timeline.
Claims 7, 8, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Osman in view of Liu as applied to claims 1, 6, 10, and 15 above, and further in view of Sanders et al. (US 20210001216 A1; hereinafter Sanders).
Regarding claims 7 and 16, Osman discloses wherein the predicted in-game action of the user is determined by inputting the video game data to a machine learning model (AI prediction engine is included to determine a predicted state; ¶76). However, Osman does not explicitly disclose training the machine learning model using the past video game data and past in-game actions of the one or more second users.
However, Sanders focuses on generating video frames for video games with a prediction model, which relates to Osman because they both have a prediction model for generating frames. Sanders teaches training the machine learning model using the past video game data and past in-game actions of the one or more second users (prediction model is trained to synthesize video frames based on previous history and received player inputs; ¶27).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Osman to implement the teachings of Sanders for the benefit of having a more accurate prediction. The best way to make accurate predictions is to train the model based on the user history and inputs in order to have more accurate predictions, otherwise the system will just be guessing or learning as it goes.
Regarding claim 8, Osman discloses wherein the machine learning model is executed by a second data processing apparatus and the circuitry is configured to transmit the video game data to the second data processing apparatus (there are multiple processing devices, such as servers and client devices that transmit video and audio data in ¶37-38 and each of the distributed game engines 302 and distributed prediction engine 304 include a machine learning engine in ¶106) and, in response, receive the data indicating the predicted in-game action of the user from the second data processing apparatus (machine learning model is trained to predict one or more subsequent frames based on previously rendered frames and/or previously received user input in ¶27 and these all these processing devices are in constant communication with each other to transmit data as shown in Fig 1).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Osman in view of Liu in further view of Sanders as applied to claims 7 and 8 above, and further in view of Bleasdale-Shepherd et al. (US 20210146241 A1; hereinafter Bleasdale).
Regarding claim 9, Osman does not disclose if the data indicating the instructed in-game action of the user is received before the data indicating the predicted in-game action of the user, the circuitry is configured to begin output of the second sequence of video frames without beginning output of the first sequence of video frames.
However, Bleasdale focuses on using machine learning to predict what a player will do with a game controller, which relates to Osman because they are both using machine learning to help predict the next action of a player/user. Bleasdale teaches if the data indicating the instructed in-game action of the user is received before the data indicating the predicted in-game action of the user (when criteria is not met, predicted game control data is not generated, which means that the system will wait for actual game control data from user; ¶81), the circuitry is configured to begin output of the second sequence of video frames without beginning output of the first sequence of video frames (because the criteria is not met, predicted game control data will not be generated, and will instead wait for actual user control data to be generated; ¶81).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Osman to implement the teachings of Bleasdale because it allows for the system to output the correct frames as soon as they have them. The purpose of the prediction is to predict before the user makes an input but if the inputs have already been received then there is no point to make a prediction and the system will just output the input from the user.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSE ANGELES whose telephone number is (703)756-5338. The examiner can normally be reached Mon-Thu 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Dmitry Suhol can be reached at (571) 272-4430. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOSE ANGELES/ Examiner, Art Unit 3715
/Jay Trent Liddle/ Primary Examiner, Art Unit 3715
Prosecution Insights