Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on July 27th, 2026 has been entered.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4, 5, 7, 9, 12, 13, 15, 16, 18, 19, 21, and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Beltran et al. (US20200159644, Beltran hereinafter) in view of Sirianni et al. (US10949338, Sirianni hereinafter).
Regarding claim 1, Beltran discloses: A system, comprising: one or more processors; and one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations (Beltran, see paragraph [0086], “…Device 1400 includes a central processing unit (CPU) 1402 for running software applications and optionally an operating system.”), (Beltran, see paragraph [0087], “…Storage 1406 provides non-volatile storage and other computer readable media for applications and data and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other optical storage devices, as well as signal transmission and storage media…”) comprising:
receiving video game execution data of a video game (Beltran, see paragraph [0029], “…a set of raw test data 212 is produced as a result of the game tester 101 interacting with the video game application 210 during the video game session 201.”), (Beltran, see paragraph [0044], “…When bugs are identified, player-generated snapshot files 514 pertaining to those player-generated bugs are recorded in the snapshot database 502…”) including:
a rendered output of the video game generated during the execution of the video game (Beltran, see paragraph [0031], “As the game tester 101 interacts with the video game application 201 for uncovering bugs 106, 108, and 110, a video component 218 is generated by the system 204 as instructed by the video game application 210. The video component 218 may include both an audio component and a video component in the form of a sequence of video frames…”); and
telemetry data of the video game generated during the execution of the video game (Beltran, see paragraph [0031], “…as the game tester 101 interacts with the video game application 210 various network and system properties associated with the video game session 201 are captured in real time or near real time in network data 220 and system data 222, respectively, as a part of the raw test data 212… system data 222 may include various parameters associated with the system 204 or testing device 100 that executes the video game application 210, including a CPU clock, GPU clock, memory usage, frame rate, resolution, and other system properties…”); and
configuring a machine learning (ML) model to at least one of (Beltran, see paragraph [0045], “…The bug classification model 532 provides statistically relevant combinations, permutations, and time-dependent sequences of control inputs that are predictive of causing the player-generated bug or a newly identified bug…”), (Beltran, see paragraph [0044], “…The player-generated snapshot files 514, including the control inputs 516 are inputted into the machine learning module 506 via an application interface 522. In some embodiments, the player-generated snapshot files 514 may be used as a training data set 528 or ground truth…”), (Beltran, see paragraph [0033], “…Each snapshot file 232 is contemplated to be associated with a particular bug and include a portion of the sequence of control inputs 214, the game state data 216, the video component 218, the network data 220, the system data 222, the seed data 224 and the portion of the bug log 226 associated with the bug…”).
Beltran does not appear to distinctly disclose:
textual data generated by instrumented code of the video game during execution of the video game,
the instrumented code including code for one or more of: print debugging, logging, or tracing,
the textual data indicating at least one of: where program execution is in video game code or code execution paths of the video game; and
However, Sirianni discloses:
textual data generated by instrumented code of the video game during execution of the video game (Sirianni, see col 9 lines 24-28, “…Instrumentation 104 stores, into a path log of a bug detection database (e.g., data store 106), an indication of each line of the source code encountered during the respective execution of the application…”), (Sirianni, see col 13 lines 30-33, “…the techniques described herein may modify AFLs path detection that inserts logging instructions into the target binary on compilation.”), (Sirianni, see col 15 line 63-col 16 line 2, “One other potential market could be game applications. Such applications are computationally intensive and typically written with a very fast development schedule. The techniques described herein may be perfectly positioned to support game developers, since these techniques may increase productivity and decrease the expertise required to implement the optimizations.”),
the instrumented code including code for one or more of: (Sirianni, see col 13 lines 30-33, “…the techniques described herein may modify AFLs path detection that inserts logging instructions into the target binary on compilation.”), (Sirianni, see col 10 lines 45-46, “The software under test 103's built-in logging (e.g., verbose mode) may also be utilized.”),
the textual data indicating at least one of: (Sirianni, see col 9 lines 24-28, “…Instrumentation 104 stores, into a path log of a bug detection database (e.g., data store 106), an indication of each line of the source code encountered during the respective execution of the application…”), (Sirianni, see col 11 lines 23-28, “…A third option includes using n-grams of sequential program steps as features, where a given number of code lines that executed before and after the current line being modeled are recorded in order to form an n-gram representing the contextual code execution path…”), (Sirianni, see col 15 line 63-col 16 line 2); and
configuring a machine learning (ML) model to at least one of detect (Sirianni, see col 7 lines 59-61, “…The inputs to the bug detector 107 are, typically, the outputs from testing, such as logs and recorded instrumentation…”), (Sirianni, see col 12 lines 9-14, “Machine learning technology for bug detector 107 may include one-class linear support vector machines (SVMs). These are capable of unsupervised learning, meaning that it does not require class labels for the training data, and instead learns a decision boundary describing all the training datapoints, treated as a single class…”), (Sirianni, see col 15 line 63-col 16 line 2).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified a system for testing games as taught by Beltran to include textual data generated by instrumented code during execution, the instrumented code including code for one or more of: logging, the textual data indicating at least one of: code execution paths, and configuring a machine learning (ML) model to at least one of detect a type of events in the execution using at least the data generated by the instrumented code as training data for the ML model as taught by Sirianni for the result of allowing a detected problem to be traced back to the code that caused it.
Regarding claim 4, Beltran discloses:
wherein the configuring the ML model includes configuring the ML model to output changes to one or more settings of the video game based on at least one of a predicted event of the type of events (Beltran, see paragraph [0046], “…the machine learning module 506 may also provide a set of machine-learned variances that instruct the variance implementation module 536 on how to introduce variance during the automated gaming sessions. In other embodiments, the developer may choose to load-test the video game application by introducing variances via the variance implementation module. For example, the developer is envisioned to be enabled to specify a frame rate, resolution, CPU clock and usage, GPU clock and usage, and memory usage parameters to be simulated during execution of the automated video game sessions.…”), (Beltran, see paragraph [0051], “…similar to generating machine-learned control inputs 530 based upon the bug classification model 532, the machine learning module 506 may also generate machine-learned variances 503 that are predictively causative of the bug…”), (Beltran, see paragraph [0075], “…The results are fed back into the machine learning module 504 to update the bug classification model and to generate a new set of machine-learned control inputs 906…”).
Regarding claim 5, Beltran discloses:
wherein the execution is a first execution, and wherein the second video game execution data is output during a second execution of the video game (Beltran, see paragraph [0046], “…The automated game testing module 504 is configured to spin up a plurality of game sessions to test the sets of machine-learned control inputs 514 for the occurrence of a bug or not…”), (Beltran, see paragraph [0049], “The results of the automated sessions, whether reproducing the bug or not, are recorded as machine-generated snapshot files 530…”) and
the ML model is configured to output the changes to the one or more settings of the video game during the second execution of the video game (Beltran, see paragraph [0051], “…The automated game testing module 504 may be implemented via the variance implementation module 536 such that system and network conditions specified by the machine-learned variances 503 are simulated, mimicked, or reproduced during execution of the automated game sessions…”).
Regarding claim 7, Beltran discloses:
a gameplay scenario associated with the event (Beltran, see paragraph [0055], “… From these, rules 548, categories 550, game scenarios 552, and system scenarios 554 may be constructed…”), (Beltran, see paragraph [0056], “…For example, there may be a game scenario 552 in which an AI character falls through the world when the character attempts to jump…”).
Beltran does not appear to distinctly disclose:
wherein the configuring the ML model includes configuring the ML model to output data for an event that is at least one of a predicted event of the type of events or a detected event of the type of events in second video game execution data,
the data output by the ML model including at least one of: a code execution path associated with the event; and
a build or version of the video game in which the event does not occur.
However, Sirianni discloses:
wherein the configuring the ML model includes configuring the ML model to output data for an event that is at least one of (Sirianni, see col 6 lines 37-39, “…The bug detector produces a list of lines associated with the anomaly and gives context into the state of the program at the time…”), (Sirianni, see col 9 lines 34-38, “When the different executions are completed, bug detector 107 analyzes, using a machine learning model, the path logs and the output logs stored in the bug detection database to identify an abnormality indicative of a potential bug in the source code…”), (Sirianni, see col 15 line 63-col 16 line 2, “One other potential market could be game applications. Such applications are computationally intensive and typically written with a very fast development schedule. The techniques described herein may be perfectly positioned to support game developers, since these techniques may increase productivity and decrease the expertise required to implement the optimizations.”),
the data output by the ML model including at least one of: a code execution path associated with the event (Sirianni, see col 6 lines 37-39, “…The bug detector produces a list of lines associated with the anomaly and gives context into the state of the program at the time…”), (Sirianni, see col 9 lines 24-28, “…Instrumentation 104 stores, into a path log of a bug detection database (e.g., data store 106), an indication of each line of the source code encountered during the respective execution of the application…”), (Sirianni, see col 11 lines 23-28, “…A third option includes using n-grams of sequential program steps as features, where a given number of code lines that executed before and after the current line being modeled are recorded in order to form an n-gram representing the contextual code execution path…”); or
a build or version of the video game in which the event does not occur (Sirianni, see col 9 lines 56-65, “Bug detector 107 may determine whether a particular identified potential bug is still present in a later iteration of software or if it has been fixed… The visualization interface tracks detected bugs between software versions to automatically determine fixed, unfixed, and new bugs.”), (Sirianni, see col 15 line 63-col 16 line 2).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified a system for testing games as taught by Beltran to include wherein the configuring the ML model includes configuring the ML model to output data for an event that is a detected event of the type of events in second execution data, the data output by the ML model including at least one of: a code execution path associated with the event or a build or version in which the event does not occur as taught by Sirianni for the result of showing developers which code contains the problem in the event that a problem is detected or predicted.
Regarding claim 9, Beltran discloses: A computer-implemented method comprising:
receiving video game execution data of a video game (Beltran, see paragraph [0029], “…a set of raw test data 212 is produced as a result of the game tester 101 interacting with the video game application 210 during the video game session 201.”), (Beltran, see paragraph [0044], “…When bugs are identified, player-generated snapshot files 514 pertaining to those player-generated bugs are recorded in the snapshot database 502…”) including:
a rendered output of the video game generated during the execution of the video game (Beltran, see paragraph [0031], “As the game tester 101 interacts with the video game application 201 for uncovering bugs 106, 108, and 110, a video component 218 is generated by the system 204 as instructed by the video game application 210. The video component 218 may include both an audio component and a video component in the form of a sequence of video frames…”); and
telemetry data of the video game generated during the execution of the video game (Beltran, see paragraph [0031], “…as the game tester 101 interacts with the video game application 210 various network and system properties associated with the video game session 201 are captured in real time or near real time in network data 220 and system data 222, respectively, as a part of the raw test data 212… system data 222 may include various parameters associated with the system 204 or testing device 100 that executes the video game application 210, including a CPU clock, GPU clock, memory usage, frame rate, resolution, and other system properties…”); and
configuring a machine learning (ML) model to at least one of (Beltran, see paragraph [0045], “…The bug classification model 532 provides statistically relevant combinations, permutations, and time-dependent sequences of control inputs that are predictive of causing the player-generated bug or a newly identified bug…”), (Beltran, see paragraph [0044], “…The player-generated snapshot files 514, including the control inputs 516 are inputted into the machine learning module 506 via an application interface 522. In some embodiments, the player-generated snapshot files 514 may be used as a training data set 528 or ground truth…”), (Beltran, see paragraph [0033], “…Each snapshot file 232 is contemplated to be associated with a particular bug and include a portion of the sequence of control inputs 214, the game state data 216, the video component 218, the network data 220, the system data 222, the seed data 224 and the portion of the bug log 226 associated with the bug…”).
Beltran does not appear to distinctly disclose:
textual data generated by instrumented code of the video game during execution of the video game,
the instrumented code including code for one or more of: print debugging, logging, or tracing,
the textual data indicating at least one of: where program execution is in video game code or code execution paths of the video game; and
However, Sirianni discloses:
textual data generated by instrumented code of the video game during execution of the video game (Sirianni, see col 9 lines 24-28, “…Instrumentation 104 stores, into a path log of a bug detection database (e.g., data store 106), an indication of each line of the source code encountered during the respective execution of the application…”), (Sirianni, see col 13 lines 30-33, “…the techniques described herein may modify AFLs path detection that inserts logging instructions into the target binary on compilation.”), (Sirianni, see col 15 line 63-col 16 line 2, “One other potential market could be game applications. Such applications are computationally intensive and typically written with a very fast development schedule. The techniques described herein may be perfectly positioned to support game developers, since these techniques may increase productivity and decrease the expertise required to implement the optimizations.”),
the instrumented code including code for one or more of: (Sirianni, see col 13 lines 30-33, “…the techniques described herein may modify AFLs path detection that inserts logging instructions into the target binary on compilation.”), (Sirianni, see col 10 lines 45-46, “The software under test 103's built-in logging (e.g., verbose mode) may also be utilized.”),
the textual data indicating at least one of: (Sirianni, see col 9 lines 24-28, “…Instrumentation 104 stores, into a path log of a bug detection database (e.g., data store 106), an indication of each line of the source code encountered during the respective execution of the application…”), (Sirianni, see col 11 lines 23-28, “…A third option includes using n-grams of sequential program steps as features, where a given number of code lines that executed before and after the current line being modeled are recorded in order to form an n-gram representing the contextual code execution path…”), (Sirianni, see col 15 line 63-col 16 line 2); and
configuring a machine learning (ML) model to at least one of detect or (Sirianni, see col 7 lines 59-61, “…The inputs to the bug detector 107 are, typically, the outputs from testing, such as logs and recorded instrumentation…”), (Sirianni, see col 12 lines 9-14, “Machine learning technology for bug detector 107 may include one-class linear support vector machines (SVMs). These are capable of unsupervised learning, meaning that it does not require class labels for the training data, and instead learns a decision boundary describing all the training datapoints, treated as a single class…”), (Sirianni, see col 15 line 63-col 16 line 2).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified a system for testing games as taught by Beltran to include textual data generated by instrumented code during execution, the instrumented code including code for one or more of: logging, the textual data indicating at least one of: code execution paths, and configuring a machine learning (ML) model to at least one of detect a type of events in the execution using at least the data generated by the instrumented code as training data for the ML model as taught by Sirianni for the result of allowing a detected problem to be traced back to the code that caused it.
Regarding claim 12, Beltran discloses:
wherein the configuring the ML model includes configuring the ML model to output changes to one or more settings of the video game based on at least one of a predicted event of the type of events (Beltran, see paragraph [0046], “…the machine learning module 506 may also provide a set of machine-learned variances that instruct the variance implementation module 536 on how to introduce variance during the automated gaming sessions. In other embodiments, the developer may choose to load-test the video game application by introducing variances via the variance implementation module. For example, the developer is envisioned to be enabled to specify a frame rate, resolution, CPU clock and usage, GPU clock and usage, and memory usage parameters to be simulated during execution of the automated video game sessions.…”), (Beltran, see paragraph [0051], “…similar to generating machine-learned control inputs 530 based upon the bug classification model 532, the machine learning module 506 may also generate machine-learned variances 503 that are predictively causative of the bug…”), (Beltran, see paragraph [0075], “…The results are fed back into the machine learning module 504 to update the bug classification model and to generate a new set of machine-learned control inputs 906…”).
Regarding claim 13, Beltran discloses:
wherein the execution is a first execution, and wherein the second video game execution data is output during a second execution of the video game (Beltran, see paragraph [0046], “…The automated game testing module 504 is configured to spin up a plurality of game sessions to test the sets of machine-learned control inputs 514 for the occurrence of a bug or not…”), (Beltran, see paragraph [0049], “The results of the automated sessions, whether reproducing the bug or not, are recorded as machine-generated snapshot files 530…”) and
the ML model is configured to output the changes to the one or more settings of the video game during the second execution of the video game (Beltran, see paragraph [0051], “…The automated game testing module 504 may be implemented via the variance implementation module 536 such that system and network conditions specified by the machine-learned variances 503 are simulated, mimicked, or reproduced during execution of the automated game sessions…”).
Regarding claim 15, Beltran discloses: One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations (Beltran, see paragraph [0086], “…Device 1400 includes a central processing unit (CPU) 1402 for running software applications and optionally an operating system.”), (Beltran, see paragraph [0087], “…Storage 1406 provides non-volatile storage and other computer readable media for applications and data and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other optical storage devices, as well as signal transmission and storage media…”) comprising:
receiving video game execution data of a video game (Beltran, see paragraph [0029], “…a set of raw test data 212 is produced as a result of the game tester 101 interacting with the video game application 210 during the video game session 201.”), (Beltran, see paragraph [0044], “…When bugs are identified, player-generated snapshot files 514 pertaining to those player-generated bugs are recorded in the snapshot database 502…”) including:
a rendered output of the video game generated during the execution of the video game (Beltran, see paragraph [0031], “As the game tester 101 interacts with the video game application 201 for uncovering bugs 106, 108, and 110, a video component 218 is generated by the system 204 as instructed by the video game application 210. The video component 218 may include both an audio component and a video component in the form of a sequence of video frames…”); and
telemetry data of the video game generated during the execution of the video game (Beltran, see paragraph [0031], “…as the game tester 101 interacts with the video game application 210 various network and system properties associated with the video game session 201 are captured in real time or near real time in network data 220 and system data 222, respectively, as a part of the raw test data 212… system data 222 may include various parameters associated with the system 204 or testing device 100 that executes the video game application 210, including a CPU clock, GPU clock, memory usage, frame rate, resolution, and other system properties…”);
inputting (Beltran, see paragraph [0044], “…The player-generated snapshot files 514, including the control inputs 516 are inputted into the machine learning module 506 via an application interface 522. In some embodiments, the player-generated snapshot files 514 may be used as a training data set 528 or ground truth…”), (Beltran, see paragraph [0033], “…Each snapshot file 232 is contemplated to be associated with a particular bug and include a portion of the sequence of control inputs 214, the game state data 216, the video component 218, the network data 220, the system data 222, the seed data 224 and the portion of the bug log 226 associated with the bug…”); and
receiving, from the ML model, data associated with an event of the type of events in the execution of the video game (Beltran, see paragraph [0045], “…The bug classification model 532 provides statistically relevant combinations, permutations, and time-dependent sequences of control inputs that are predictive of causing the player-generated bug or a newly identified bug…”).
Beltran does not appear to distinctly disclose:
textual data generated by instrumented code of the video game during an execution of the video game,
the instrumented code including code for one or more of: print debugging, logging, or tracing,
the textual data indicating at least one of: where program execution is in video game code or code execution paths of the video game; and
However, Sirianni discloses:
textual data generated by instrumented code of the video game during an execution of the video game (Sirianni, see col 9 lines 24-28, “…Instrumentation 104 stores, into a path log of a bug detection database (e.g., data store 106), an indication of each line of the source code encountered during the respective execution of the application…”), (Sirianni, see col 13 lines 30-33, “…the techniques described herein may modify AFLs path detection that inserts logging instructions into the target binary on compilation.”), (Sirianni, see col 15 line 63-col 16 line 2, “One other potential market could be game applications. Such applications are computationally intensive and typically written with a very fast development schedule. The techniques described herein may be perfectly positioned to support game developers, since these techniques may increase productivity and decrease the expertise required to implement the optimizations.”),
the instrumented code including code for one or more of: (Sirianni, see col 13 lines 30-33, “…the techniques described herein may modify AFLs path detection that inserts logging instructions into the target binary on compilation.”), (Sirianni, see col 10 lines 45-46, “The software under test 103's built-in logging (e.g., verbose mode) may also be utilized.”),
the textual data indicating at least one of: (Sirianni, see col 9 lines 24-28, “…Instrumentation 104 stores, into a path log of a bug detection database (e.g., data store 106), an indication of each line of the source code encountered during the respective execution of the application…”), (Sirianni, see col 11 lines 23-28, “…A third option includes using n-grams of sequential program steps as features, where a given number of code lines that executed before and after the current line being modeled are recorded in order to form an n-gram representing the contextual code execution path…”), (Sirianni, see col 15 line 63-col 16 line 2);
inputting the data generated by the instrumented code, (Sirianni, see col 7 lines 59-61, “…The inputs to the bug detector 107 are, typically, the outputs from testing, such as logs and recorded instrumentation…”), (Sirianni, see col 15 line 63-col 16 line 2); and
receiving, from the ML model, data associated with an event of the type of events in the execution of the video game (Sirianni, see col 6 lines 37-39, “…The bug detector produces a list of lines associated with the anomaly and gives context into the state of the program at the time…”), (Sirianni, see col 15 line 63-col 16 line 2).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified a system for testing games as taught by Beltran to include textual data generated by instrumented code during an execution, the instrumented code including code for one or more of: logging, the textual data indicating at least one of: code execution paths, inputting the data generated by the instrumented code into a machine learning (ML) model configured to at least one of detect events of a type of events in the execution, and receiving, from the ML model, data associated with an event of the type of events in the execution as taught by Sirianni for the result of allowing a detected problem to be traced back to the code that caused it.
Regarding claim 16, Beltran discloses:
a gameplay scenario associated with the event (Beltran, see paragraph [0056], “…For example, there may be a game scenario 552 in which an AI character falls through the world when the character attempts to jump…”);
Beltran does not appear to distinctly disclose:
wherein the data associated with the event includes at least one of: a code execution path associated with the event; or
a build or version of the video game in which the event does not occur.
However, Sirianni discloses:
wherein the data associated with the event includes at least one of: a code execution path associated with the event (Sirianni, see col 6 lines 37-39, “…The bug detector produces a list of lines associated with the anomaly and gives context into the state of the program at the time…”); or
a build or version of the video game in which the event does not occur (Sirianni, see col 9 lines 56-65, “Bug detector 107 may determine whether a particular identified potential bug is still present in a later iteration of software or if it has been fixed… The visualization interface tracks detected bugs between software versions to automatically determine fixed, unfixed, and new bugs.”), (Sirianni, see col 15 line 63-col 16 line 2, “One other potential market could be game applications. Such applications are computationally intensive and typically written with a very fast development schedule. The techniques described herein may be perfectly positioned to support game developers, since these techniques may increase productivity and decrease the expertise required to implement the optimizations.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified a system for testing games as taught by Beltran to include wherein the data associated with the event includes at least one of: a code execution path associated with the event or a build or version in which the event does not occur as taught by Sirianni for the result of showing developers which code contains the problem in the event that a problem is detected or predicted.
Regarding claim 18, Beltran discloses:
wherein the data associated with the event of the type of events in the execution of the video game includes changes to one or more settings of the video game based on the event (Beltran, see paragraph [0045], “…The bug classification model 532 provides statistically relevant combinations, permutations, and time-dependent sequences of control inputs that are predictive of causing the player-generated bug or a newly identified bug…”), (Beltran, see paragraph [0051], “…similar to generating machine-learned control inputs 530 based upon the bug classification model 532, the machine learning module 506 may also generate machine-learned variances 503 that are predictively causative of the bug…”), (Beltran, see paragraph [0046], “…the machine learning module 506 may also provide a set of machine-learned variances that instruct the variance implementation module 536 on how to introduce variance during the automated gaming sessions. In other embodiments, the developer may choose to load-test the video game application by introducing variances via the variance implementation module. For example, the developer is envisioned to be enabled to specify a frame rate, resolution, CPU clock and usage, GPU clock and usage, and memory usage parameters to be simulated during execution of the automated video game sessions.…”).
Regarding claim 19, Beltran discloses:
wherein the event is a predicted event in the execution of the video game (Beltran, see paragraph [0051], “…similar to generating machine-learned control inputs 530 based upon the bug classification model 532, the machine learning module 506 may also generate machine-learned variances 503 that are predictively causative of the bug…”), the operations further comprising:
changing the settings of the video game based on the changes to one or more settings of the video game prior to an occurrence of the predicted event (Beltran, see paragraph [0060], “…When the automated game sessions are spun up by the automated game testing module 504, the operating system 542 may implement the set of conditions specified by the variances packages 606 during execution of the automated game sessions.”).
Regarding claim 21, Beltran does not appear to distinctly disclose:
wherein the textual data is output to at least one of: a debugging console, a screen, or a log file.
However, Sirianni discloses:
wherein the textual data is output to at least one of: a debugging console, a screen, or a log file (Sirianni, see col 9 lines 24-28, “…Instrumentation 104 stores, into a path log of a bug detection database (e.g., data store 106), an indication of each line of the source code encountered during the respective execution of the application…”), (Sirianni, see col 10 lines 33-36, “…Existing debugging or instrumentation libraries, such as the GNU debugger (GDB), may be used to output parts of the program's internal state during execution…”), (Sirianni, see col 19 lines 9-10, “…The right-hand view enables the developer to take a detailed look at the program state when the anomaly occurred.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified a system for testing games as taught by Beltran to include wherein the textual data is output to at least one of: a debugging console, a screen, or a log file as taught by Sirianni for the result of allowing the developer to see what the code is doing while the game is running.
Regarding claim 22, Beltran does not appear to distinctly disclose:
wherein the textual data indicates state data of the video game including values of at least one of: inputs, outputs, or intermediate variables.
However, Sirianni discloses:
wherein the textual data indicates state data of the video game including values of at least one of: inputs, outputs, or intermediate variables (Sirianni, see col 9 lines 24-28, “…Instrumentation 104 stores, into a path log of a bug detection database (e.g., data store 106), an indication of each line of the source code encountered during the respective execution of the application…”), (Sirianni, see col 10 lines 31-33, “For the instrumentation component 104, a framework of tools may be leveraged for capturing the entire program state of each program execution…”), (Sirianni, see col 9 lines 28-33, “…Instrumentation 104 also stores, into an output log of the bug detection database (e.g., data store 106), an indication of each output object encountered during the respective execution of the application. Each output object comprises a local variable with a value dependent on one or more of the inputs from the unique set of one or more inputs.”), (Sirianni, see col 15 line 63-col 16 line 2, “One other potential market could be game applications. Such applications are computationally intensive and typically written with a very fast development schedule. The techniques described herein may be perfectly positioned to support game developers, since these techniques may increase productivity and decrease the expertise required to implement the optimizations.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified a system for testing games as taught by Beltran to include wherein the textual data indicates state data including values of at least one of: inputs, outputs, or intermediate variables as taught by Sirianni for the result of allowing the developer to see any problems that occurred without running the game again.
Claims 2, 6, 10, 14, 17, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Beltran and Sirianni as applied to claims 1, 9, and 15 above, and further in view of Lucas et al. (US20170266568, Lucas hereinafter).
Regarding claim 2, Beltran as modified does not appear to distinctly disclose:
wherein at least a portion of the telemetry data is overlayed into the rendered output.
However, Lucas discloses: wherein at least a portion of the telemetry data is overlayed into the rendered output (Lucas, see paragraph [0003]: “Associating telemetry data with video data of the gameplay session can ameliorate this difficulty. During the gameplay session, both telemetry data and video data are recorded. The gameplay session can have a session identifier (session ID). The telemetry data and video data of the gameplay session can both be linked to the same session ID.”);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified a system for testing games as taught by Beltran to include wherein at least a portion of the telemetry data is overlayed into the rendered output as taught by Lucas for the result of allowing developers to make relevant conclusions based on the data.
Regarding claim 6, Beltran as modified does not appear to distinctly disclose:
wherein an event of the type of events is one of: a frame rate drop; a processing load above a first threshold; a pattern in the processing load; a network connectivity loss; or a storage input/output load above a second threshold.
However, Lucas discloses: wherein an event of the type of events is one of: (Lucas, see paragraph [0017]: “…The system described herein can identify an event (for example, a bug or a crash) automatically or based on the user's input (for example, a bug report). The system can associate the telemetry data and the video data with the event based on the session ID and the timestamp of the event…”);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified a system for testing games as taught by Beltran to include wherein an event of the type of events is one of a network connectivity loss as taught by Lucas for the result of locating performance related problems while the game is running.
Regarding claim 10, Beltran as modified does not appear to distinctly disclose:
wherein at least a portion of the telemetry data is overlayed into the rendered output.
However, Lucas discloses: wherein at least a portion of the telemetry data is overlayed into the rendered output (Lucas, see paragraph [0003]: “Associating telemetry data with video data of the gameplay session can ameliorate this difficulty. During the gameplay session, both telemetry data and video data are recorded. The gameplay session can have a session identifier (session ID). The telemetry data and video data of the gameplay session can both be linked to the same session ID.”);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified a system for testing games as taught by Beltran to include wherein at least a portion of the telemetry data is overlayed into the rendered output as taught by Lucas for the result of allowing developers to make relevant conclusions based on the data.
Regarding claim 14, Beltran as modified does not appear to distinctly disclose:
wherein an event of the type of events is one of: a frame rate drop; a processing load above a first threshold; a pattern in the processing load; a network connectivity loss; or a storage input/output load above a second threshold.
However, Lucas discloses: wherein an event of the type of events is one of: (Lucas, see paragraph [0017]: “…The system described herein can identify an event (for example, a bug or a crash) automatically or based on the user's input (for example, a bug report). The system can associate the telemetry data and the video data with the event based on the session ID and the timestamp of the event…”);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified a system for testing games as taught by Beltran to include wherein an event of the type of events is one of a network connectivity loss as taught by Lucas for the result of locating performance related problems while the game is running.
Regarding claim 17, Beltran as modified does not appear to distinctly disclose:
wherein an event of the type of events is one of: a frame rate drop; a processing load above a first threshold; a pattern in the processing load; a network connectivity loss; or a storage input/output load above a second threshold.
However, Lucas discloses: wherein an event of the type of events is one of: (Lucas, see paragraph [0017]: “…The system described herein can identify an event (for example, a bug or a crash) automatically or based on the user's input (for example, a bug report). The system can associate the telemetry data and the video data with the event based on the session ID and the timestamp of the event…”);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified a system for testing games as taught by Beltran to include wherein an event of the type of events is one of a network connectivity loss as taught by Lucas for the result of locating performance related problems while the game is running.
Regarding claim 20, Beltran as modified does not appear to distinctly disclose:
wherein at least a portion of the telemetry data is overlayed into the rendered output.
However, Lucas discloses: wherein at least a portion of the telemetry data is overlayed into the rendered output (Lucas, see paragraph [0003]: “Associating telemetry data with video data of the gameplay session can ameliorate this difficulty. During the gameplay session, both telemetry data and video data are recorded. The gameplay session can have a session identifier (session ID). The telemetry data and video data of the gameplay session can both be linked to the same session ID.”);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified a system for testing games as taught by Beltran to include wherein at least a portion of the telemetry data is overlayed into the rendered output as taught by Lucas for the result of allowing developers to make relevant conclusions based on the data.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Beltran and Sirianni as applied to claims 1 above, and further in view of Dimitropoulos et al. (US10891219, Dimitropoulos hereinafter).
Regarding claim 8, Beltran as modified does not appear to distinctly disclose:
wherein the configuring the ML model includes configuring the ML model to detect or predict the type of events in the execution of the video game on second video game execution data of a second execution of the video game not including data generated by at least a portion of the instrumented code of the video game.
However, Dimitropoulos discloses:
wherein the configuring the ML model includes configuring the ML model to (Dimitropoulos, see col 13 lines 60-62, “…a second prediction model can be used to predict the likelihood that a failure will occur based on one or more inputs to the prediction model…”), (Dimitropoulos, see col 20 lines 20-21, “The chances can be determined using a model, for example, as described with respect to FIG. 3A…”) on second video game execution data of a second execution of the video game (Dimitropoulos, see col 19 lines 21-24, “…The new signature can include, for example, a current state, a history of previous inputs, previous states, and the like of an instance of code being executed by a test execution engine.”), (Dimitropoulos, see col 20 lines 41-48, “…test farms can execute a plurality of code simulations while a debug or logging feature is off. This can save processing power and memory. The simulations can periodically check if a state of the simulation is likely to fail. If it is determined that there is more than a minimum threshold chance of failure (such as 25%, 50%, 75%), then the debugging tools and logging tools can be turned on for the simulation that is likely to fail.”) not including data generated by at least a portion of the instrumented code of the video game (Dimitropoulos, see col 20 lines 41-48, “…test farms can execute a plurality of code simulations while a debug or logging feature is off. This can save processing power and memory. The simulations can periodically check if a state of the simulation is likely to fail. If it is determined that there is more than a minimum threshold chance of failure (such as 25%, 50%, 75%), then the debugging tools and logging tools can be turned on for the simulation that is likely to fail.”), (Dimitropoulos, see col 6 lines 4-9, “…By conditionally activating debug tools based on the video game signature, utilization of computational power, memory, and storage space can be reduced because the debug features do not need to constantly run and log activity, but instead run when the video game signature indicates a threshold chance of failure…”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified a system for testing games as taught by Beltran to include wherein the configuring the ML model includes configuring the ML model to predict the type of events in the execution of the video game on second video game execution data of a second execution of the video game not including data generated by at least a portion of the instrumented code of the video game as taught by Dimitropoulos for the result of logging only for runs in which a problem is expected to occur.
Response to Arguments
Applicant’s argument: Claims 1, 2, 3, 6, 7, 8, 9, 10, 11, 14, 15, 16, 17, and 20 would not have been obvious over Lucas in view of Goossen and Kennett.
Examiner’s response: The applicant’s argument is moot as new art is used to reject the claims.
Applicant’s argument: Claims 4, 5, 12, 13, and 18 would not have been obvious over Lucas in view of Goossen, Kennett, and Xu.
Examiner’s response: The applicant’s argument is moot as new art is used to reject the claims.
Applicant’s argument: Claim 19 would not have been obvious over Lucas in view of Goossen, Kennett, Xu, and Walters.
Examiner’s response: The applicant’s argument is moot as new art is used to reject the claim.
Conclusion
Any inquiry concerning this communication or earlier communications from the
examiner should be directed to Joshua Tran whose telephone number is (571)272-5460.
The examiner can normally be reached on M-F 9-5.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s
supervisor, Hyung Sough can be reached on (571)272-6799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOSHUA TRAN/
Examiner, Art Unit 2192
/S. Sough/SPE, Art Unit 2192
Prosecution Insights