CTFR 18/753,653 CTFR 79993 Detailed Action 07-03-01-aia AIA 07-03-01-r-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. See 35 U.S.C. § 100 (note). Art Rejections Anticipation 07-07-aia AIA 07-07 The following is a quotation of the appropriate paragraphs of 35 U.S.C. § 102 that form the basis for the rejections under this section made in this Office action: A person shall be entitled to a patent unless – 07-08-aia AIA (a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention. 07-12-aia AIA (a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention. 07-15 AIA Claim s 1, 5, 19, 20 and 21 are rejected under 35 U.S.C. § 102( a)(1 ) as being anticipated by KR 20240074553 A (published 28 May 2024) (“Hwang”) 1 , 2 . Claim 1 is drawn to “a computer implemented method.” The following table illustrates the correspondence between the claimed method and the Hwang reference. Claim 1 The Hwang Reference “1. A computer implemented method comprising: The Hwang reference similarly describes a method for automatically generating screen commentary using artificial intelligence. Hwang at p. 2, ¶ 2. “examining foreground voice data of a multimedia stream that includes a video stream data and an audio stream; “identifying in dependence on the examining an open time window that is absent of foreground voice data; Hwang’s method includes analyzing a video’s audio (i.e., foreground voice data) to identify voice sections and silent sections. Id. at p. 6, ¶ 5, FIG.3 (step 202). “processing, in dependence on the identifying, media stream data of multimedia stream; “generating, in dependence on the processing, a text string for deployment in the open time window, wherein the text string describes content of the video stream; After determining the length of each of the video’s silent sections, Hwang’s method processes a provided script associated with the video. Id. at p. 6, ¶¶ 6–7, FIG.3 (step 301). The script describes the scenes in the video. Id. at p. 3, ¶ 5. Hwang’s method splits the script into sections. Id. at p. 5, ¶ 4, FIG.3 (step 101). This results in a set of text strings describing the content corresponding to the silent sections. Id. Hwang further evaluates if each string will fit into identified silent sections. Id. at p. 6, ¶¶ 6–7, FIG.3 (step 301). This produces a set of text strings that are fit into each silence section, possibly with length modification. Id. at p. 6, ¶ 8 to p. 7, ¶ 4, FIG.3. In an alternative embodiment, Hwang considers the length of each silent section and generates a text string by editing each silent section’s corresponding text string to fit within the length of the silent section when synthesized as speech. Id. at p. 7, ¶ 6 to p. 8, ¶ 4. “converting the text string into a synthesized voice segment; and “adapting the audio stream data so that the synthesized voice segment is included in the audio stream and time bounded within the open time window.” Hwang’s method converts the text strings into synthetic speech using a text-to-speech (TTS) system and adds the synthetic speech to the video’s silent sections. Id. at p. 7, ¶ 4, p. 8, ¶ 4, FIGs.3, 6 (step 303, step 601). Table 1 For the foregoing reasons, the Hwang reference anticipates all limitations of the claim. Claim 5 depends on claim 1, and further requires the following: “wherein the processing media stream data of the multimedia stream includes processing media stream data to predict a time duration of the open time window, and wherein the generating is performed in dependence on the predicted time duration.” Hwang uses an AI system, like the Wave-U-Net algorithm, to automatically predict whether a section of audio contains voice or silence. Hwang at p. 6, ¶ 5, FIG.3 (step 202). For the foregoing reasons, the Hwang reference anticipates all limitations of the claim. Claim 21 depends on claim 1, and further requires the following: “wherein the generating, in dependence on the processing, the text string for deployment in the open time window includes “performing evaluating of text string data of the text string according to a broadcast emulation factor, “wherein the performing evaluating includes determining a degree to which a semantic meaning of the text string data emulates a target broadcast format.” The obviousness rejection of claim 6, incorporated herein, shows the obviousness of combining the teachings of Hwang and Couleaud to produce a method and system that uses machine learning models to automatically generate video commentary. Couleaud teaches and suggests generating a commentary script with a trained ontology model that will evaluate possible outputs and ultimately select a final text output that reflects a particular type of broadcast, such as a particular type of sports broadcast. Couleaud at ¶ 66. For the foregoing reasons, the Hwang reference anticipates all limitations of the claim. Claim 19 is drawn to “a system.” The following table illustrates the correspondence between the claimed system and the Hwang reference. Claim 19 The Hwang Reference “19. A system comprising: “a memory; “at least one processor in communication with the memory; and “program instructions executable by one or more processor via the memory to perform a method comprising: The Hwang reference similarly describes a system that includes a computerized server that implements a method for automatically generating screen commentary using artificial intelligence. Hwang at p. 2, ¶ 2. The server includes the claimed memory, processor and program instructions. Id. at p. 3, ¶ 8, p. 4, ¶ 1, FIG.2. “examining foreground voice data of a multimedia stream that includes a video stream data and an audio stream; “identifying in dependence on the examining an open time window that is absent of foreground voice data; Hwang’s method includes analyzing a video’s audio (i.e., foreground voice data) to identify voice sections and silent sections. Id. at p. 6, ¶ 5, FIG.3 (step 202). “processing, in dependence on the identifying, media stream data of multimedia stream… “generating, in dependence on the processing, a text string for deployment in the open time window, wherein the text string describes content of the video stream; After determining the length of each of the video’s silent sections, Hwang’s method processes a provided script associated with the video. Id. at p. 6, ¶¶ 6–7, FIG.3 (step 301). The script describes the scenes in the video. Id. at p. 3, ¶ 5. Hwang’s method splits the script into sections. Id. at p. 5, ¶ 4, FIG.3 (step 101). This results in a set of text strings describing the content corresponding to the silent sections. Id. Hwang further evaluates if each string will fit into identified silent sections. Id. at p. 6, ¶¶ 6–7, FIG.3 (step 301). This produces a set of text strings that are fit into each silence section, possibly with length modification. Id. at p. 6, ¶ 8 to p. 7, ¶ 4, FIG.3. In an alternative embodiment, Hwang considers the length of each silent section and generates a text string by editing each silent section’s corresponding text string to fit within the length of the silent section when synthesized as speech. Id. at p. 7, ¶ 6 to p. 8, ¶ 4. “ wherein the multimedia stream is a delayed instance of a live stream ; Hwang describes processing a previously recorded stream. Id. at p. 5, ¶ 4. “converting the text string into a synthesized voice segment; and “adapting the audio stream data so that the synthesized voice segment is included in the audio stream and time bounded within the open time window , wherein the adapting includes modifying the delayed instance of the live stream .” Hwang’s method converts the text strings into synthetic speech using a text-to-speech (TTS) system and adds the synthetic speech to the video’s silent sections. Id. at p. 7, ¶ 4, p. 8, ¶ 4, FIGs.3, 6 (step 303, step 601). Table 2 For the foregoing reasons, the Hwang reference anticipates all limitations of the claim. Claim 20 is drawn to “a computer program product.” The following table illustrates the correspondence between the claimed product and the Hwang reference. Claim 20 The Hwang Reference “20. A computer program product comprising: “a computer readable storage medium readable by one or more processing circuit and storing instructions for execution by one or more processor for performing a method comprising: The Hwang reference similarly describes a system that includes a computerized server that implements a method for automatically generating screen commentary using artificial intelligence. Hwang at p. 2, ¶ 2. The server includes the claimed computer readable storage medium readable by a processor and storing program instructions. Id. at p. 3, ¶ 8, p. 4, ¶ 1, FIG.2. “examining foreground voice data of a multimedia stream that includes a video stream data and an audio stream; “identifying in dependence on the examining an open time window that is absent of foreground voice data; Hwang’s method includes analyzing a video’s audio (i.e., foreground voice data) to identify voice sections and silent sections. Id. at p. 6, ¶ 5, FIG.3 (step 202). “processing, in dependence on the identifying, media stream data of multimedia stream; “generating, in dependence on the processing, a text string for deployment in the open time window, wherein the text string describes content of the video stream… After determining the length of each of the video’s silent sections, Hwang’s method processes a provided script associated with the video. Id. at p. 6, ¶¶ 6–7, FIG.3 (step 301). The script describes the scenes in the video. Id. at p. 3, ¶ 5. Hwang’s method splits the script into sections. Id. at p. 5, ¶ 4, FIG.3 (step 101). This results in a set of text strings describing the content corresponding to the silent sections. Id. Hwang further evaluates if each string will fit into identified silent sections. Id. at p. 6, ¶¶ 6–7, FIG.3 (step 301). This produces a set of text strings that are fit into each silence section, possibly with length modification. Id. at p. 6, ¶ 8 to p. 7, ¶ 4, FIG.3. In an alternative embodiment, Hwang considers the length of each silent section and generates a text string by editing each silent section’s corresponding text string to fit within the length of the silent section when synthesized as speech. Id. at p. 7, ¶ 6 to p. 8, ¶ 4. “ and is generated from processing of video data of the multimedia stream ; Hwang describes generating the text either manually or through AI machine vision processing. Id. at p.3, ¶ 5. “converting the text string into a synthesized voice segment; and “adapting the audio stream data so that the synthesized voice segment is included in the audio stream and time bounded within the open time window.” Hwang’s method converts the text strings into synthetic speech using a text-to-speech (TTS) system and adds the synthetic speech to the video’s silent sections. Id. at p. 7, ¶ 4, p. 8, ¶ 4, FIGs.3, 6 (step 303, step 601). Table 3 For the foregoing reasons, the Hwang reference anticipates all limitations of the claim. Obviousness 07-20-aia AIA The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claims 2, 3, 6–10 and 13, 14 and 16–18 are re jected under 35 U.S.C. § 103 as being unpatentable over th e combination of Hwang and US Patent Application Publication 2024/0406509 (filed 31 May 2023) (“Couleaud”). Cl aim 6 depends on claim 1, and further requires the following: “wherein the processing media stream data of the multimedia stream includes processing media stream data to predict a style of the open time window, and wherein the generating is performed in dependence on the predicted style.” The anticipation rejection of claim 1, incorporated herein, shows that the Hwang reference anticipates the method of that claim. Hwang’s described method similarly includes adding a synthesized voice that describes content of a video in identified silence intervals of the video. Hwang at p. 3, ¶ 4. Hwang’s method differs from this claim, however, because Hwang does not predict a style of an open time window in order to generate a text string. Rather, Hwang relies strictly on a user-provided script that is segmented and possibly edited to match the silence sections—though Hwang recognizes the possibility of automatically generating a script based on image and voice analysis. Id. at p. 3, ¶ 5. While this works well for pre-scripted content, like a movie or TV show, it is implausible for unscripted content like sports due to the large amount of manual work required to generate a script for such events. As alluded to by Hwang at p. 3 ¶ 5, workers in the field of video commentary have been developing commentary systems that can leverage machine vision and large language models to automatically analyze videos and produce a script for describing the videos. In particular, several workers have been developing automated soccer commentary systems. The Couleaud reference, for example, describes a method and system for producing sports commentary. Couleaud at ¶¶ 1–3. Couleaud’s system uses machine vision (e.g., deep learning) and speech analysis (e.g., LLM) to infer information about events occurring in a sports video and to produce an event description. Id. at ¶ 4. A virtual persona model processes the event description to create a synthesized commentary voice describing the event. Id. at ¶¶ 4–7. Each virtual persona model is customized to meet a user’s preferences and can reflect characteristics of a real sports commentator. Id. Couleaud’s method and system then outputs the custom, synthesized commentary voice as part of the sports video much in the same way that Hwang outputs a synthetic voice commentary. Id. at ¶ 12. Alternatively, Couleaud’s system simply generates a script. Id. The audio/text output is formed in a requested style, reflecting a particular commentator’s personality, method of delivery/style, tone, idiolect, personality, etc. as well as the nature of the event, such as the event’s urgency. Id. at ¶¶ 3, 5, 6, 9. Couleaud further teaches and suggests adding any desired type of commentary, including play-by-play commentary and color commentary. Id. at ¶ 13. For example, if an event includes narration, Couleaud determines if the time to the next event is of sufficient length. Id. at ¶¶ 78–93, FIG.6. If it is, Couleaud inserts additional commentary, such as color commentary. Id. One of ordinary skill would have recognized from the ubiquity of sports broadcasting in daily life that many sporting events already include some degree of commentary. Couleaud thus reasonably suggests automating at least some of the commentary, including adding any desired type of custom commentary that is otherwise missing. Id. at ¶¶ 110–115, FIG.9. For example, an existing play-by-play commentary may be enhanced by adding color commentary by a user’s favorite sports announcer that reacts to plays, an existing commentary and time between events. Id. Read in light of Hwang, Couleaud and Hwang reasonably suggest adding additional sports commentary to just the silent sections detected according to the methods of Hwang in order to enhance a sports broadcast with Couleaud’s custom, synthesized commentary voice that reflects a commentator’s style. For the foregoing reasons, the combination of the Hwang and the Couleaud references makes obvious all limitations of the claim. Claim 2 depends on claim 1, and further requires the following: “wherein the adapting includes modifying a delayed instance of the multimedia stream.” The obviousness rejection of claim 6, incorporated herein, shows the obviousness of combining the teachings of Hwang and Couleaud to produce a method and system that uses machine learning models to automatically generate video commentary. As explained in that rejection, the method and system would produce a text script describing events in a video based on the video’s images and audio. Hwang at p. 3, ¶ 5; Couleaud at ¶ 4. The text script would then be synthesized by a TTS module to create a custom synthetic voice in a user-preferred style of a commentator. Hwang at p. 7, ¶ 4. The custom, synthetic voice would be added specifically to silent sections that do not already contain commentary. See Hwang at p. 6, ¶¶ 6–7. In connection with that prior art combination, the Couleaud reference teaches and suggests modifying a live stream. Couleaud at ¶ 4. The live stream would inherently be delayed long enough for Couleaud’s processing to complete. Further, Hwang describes generating a text from a previously recorded stream. Hwang at p. 5, ¶ 4. For the foregoing reasons, the combination of the Hwang and the Couleaud references makes obvious all limitations of the claim. Claim 3 depends on claim 1, and further requires the following: “wherein the processing media stream data of the multimedia stream includes processing video data timestamped within the open time window.” The obviousness rejection of claim 6, incorporated herein, shows the obviousness of combining the teachings of Hwang and Couleaud to produce a method and system that uses machine learning models to automatically generate video commentary. As explained in that rejection, the method and system would produce a text script describing events in a video based on the video’s images and audio. Hwang at p. 3, ¶ 5; Couleaud at ¶ 4. The text script would then be synthesized by a TTS module to create a custom synthetic voice in a user-preferred style of a commentator. Hwang at p. 7, ¶ 4. The custom, synthetic voice would be added specifically to silent sections that do not already contain commentary. See Hwang at p. 6, ¶¶ 6–7. In connection with that prior art combination, the Hwang reference similarly uses time codes, or time stamps, to organize and refer to sections of video that can be commentated. Hwang at p. 5, ¶¶ 4, 6, p. 6, ¶¶ 5–7. Thus, the video in a silence interval processed according to Couleaud’s teachings in order to generate a script would be time coded. For the foregoing reasons, the combination of the Hwang and the Couleaud references makes obvious all limitations of the claim. Claim 7 depends on claim 1, and further requires the following: “wherein the processing media stream data of the multimedia stream includes processing media stream data to predict a semantic meaning of foreground voice segment data associated to the open time window, and “wherein the generating is performed in dependence on the predicted semantic meaning.” The obviousness rejection of claim 6, incorporated herein, shows the obviousness of combining the teachings of Hwang and Couleaud to produce a method and system that uses machine learning models to automatically generate video commentary. As explained in that rejection, the method and system would produce a text script describing events in a video based on the video’s images and audio. Hwang at p. 3, ¶ 5; Couleaud at ¶ 4. The text script would then be synthesized by a TTS module to create a custom synthetic voice in a user-preferred style of a commentator. Hwang at p. 7, ¶ 4. The custom, synthetic voice would be added specifically to silent sections that do not already contain commentary. See Hwang at p. 6, ¶¶ 6–7. In connection with that prior art combination, the Couleaud reference further teaches and suggests using deep learning models to analyze and predict (i.e., infer) a semantic meaning of foreground voice segments. For example, Couleaud describes inferring the urgency of an event and to infer if the event is important or exciting, such as scoring a goal in a game or if it is more mundane, such as a time out. Couleaud at ¶ 9. The commentary script and synthesized voice then reflect the inferred urgency (i.e., predicted semantic meaning) of the event. For the foregoing reasons, the combination of the Hwang and the Couleaud references makes obvious all limitations of the claim. Claim 8 depends on claim 1, and further requires the following: “wherein the generating, in dependence on the processing, the text string for deployment in the open time window includes performing evaluating of text string data of the text string according to time window fitment factor, “wherein the performing evaluating includes determining a degree to which a predicted time for voice synthesized rendering of the text string data matches a predicted duration of the open time window.” The obviousness rejection of claim 6, incorporated herein, shows the obviousness of combining the teachings of Hwang and Couleaud to produce a method and system that uses machine learning models to automatically generate video commentary. As explained in that rejection, the method and system would produce a text script describing events in a video based on the video’s images and audio. Hwang at p. 3, ¶ 5; Couleaud at ¶ 4. The text script would then be synthesized by a TTS module to create a custom synthetic voice in a user-preferred style of a commentator. Hwang at p. 7, ¶ 4. The custom, synthetic voice would be added specifically to silent sections that do not already contain commentary. See Hwang at p. 6, ¶¶ 6–7. In connection with that prior art combination, both Hwang and Couleaud teach and suggest evaluating whether a synthesized voice readout of a text string will fit in an amount of time available for commentary. Hwang at ¶¶ 61–70, FIGs.3, 6; Couleaud at ¶¶ 78–80, FIG.6. For the foregoing reasons, the combination of the Hwang and the Couleaud references makes obvious all limitations of the claim. Claim 9 depends on claim 1, and further requires the following: “wherein the generating, in dependence on the processing, the text string for deployment in the open time window includes performing evaluating of text string data of the text string according to a style matching factor, “wherein the performing evaluating includes determining a degree to which an extracted sentiment of the text string data extracted by subjecting the text string data to natural language processing matches an extracted sentiment of the open time window, “wherein extracting sentiment of the open time window includes processing video data of the open time window.” The obviousness rejection of claim 6, incorporated herein, shows the obviousness of combining the teachings of Hwang and Couleaud to produce a method and system that uses machine learning models to automatically generate video commentary. As explained in that rejection, the method and system would produce a text script describing events in a video based on the video’s images and audio. Hwang at p. 3, ¶ 5; Couleaud at ¶ 4. The text script would then be synthesized by a TTS module to create a custom synthetic voice in a user-preferred style of a commentator. Hwang at p. 7, ¶ 4. The custom, synthetic voice would be added specifically to silent sections that do not already contain commentary. See Hwang at p. 6, ¶¶ 6–7. In connection with that prior art combination, the Couleaud reference further teaches and suggests generating text descriptions that match the urgency of an event and thus have a sentiment, or urgency, that matches the inferred urgency, or sentiment, extracted from a machine vision analysis of a sports video. Couleaud at ¶¶ 9, 36, 57. For the foregoing reasons, the combination of the Hwang and the Couleaud references makes obvious all limitations of the claim. Claim 10 depends on claim 1, and further requires the following: “wherein the generating, in dependence on the processing, the text string for deployment in the open time window includes “evaluating text string data of the text string according to a semantic meaning redundancy factor, and “qualify[ing] the text string data for deployment responsively to a determination that a sematic meaning of the text string data is not redundant to a semantic meaning of foreground voice data associated to the open time window.” The obviousness rejection of claim 6, incorporated herein, shows the obviousness of combining the teachings of Hwang and Couleaud to produce a method and system that uses machine learning models to automatically generate video commentary. As explained in that rejection, the method and system would produce a text script describing events in a video based on the video’s images and audio. Hwang at p. 3, ¶ 5; Couleaud at ¶ 4. The text script would then be synthesized by a TTS module to create a custom synthetic voice in a user-preferred style of a commentator. Hwang at p. 7, ¶ 4. The custom, synthetic voice would be added specifically to silent sections that do not already contain commentary. See Hwang at p. 6, ¶¶ 6–7. In connection with that prior art combination, the Couleaud reference further teaches and suggests generating text descriptions that match a particular style that accords with an inferred sentiment extracted from a video through machine vision. Couleaud at ¶¶ 4, 68. Couleaud describes generating a script based on prevailing event circumstances. Id. For example, a commentary script is generated that includes details about a player that just scored. Id. This suggests not generating a script about players that have not just scored—namely, generating scripts that are redundant relevant to a semantic meaning of foreground voice data associated with an open time window. See id. For the foregoing reasons, the combination of the Hwang and the Couleaud references makes obvious all limitations of the claim. Claim 13 depends on claim 1, and further requires the following: “wherein the processing media stream data of the multimedia stream includes recognizing based on processing video data of open time window, a certain attribute indicative of an alert game event of a sports game, “prompting a sentiment predicting machine learning model with use of the certain attribute, “wherein the sentiment predicting machine learning model has been trained with labeled training data associating certain sentiment to the certain attribute, “outputting a predicted sentiment based on the prompting, and “configuring acoustical characteristics of the synthesized voice segment in dependence on the predicted sentiment.” The obviousness rejection of claim 6, incorporated herein, shows the obviousness of combining the teachings of Hwang and Couleaud to produce a method and system that uses machine learning models to automatically generate video commentary. As explained in that rejection, the method and system would produce a text script describing events in a video based on the video’s images and audio. Hwang at p. 3, ¶ 5; Couleaud at ¶ 4. The text script would then be synthesized by a TTS module to create a custom synthetic voice in a user-preferred style of a commentator. Hwang at p. 7, ¶ 4. The custom, synthetic voice would be added specifically to silent sections that do not already contain commentary. See Hwang at p. 6, ¶¶ 6–7. In connection with that prior art combination, the Couleaud reference teaches and suggests performing machine vision processing using a deep learning model to analyze video and identify events. Couleaud at ¶ 4. The events are further analyzed to infer their urgency, or a predicted sentiment. Id. at ¶¶ 9, 36. The resulting commentary script and synthesized voices will then reflect this urgency. Id. Couleaud further teaches and suggests using a model that is trained in a supervised manner (i.e., with labeled data). Id. at ¶ 93. For the foregoing reasons, the combination of the Hwang and the Couleaud references makes obvious all limitations of the claim. Claim 14 depends on claim 1, and further requires the following: “wherein the method includes selecting text string data for deployment, “identifying a break point of the text string data that divides the text string data into the text string and a second text string, “storing the second text string to a data repository, and “deploying the second text string to a next open time window.” The obviousness rejection of claim 6, incorporated herein, shows the obviousness of combining the teachings of Hwang and Couleaud to produce a method and system that uses machine learning models to automatically generate video commentary. As explained in that rejection, the method and system would produce a text script describing events in a video based on the video’s images and audio. Hwang at p. 3, ¶ 5; Couleaud at ¶ 4. The text script would then be synthesized by a TTS module to create a custom synthetic voice in a user-preferred style of a commentator. Hwang at p. 7, ¶ 4. The custom, synthetic voice would be added specifically to silent sections that do not already contain commentary. See Hwang at p. 6, ¶¶ 6–7. In connection with that prior art combination for selecting text to be deployed, the Hwang reference teaches and suggests generating all script text at once and then splitting it to fit into each silence interval, storing each text segment in memory (FIG.4) until it can be deployed into each respective silence interval. Hwang at p. 3, ¶ 5, p. 6, ¶¶ 6–7, FIGs.3, 4. For the foregoing reasons, the combination of the Hwang and the Couleaud references makes obvious all limitations of the claim. Claim 16 depends on claim 1, and further requires the following: “wherein the processing media stream data of the multimedia stream includes processing media stream data to predict (i) a time duration of the open time window, (ii) a sentiment of the open time window, and (iii) a semantic meaning of foreground voice segment data associated to the open time window, “wherein the generating, in dependence on the processing, the text string for deployment in the open time window includes selecting the text string to “(a) match the time duration of the open time window, “(b) match the sentiment of the open time window, and “(c) avoid redundancy with the semantic meaning of foreground voice segment data associated to the open time window, “wherein the generating, in dependence on the processing, the text string for deployment in the open time window includes “(1) obtaining one or more video data frame timestamped about a time of the open time window, and “(2) applying the one or more video data frame timestamped about a time of the open time window as model prompting data to a visual language model (VLM) that has been trained with iterations of training data, wherein the iterations of training data include historical frame data labeled with descriptive text, “(3) obtaining output text output from the VLM responsively to the applying, “(4) prompting a large language model (LLM) using the output text output from the VLM, “(5) evaluating VLM-output text strings output by the LLM responsively to the prompting as candidate text strings for deployment in the open time window, wherein the VLM-output text strings include the text string, and “(6) selecting the text string from the candidate text strings based on the evaluating, wherein the evaluating includes a time window fitment evaluation factor, a style matching factor, and a factor that includes evaluation of a semantic meaning of the candidate text strings.” The obviousness rejection of claim 6, incorporated herein, shows the obviousness of combining the teachings of Hwang and Couleaud to produce a method and system that uses machine learning models to automatically generate video commentary. As explained in that rejection, the method and system would produce a text script describing events in a video based on the video’s images and audio. Hwang at p. 3, ¶ 5; Couleaud at ¶ 4. The text script would then be synthesized by a TTS module to create a custom synthetic voice in a user-preferred style of a commentator. Hwang at p. 7, ¶ 4. The custom, synthetic voice would be added specifically to silent sections that do not already contain commentary. See Hwang at p. 6, ¶¶ 6–7. In connection with that prior art combination, the Couleaud reference similarly teaches and suggests generating a commentary script that fits into a time between events, reflects the urgency of the time and avoids semantic redundancy. For example, Couleaud teaches and suggests generating a textual commentary script that is of the correct length, includes ancillary data that is not urgent and corresponds to a silence interval between plays and reflects the background of a player that just scored. Couleaud at ¶¶ 68, 79, FIG.6. Further, Couleaud teaches and suggests obtaining Hwang’s time-stamped video and feeding the frames to a deep learning machine vision or VLM as claimed to generate event descriptions. Couleaud at ¶ 4. Couleaud teaches developing the models through supervised training (i.e., with labeled training datasets). Id. at ¶ 93. Couleaud feeds the descriptions from its machine vision, or VLM, into an ontology model, or LLM, to generate a natural language incident script that is further processed by a virtual persona model (i.e., another LLM) to evaluate text that is ultimately selected as an output to reflect time constraints, style and semantic meaning in the style of a particular commentator. Id. at ¶¶ 94–109, FIGs.7, 8. For the foregoing reasons, the combination of the Hwang and the Couleaud references makes obvious all limitations of the claim. Claim 17 depends on claim 1, and further requires the following: “wherein the generating, in dependence on the processing, the text string for deployment in the open time window includes “(a) obtaining one or more video data frame timestamped about a time of the open time window, and “(b) applying the one or more video data frame timestamped about a time of the open time window as model prompting data to a machine learning model that has been trained with iterations of training data, “wherein the iterations of training data include historical frame data labeled with descriptive text.” The obviousness rejection of claim 6, incorporated herein, shows the obviousness of combining the teachings of Hwang and Couleaud to produce a method and system that uses machine learning models to automatically generate video commentary. As explained in that rejection, the method and system would produce a text script describing events in a video based on the video’s images and audio. Hwang at p. 3, ¶ 5; Couleaud at ¶ 4. The text script would then be synthesized by a TTS module to create a custom synthetic voice in a user-preferred style of a commentator. Hwang at p. 7, ¶ 4. The custom, synthetic voice would be added specifically to silent sections that do not already contain commentary. See Hwang at p. 6, ¶¶ 6–7. In connection with that prior art combination, the Couleaud reference similarly teaches and suggests generating a commentary script that fits into a time between events, reflects the urgency of the time and avoids semantic redundancy. For example, Couleaud teaches and suggests generating a textual commentary script that is of the correct length, includes ancillary data that is not urgent and corresponds to a silence interval between plays and reflects the background of a player that just scored. Couleaud at ¶¶ 68, 79, FIG.6. Further, Couleaud teaches and suggests obtaining Hwang’s time-stamped video and feeding the frames to a deep learning machine vision or VLM as claimed to generate event descriptions. Couleaud at ¶ 4. Couleaud teaches developing the models through supervised training (i.e., with labeled training datasets). Id. at ¶ 93. Couleaud feeds the descriptions from the VLM into an ontology model that operates as an LLM to generate a natural language incident script that is further processed by a virtual persona model (i.e., another LLM) to select text that is selected as an output to reflect time constraints, style and semantic meaning in the style of a particular commentator. Id. at ¶¶ 94–109, FIGs.7, 8. For the foregoing reasons, the combination of the Hwang and the Couleaud references makes obvious all limitations of the claim. Claim 18 depends on claim 1, and further requires the following: “wherein the generating, in dependence on the processing, the text string for deployment in the open time window includes “(a) obtaining one or more video data frame timestamped about a time of the open time window, and “(b) applying the one or more video data frame timestamped about a time of the open time window as model prompting data to a visual language model (VLM) that has been trained with iterations of training data, wherein the iterations of training data include historical frame data labeled with descriptive text, “(c) obtaining output text output from the VLM responsively to the applying, “(d) prompting a large language model (LLM) using the output text output from the VLM, “(e) evaluating VLM-output text strings output by the LLM responsively to the prompting as candidate text strings for deployment in the open time window, wherein the VLM-output text strings include the text string, and “(f) selecting the text string from the candidate text strings based on the evaluating, wherein the evaluating include time window fitment evaluation factor, a style matching factor, and a factor that includes evaluation of a semantic meaning of the candidate text strings.” The obviousness rejection of claim 6, incorporated herein, shows the obviousness of combining the teachings of Hwang and Couleaud to produce a method and system that uses machine learning models to automatically generate video commentary. As explained in that rejection, the method and system would produce a text script describing events in a video based on the video’s images and audio. Hwang at p. 3, ¶ 5; Couleaud at ¶ 4. The text script would then be synthesized by a TTS module to create a custom synthetic voice in a user-preferred style of a commentator. Hwang at p. 7, ¶ 4. The custom, synthetic voice would be added specifically to silent sections that do not already contain commentary. See Hwang at p. 6, ¶¶ 6–7. In connection with that prior art combination, the Couleaud reference similarly teaches and suggests generating a commentary script that fits into a time between events, reflects the urgency of the time and avoids semantic redundancy. For example, Couleaud teaches and suggests generating a textual commentary script that is of the correct length, includes ancillary data that is not urgent and corresponds to a silence interval between plays and reflects the background of a player that just scored. Couleaud at ¶¶ 68, 79, FIG.6. Further, Couleaud teaches and suggests obtaining Hwang’s time-stamped video and feeding the frames to a deep learning machine vision or VLM as claimed to generate event descriptions. Couleaud at ¶ 4. Couleaud teaches developing the models through supervised training (i.e., with labeled training datasets). Id. at ¶ 93. Couleaud feeds the descriptions from the VLM into an ontology model that operates as an LLM to generate a natural language incident script that is further processed by a virtual persona model (i.e., another LLM) to select text that is selected as an output to reflect time constraints, style and semantic meaning in the style of a particular commentator. Id. at ¶¶ 94–109, FIGs.7, 8. For the foregoing reasons, the combination of the Hwang and the Couleaud references makes obvious all limitations of the claim. Summary Claims 1–3, 5–10, 13, 14 and 16–21 are rejected under at least one of 35 U.S.C. §§ 102 and 103 as being unpatentable over the cited prior art. In the event the determination of the status of the application as subject to AIA 35 U.S.C. §§ 102 and 103 (or as subject to pre-AIA 35 U.S.C. §§ 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status . 07-20-02-aia AIA This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 C.F.R. § 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. § 102(b)(2)(C) for any potential 35 U.S.C. § 102(a)(2) prior art against the later invention. Allowable Subject Matter Claims 11, 12 and 22 are objected to for reciting allowable subject matter while depending on a rejected base claim. The claims would be allowable if rewritten in independent form including all limitations of their base claim and any and all intervening claims. Claim 11 depends on claim 1, and further requires the following: “wherein the generating, in dependence on the processing, the text string for deployment in the open time window includes “performing evaluating of text string data of the text string according to an audio-only broadcast emulation factor, “wherein the performing evaluating includes determining a degree to which a semantic meaning of the text string data emulates an audio-only broadcast.” The obviousness rejection of claim 6, incorporated herein, shows the obviousness of combining the teachings of Hwang and Couleaud to produce a method and system that uses machine learning models to automatically generate video commentary. While Couleaud teaches and suggests generating a commentary script with a trained model that will evaluate possible outputs and ultimately select a final text output based on numerous criteria, Couleaud does not teach or suggest the use of an audio-only broadcast criteria in that evaluation. Claim 12 depends on claim 1, and further requires the following: “wherein the method includes identifying an announcer associated to the foreground voice data, “pulling supplemental domain data of the announcer responsively to the identifying, “prompting a neural network machine learning model using the supplemental domain data, “predicting a duration of the open time window based on an output of the predictive model from the prompting, “selecting the text string in dependence on the predicted duration, “recording data specifying an actual duration of the open time window, “further training the neural network machine learning model using the recorded data specifying the actual duration of the open time window, “discovering a subsequent open time window during streaming of the multimedia stream, “re-prompting the neural network machine learning model responsively to the discovering, and “predicting a duration of the subsequent open time window in dependence on the further training.” The obviousness rejection of claim 6, incorporated herein, shows the obviousness of combining the teachings of Hwang and Couleaud to produce a method and system that uses machine learning models to automatically generate video commentary. While Hwang describes the use of a trained model to predict voice and silence intervals, Hwang does not rely on identification of a commentator and the further training of the voice/silence model to improve future predictions of open time windows (i.e., silence intervals). Claim 22 depends on claim 1, and further requires the following: “wherein the method includes identifying supplemental data associated to the foreground voice data, “prompting a machine learning model using the supplemental data, “predicting a characteristic of the open time window based on an output of the machine learning model from the prompting, “selecting the text string in dependence on the predicted characteristic, “recording data specifying an actual characteristic of the open time window, “further training the machine learning model using the recorded data specifying the actual characteristic of the open time window, “discovering a subsequent open time window during streaming of the multimedia stream, “re-prompting the machine learning model responsively to the discovering, and “predicting a characteristic of the subsequent open time window in dependence on the further training.” The obviousness rejection of claim 6, incorporated herein, shows the obviousness of combining the teachings of Hwang and Couleaud to produce a method and system that uses machine learning models to automatically generate video commentary. While Hwang and Couleaud describe the use of trained model to predicts voice and silence sections and to produce commentary about a sports event, the references do not describe, teach or suggest the further training of the voice/silence model to improve future predictions about the characteristics (e.g., length) of open time windows (i.e., silence sections). Response to Applicant’s Arguments Applicant’s Reply at (30 March 2026) includes comments pertaining to the rejections presented in the Non-Final Rejection (20 January 2026) and repeated herein. The Examiner has reviewed the comments and finds them unpersuasive. Regarding claim 1, Applicant comments that the Hwang reference does not describe the claimed generation of a text string. (Reply at 14). According to Applicant, Hwang does not generate a text string from processing multimedia stream data itself. Rather, Hwang parses script data, editing commentary sentences and fitting preexisting textual material to already-identified silent sections. Applicant’s summary of Hwang does not account for Hwang’s mapping process that processes media stream data and textual script data together to map/insert text data into silence sections. In particular, Hwang describes processing a multimedia stream to detect voice sections and silence sections. Those sections are then processed further in a mapping operation to produce a set of timecodes and length values corresponding to each section. Hwang at p. 6, ¶¶ 5–7, FIG.3 (steps 203, 301). In particular, Hwang uses the timecodes and length values of each segment to insert and, as necessary, edit script text into each silence section. Id. Notably, Applicant’s disclosed processing of media stream data includes determining a length of an open time window, or silence section. (Spec. at ¶ 41). Additional processing of Hwang’s stream may include image and audio analysis in order to automatically generate the text of the script. Hwang at p. 3, ¶ 5. Applicant comments that Hwang does not describe examining foreground voice data and identifying in an open time window that is absent of foreground voice data. (Reply at 14–15). Applicant contrasts Hwang’s approach as being drawn to generic silence detection. Hwang, however, specifically describes the use of Wave-U-Net to detect audio sections containing voice and those that do not contain voice. Hwang at p. 6, ¶ 4. Hwang characterizes the latter as a silence section, but in context that simply means the analyzed audio does not contain voice. See id. Applicant has not made any convincing showing that separating audio into voice sections and non-voice sections is materially different than the plain language of “examining foreground voice data of a multimedia stream that includes a video stream data and an audio stream [and] identifying in dependence on the examining an open time window that is absent of foreground voice data.” Hwang just as well identifies non-voice, or silence, sections that do not contain a voice and are suitable for insertion of machine-generated speech. Applicant comments that Hwang does not describe generating a text string for deployment in an open time window and that the audio stream be adapted so that the synthesized voice segment is included in the audio stream and time bounded within the open time window. (Reply at 15). Hwang, however, describes editing the length of a section of text so that when a voice is synthesized from the text, the voice will fit in the silence section. Hwang at p. 7, ¶¶ 1–4, FIGs.5, 6. The addition of the synthesized voice in the silence sections of the existing audio stream is an adaptation of the audio stream itself that results in an audio stream having synthesized voice segments included in the audio stream’s open time windows, or silence sections. Applicant does not explain in any more detail how Hwang’s process differs from the plain language of the claim. Applicant comments that the rejection of claim 1 is based on a machine translation of the Hwang reference. (Reply at 15). Applicant requests a human translation based on alleged inaccuracies in the provided translation of Hwang. ( Id. ) To support this view, Applicant provides an alternative machine translation from Espacenet and a table comparing a characterization of the Examiner’s positions to a characterization of the Espacenet translation ( Id. at 16–27). An applicant is entitled to request a human translation of a foreign-language reference. MPEP § 2120(II). The standard for granting such a request is whether the applicant has provided evidence that the provided machine translation does not accurately represent the reference’s contents. Id. Applicant’s evidence here does not meet this standard. First, Applicant has provided an Espacenet translation of the Hwang reference, but has not made any comparison between the provided translation and the Espacenet translation. Second, Applicant has compared a characterization of the Examiner’s position against a characterization of the Espacenet translation. Whatever value this type of evidence may have, it does not touch on the question of whether the provided translation differs materially from the Espacenet translation. Without any demonstrated deficiencies in the accuracy of the provided translation, there is no basis to grant Applicant’s request for a human translation of the Hwang reference. Regarding claim 2, Applicant comments that the combination of Hwang and Couleaud do not teach modifying a delayed instance of the multimedia stream. (Reply at 27–28). Applicant alleges a distinction between modifying a livestream, which produces a delayed version, and modifying a delayed instance of the stream. The plain language requires delaying a stream and modifying the delayed stream. The Hwang-Couleaud system would operate likewise by delaying an existing stream in order to provide enough time to modify it. Additionally, outside of a real-time embodiment, Hwang describes recording a stream so that it can be effectively delayed and analyzed offline by a script provider in order to generate a script. Hwang at p. 4, ¶ 2. In that context, the script provider would be an automatic script generator, such as the one taught by Couleaud. Regarding claim 6, Applicant comments that Couleaud’s determination of a style does not correspond to the claimed prediction of a style for an open time window. (Reply at 28–30). Couleaud teaches a method of performing machine analysis of a video in order to generate text representing how a particular commentator would speak during a particular section of the video. Couleaud at ¶¶ 1–7. This process involves predicting the style of speech at that time based on the mannerisms of the commentator and the urgency of the section. Id. at ¶¶ 5–6, 9–10. For example, during an exciting event in a sports broadcast, a commentator may speak in an appropriately excited manner. See id. Or when the commentator is providing color commentary, it may be relaxed because it is additional information intended for downtimes in a sports broadcast. Sed id. In other words, Couleaud teaches and suggests that when generating commentary text from a machine analysis, there needs to be a prediction of style for the text in accordance with the character of the speaker and the relevance of the information to the scene. Regarding claim 10, Applicant comments that Couleaud does not teach or suggest determining a semantic meaning redundancy factor and qualifying a text string for deployment when the text string is not redundant to the semantic meaning of foreground voice data associated to the open time window. (Reply at 31–32). Couleaud’s system includes a number of cascaded models that are trained neural networks (NN) models. Couleaud at ¶¶ 4, 16, 68, 93, 104, 107, FIGs.3, 8. As one of ordinary skill would have known, a NN operates with a set of layers of weighted neurons that predicts from its inputs the best response among all possible responses. See id. at ¶ 93. Since Couleaud teaches and suggests only outputting responses that are relevant to the current situation (e.g., providing color commentary about a player that just scored, and not about a player that did not score), it follows that one of ordinary skill would have trained Couleaud’s models to consider a semantic meaning redundancy factor (e.g., relevant responses to a current state of play in a sports match) and qualify a response for output only if it is not redundant (i.e., the response’s semantic meaning is not irrelevant to the current context). Regarding claim 14, Applicant comments that the Hwang reference does not describe storing a second text string in a data repository and deploying the second text string to a next open time window. (Reply at 32–34). Hwang’s system splits a text string into sections that are fit into corresponding silence sections. The rejection of claim 14 shows the obviousness of generating a commentary text string based on the teachings of Couleaud and using that text string in Hwang’s system. Hwang’s system operates by parsing (102) a text string into sections corresponding to silence sections in an audio stream. Hwang at p. 4, ¶¶ 5–6, FIG.4 (depicting stored text strings). This creates a set of text strings that are stored in memory. Id. The texts strings are then sequentially fit (steps 301, 302, 303) into each silence section. Id. at p. 7, ¶ 4, FIG.3. Based on the sequential nature of computing, this logically requires, or at least suggests in the obviousness sense, splitting the text string into a number of sections and storing those sections in memory until each text string can be synthesized and inserted into a corresponding portion of the audio stream. Hwang at p. 7, ¶ 4. Regarding claims 19 and 20, Applicant repeats comments addressed above in connection with claims 2 and 6. (Reply at 34–35). For the foregoing reasons, Applicant has not persuasively established any error in the Office action. All the rejections will be maintained. Conclusion 07-40 AIA Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL . See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 C.F.R. § 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 C.F.R. § 1.17(a)) pursuant to 37 C.F.R. § 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action. Any inquiry concerning this communication or earlier communications from the examiner should be directed to WALTER F BRINEY III whose telephone number is (571)272-7513. The examiner can normally be reached M-F 8 am-4:30 pm. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Carolyn Edwards can be reached at 571-270-7136. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /Walter F Briney III/ Walter F Briney IIIPrimary ExaminerArt Unit 2692 5/28/2026 Application/Control Number: 18/753,653 Page 2 Art Unit: 2692 Application/Control Number: 18/753,653 Page 3 Art Unit: 2692 Application/Control Number: 18/753,653 Page 4 Art Unit: 2692 Application/Control Number: 18/753,653 Page 5 Art Unit: 2692 Application/Control Number: 18/753,653 Page 6 Art Unit: 2692 Application/Control Number: 18/753,653 Page 7 Art Unit: 2692 Application/Control Number: 18/753,653 Page 8 Art Unit: 2692 Application/Control Number: 18/753,653 Page 9 Art Unit: 2692 Application/Control Number: 18/753,653 Page 10 Art Unit: 2692 Application/Control Number: 18/753,653 Page 11 Art Unit: 2692 Application/Control Number: 18/753,653 Page 12 Art Unit: 2692 Application/Control Number: 18/753,653 Page 13 Art Unit: 2692 Application/Control Number: 18/753,653 Page 14 Art Unit: 2692 Application/Control Number: 18/753,653 Page 15 Art Unit: 2692 Application/Control Number: 18/753,653 Page 16 Art Unit: 2692 Application/Control Number: 18/753,653 Page 17 Art Unit: 2692 Application/Control Number: 18/753,653 Page 18 Art Unit: 2692 Application/Control Number: 18/753,653 Page 19 Art Unit: 2692 Application/Control Number: 18/753,653 Page 20 Art Unit: 2692 Application/Control Number: 18/753,653 Page 21 Art Unit: 2692 Application/Control Number: 18/753,653 Page 22 Art Unit: 2692 Application/Control Number: 18/753,653 Page 23 Art Unit: 2692 Application/Control Number: 18/753,653 Page 24 Art Unit: 2692 Application/Control Number: 18/753,653 Page 25 Art Unit: 2692 Application/Control Number: 18/753,653 Page 26 Art Unit: 2692 Application/Control Number: 18/753,653 Page 27 Art Unit: 2692 Application/Control Number: 18/753,653 Page 28 Art Unit: 2692 Application/Control Number: 18/753,653 Page 29 Art Unit: 2692 Application/Control Number: 18/753,653 Page 30 Art Unit: 2692 Application/Control Number: 18/753,653 Page 31 Art Unit: 2692 Application/Control Number: 18/753,653 Page 32 Art Unit: 2692 Application/Control Number: 18/753,653 Page 33 Art Unit: 2692 Application/Control Number: 18/753,653 Page 34 Art Unit: 2692 Application/Control Number: 18/753,653 Page 35 Art Unit: 2692 Application/Control Number: 18/753,653 Page 36 Art Unit: 2692 Application/Control Number: 18/753,653 Page 37 Art Unit: 2692 Application/Control Number: 18/753,653 Page 38 Art Unit: 2692 1 All citations to Hwang’s text refer to the pages and paragraphs included in the machine translation provided by the Office. All citations to Hwang’s figures are made with reference to the original drawings of the reference publication and the OCR translation of the drawings. 2 A new copy of Hwang is being provided with this Office action. The new copy includes a higher quality scan of the original document and includes OCR translations of the drawings.
Prosecution Insights