DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The IDS filed on January 30, 2026 has been considered and placed in the application file.
Response to Arguments
Applicant's arguments filed January 30, 2026 have been fully considered and are persuasive regarding the rejection under 35 U.S.C. 112(b), but are not persuasive regarding the prior art rejections.
Applicant argues that the combined teachings of Dareddy and Chen do not teach
extracting “auxiliary information from a second portion of the video content, the second portion being at a predetermined time period after the first portion from which the input region is extracted”, as recited in independent claim 1. In particular, Applicant argues that Chen determines a scene change between two shots of the same tile, not from auxiliary information extracted a predetermined time period after the input region.
The examiner disagrees. The examiner relied on Chen for its teaching of detecting a scene change event in a video based on images of the video that are a predetermined time period after the scene change event. Col. 2, line 1 – Col. 3, line 23 of Chen discloses processing shots of video sequences that occur later in time to determine whether an earlier-in-time shot of the video sequences contains a scene change event. For example, Fig. 4A, Col. 9, line 65 – Col. 10, line 28 discloses using features of shots 6 and 7 as auxiliary information to determine whether the earlier-in-time shot 5 includes a scene change event. Shots 6 and 7 each comprise a sequence of image frames that follow the sequence of image frames comprising shot 5 by a predetermined time period equal to how far in time shots 6 and 7 are from shot 5 and the duration of each shot.
In the present disclosure, the auxiliary information of the second portion can be audio data or image data whereas the input region of the first portion is image data. In embodiments where the auxiliary information of the present disclosure is image data rather than audio data, the image data extracted from the second portion of the video content can be the same type of information as the image data extracted from the input region of the first portion since it is all image data extracted from the video content. There is nothing in claim 1 that requires the first and second portions of the video content to be different types of information or to be located in different, non-overlapping regions of the video content. In Chen, the image data of shots 6 and 7 that is used as auxiliary information to determine if a scene change event occurred in shot 5 is a predetermined time period after the scene change event of shot 5. As far as amended claim 1 is concerned, it makes no difference whether the auxiliary information of the second portion and the image data of the first portion are the same type of video content since the first portion comprises image data and the second portion can comprise image data, and it all is included in the video content. For these reasons, the combined teachings of Dareddy and Chen teach the limitations of claims 1, 6 and 7.
Claim Interpretation
The claims in this application are given their broadest reasonable interpretation (BRI) using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The BRI of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification.
In the following, some of the terms in the claims have been given BRIs in light of the specification. These BRIs are used for purposes of searching for prior art and examining the claims, but cannot be incorporated into the claims. Should Applicant believe that different interpretations are appropriate, Applicant should point to the portions of the specification that clearly support a different interpretation.
The term “input region” is interpreted as video content within the “target region” described with reference to Fig. 5 as the central region 70 of the displayed screen (para. [0033]).
The term “auxiliary information” is interpreted as corresponding to user interface (UI) elements of the displayed screen, such as first and second hit-point (HP) gauges 75 and 76, respectively, shown in Fig. 5, or as sound or audio data (paras. [0014], [0033], [0049]).
The phrase “correct data” is interpreted as information indicative of the occurrence of an event or information indicative of an event type (paras. [0034], [0050], [0051]).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 3, 6-8 and 10-15 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Publ. Appl. No. 2020/0186897 by Dareddy et al. (hereinafter referred to as “Dareddy”) in view of U.S. Patent No. 11,748,988 to Chen et al. (hereinafter referred to as “Chen”).
Regarding claim 1, Dareddy et el. discloses an image analysis system (Fig. 5, 500) comprising:
processing circuitry (Fig. 5, 504a, 504b) configured to:
extract an input region (video frames are extracted by module 504a, Fig. 5, paras. [0095] and [0098]) from video content including multiple time-series images and time-series audio data, (para. [0045], Fig. 2, S202 and S204; the video game of Dareddy comprises time-series images and time-series audio data, Abstract: “the video game including at least a video and corresponding audio signal”), the input region being extracted from a first portion of each image of a group of the multiple time-series images of the video content (para. [0046], lines 1-3; para. [0048]), the input region includes an object (Fig. 4 shows the input region including objects);
extract auxiliary information from a second portion of the video content (See the BRI for this limitation above; paras. [0045]-[0046] of Dareddy disclose that the audio frames are in an audio portion of the video game data signal that is second portion that is different from the first portion of the video game data comprising the video frames from which the input region is extracted; paras. [0097]-[0098] discuss the audio being extracted from the video game data by module 504b of Fig. 5; Dareddy does not explicitly disclose that the second portion is a predetermined time period after the first portion);
processing the extracted auxiliary information to detect an event for the object included in the input region in the auxiliary information and generating correct data as the detected event (Fig. 5, para. [0098], module 504b processes the extracted auxiliary information and generates a correct answer as to events occurring within the video game based on audio frames input to the module 504b; para. [0068], the audio machine learning model is trained to determine the presence of a highlight event or a non-highlight event based at least in part on the audio auxiliary information (the presence of a non-highlight event constitutes the absence of a highlight event); and
train a machine learning model based on training data (training units 912a, 912b are machine learning models that train modules 504a and 504b, respectively; paras. [0145]-[0156]) including the input region and the correct data, wherein the machine learning model is trained to detect events in the video content based on the training data (paras. [0145]-[0156], the video data, the audio data and the correct answer regarding events detected based on the audio data are used as training data to train modules 504a and 504b).
Dareddy does not explicitly disclose that the second portion of the video content from which the auxiliary information is extracted is a predetermined time period after the group of multiple images from which the input region is extracted. Chen, in the same field of endeavor, discloses determining events corresponding to scene changes in a video such as a movie by extracting and processing auxiliary information comprising video shots that are a predetermined time period after the images corresponding to the scene change event to determine whether the current shot includes a scene change (see the discussion above in Response to Arguments; Chen, Col. 2, line 1 – Col. 3, line 23 discuss processing shots of video sequences to determine whether a current shot contains a scene change, or boundary; Fig. 4A, Col. 9, line 65 – Col. 10, line 28 discloses using features of shots 6 and 7 as auxiliary information to determine whether the current shot 5 includes a scene change, where shots 6 and 7 each comprise a sequence of image frames that are a predetermined time period after the sequence of image frames comprising shot 5, where the predetermined time period is equal to the distance in time between shots 5 and 6/7 and on the time duration of the shots).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present disclosure to modify the system of Dareddy to extract auxiliary information that is located a predetermined time period after the images from which the input region video is extracted as taught by Chen and to use the auxiliary information to detect the occurrence of an event in the input region. A person of ordinary skill in the art would have been motivated to make the modification to improve the accuracy of event detection by using not only video frames corresponding to the current frame to detect events, but also video frames that follow the current frame in time by a predetermined time period. The modification could have been made by a person of ordinary skill in the art with a reasonable expectation success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (modifying the software comprising learning module 504b of Dareddy to ensure that the audio frames that are processed to detect the occurrence of a highlight event include audio frames that follow the image frames from which the input region is extracted by a predetermined time period).
Regarding claim 3, as indicated above, Dareddy does not explicitly disclose extracting the auxiliary information (the audio data) from a portion of the video content that is located at a predetermined time period following the group of multiple images from which the input region is extracted. Chen discloses extracting the auxiliary information (video shots that follow the current video shot in time) from a portion of the video content that is located at a predetermined time period following the current shot. In Chen, the predetermined time period is based on the number of frames back that the shots 6 and 7 are in time from the current shot 5. In the example described in Chen with reference to Fig. 4A, features of shots 6 and 7 are used to determine whether current shot 5 contains a scene change. In that example, the predetermined time period is based on the length of time over which each shot extends and how far back in time the shots are that are used as auxiliary information. Each shot is made up of “a contiguous sequence (e.g., series) of frames recorded (e.g., by a single camera) over an uninterrupted period of time” (Chen, Col. 2, lines 20-21). Therefore, the predetermined time period in Chen corresponds to a predetermined number of consecutive images that the following shots are from the current shot, which would be based on predetermined system settings.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present disclosure to modify the listening device 500 of Dareddy based on the teachings of Chen to extract auxiliary information comprising audio that is located a predetermined number of consecutive image frames after the current shot comprising the video input region as taught by Chen. A person of ordinary skill in the art would have been motivated to make the modification to improve the accuracy of event detection by using not only audio frames corresponding to the current frame, but also audio frames that follow the current shot in time by a consecutive predetermined number of frames. The modification could have been made by a person of ordinary skill in the art with a reasonable expectation success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (modifying the software comprising learning module 504b to ensure that the audio frames that are processed to detect the occurrence of a highlight event include audio frames that follow the group of images from which the input region is extracted by a predetermined amount of time).
Regarding claim 6, the rejection of claim 1 above applies mutatis mutandis to claim 6.
Regarding claim 7, to the extent that claim 7 recites the same limitations that are recited in claim 1, the rejection of claim 1 above applies mutatis mutandis to claim 7. The only limitations that are recited in claim 7 that are not also recited in claim 1 is the non-transitory computer-readable medium storing instructions for performing the method recited in claims 1 and 7. Dareddy et al. discloses a non-transitory computer-readable medium storing instructions for performing the method (para. [0160]).
Regarding claim 8, as indicated above in the rejection of claim 1, Dareddy discloses that the second portion of the video content from which the auxiliary information is extracted is an audio portion (paras. [0045]-[0046] of Dareddy disclose that the audio frames are in a portion of the video game data signal that is different from the portion of the video game data comprising the video frames from which the input region is extracted; paras. [0097]-[0098] discuss the audio being extracted from the video game data by module 504b of Fig. 5).
Regarding claim 10, the rejection of claim 3 above applies mutatis mutandis to claim 10.
Regarding claim 11, the rejection of claim 8 above applies mutatis mutandis to claim 11.
Regarding claim 12, the rejection of claim 9 above applies mutatis mutandis to claim 12.
Regarding claim 13, the rejection of claim 3 above applies mutatis mutandis to claim 13.
Regarding claim 14, the rejection of claim 8 above applies mutatis mutandis to claim 14.
Regarding claim 15, the rejection of claim 9 above applies mutatis mutandis to claim 15.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Dareddy in view of Chen as applied to claims 1, 3, 6-8 and 10-15 and further in view of U.S. Publ. Appl. No. 2022/0118363 A1 to Yerva et al. (hereinafter referred to as “Yerva”).
Dareddy does not explicitly disclose that the second portion of the video content from which the auxiliary information is extracted is a different region with images of the video content that are part of the portion of the video content from which the input region is extracted. Yerva, which was cited in the nonfinal Office Action mailed on February 10, 2025, a portion of which is duplicated below for convenience, discloses that the correct answer, i.e., detecting that a particular event has occurred, is generated by a correct answer generation section 506/508 on the basis of a change in the auxiliary information, i.e., on the basis of a change in the UI elements 102-120 shown in Fig. 1A. (Para. [0022] of Yerva: “[t]he information contained in at least some of these regions can change over time, and those changes can be indicative of various types of events. In at least one embodiment, events can be determined by detecting changes in one or more of these regions, and combining information for that change with information in one or more other reasons that may be used to determine a type of event that has occurred”; see also Figs. 5A and 5B and para. [0036]: “[i]n this example, the event recognition module 506 can analyze information in these primary regions, and can pass this information to an event analysis module 508. An event recognition module can use one or more event auto-recognition algorithms, processes, or deep learning approaches to recognize events, or objects and occurrences associated with various types of events.”).
Because the UI elements 102-120 that are used in Yerva as the auxiliary information are disposed around the periphery of the main scene, but are part of the video content, the second portion of the video content from which the UI element images are extracted is a different region with images of the video content than the video content of the input region, which would correspond to the primary image of the scene shown below in Fig. 1A of Yerva.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present disclosure to combine the teachings of Dareddy with the teachings of Yerva to modify the listening device 500 of Dareddy to include a module, e,g,, 504f, to extract the states of UI elements shown in Fig. 4 of Dareddy, as auxiliary information and to use changes in the UI elements in combination with the image content (Gameplay scene) in the highlight detector 506 to detect events (paras. [0073] and [0101] of Dareddy) according to the teachings of Yerva. One of ordinary skill in the art would have been motivated to make the change to improve the accuracy of event detection by using not only audio auxiliary information to detect events, but also using states of the UI elements as auxiliary information to detect events. The modification could have been made by a person of ordinary skill in the art with a reasonable expectation success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (modifying the system to use UI status indicators in combination with audio as auxiliary information to detect events in videos).
PNG
media_image1.png
200
400
media_image1.png
Greyscale
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
U.S. Publ. Appl. No. 2020/0215434 A1 to Goldman discloses a detection unit 110 that can be configured to detect an in-game interaction based on a score indicator included in a region of the video image, and/or a level of brightness associated with a region of the video image and/or a property of an audio signal associated with the video image. As well as detecting an in-game event, the avatar pose corresponding to the in-game interaction can also be detected. Information indicative of a property of the in-game interaction and the avatar pose corresponding to the in-game event can be provided to the correlation unit 130. Using this information the correlation unit 130 can select one of the avatar-status boxes assigned to the avatar and determine a position for the avatar-status box.
U.S. Publ. Appl. No. 2020/0035019 A1 to Capello et al. discloses that events may be detected by, for example, using machine learning. For example, a machine learning model may be trained with video clips of known events and labels of those events, and trained to determine a correlation between the content of those video clips and the corresponding labels. Alternatively, the model may be trained via unsupervised learning, by using a plurality of video clips of e.g. football games, and classifying sufficiently similar video clips as corresponding to a particular type of event. Once trained, the output of the model may be used by a processor to determine a virtual camera angle from which that event is to be captured. In other examples, the events may be detected based on e.g the audio associated with the video. For example, speech recognition may be used to identify words spoken by e.g. a commentator, from which the relevant event can be determined. The speech recognition may be based on the use of machine learning, using a model that has been trained with speech samples and labels indicating the type of event associated with audio of that nature.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL J SANTOS whose telephone number is (571)272-2867. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Bella can be reached at (571)272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL J. SANTOS/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights