DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on June 3, 2026 has been entered.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-19 and 21 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3-4, 8, 11, 13-14, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over a combination of Lewis et al. (US 2017/0041680), Basson et al. (US 2010/0073559), and Bhardwaj et al. (US 2014/0365675).
.
Regarding claim 1, Lewis teaches a method comprising:
causing to output content in an audio-only mode ([0024], “the mechanisms described herein can determine that the user device requesting the content is in a background playback mode in which the audio data included in the content item is to be presented, but the visual data of the content item is either inhibited from being presented or is unlikely to be viewed by a user even if it were to be presented.” [0039], “At 304, process 300 can determine if the user device requesting the content is being used in a background playback mode.” [0044]-[0045], Fig. 3);
determining, during playback of the content, that the played content reaches a section containing visual information that is not reproduced in the audio-only mode ([0025], “The mechanisms can evaluate any suitable properties of the one or more content items to determine whether the one or more content items are suitable for presentation in background playback mode, such as the presence or absence of a lengthy introduction (e.g., dialogue) to the content item before music starts, the presence or absence of periods of silence or only noise, the presence of periods of dialogue, the audio quality of the content item, repetitiveness in the audio data (e.g., where the audio data is unvaried over a relatively large portion of the content item), etc. For example, if the particular video requested by the smartphone includes a long silence at the end (e.g., with visual information and/or user interface elements prompting a user to subscribe to content from a user associated with the particular video) and/or the audio data is of poor quality, the mechanisms described herein can determine that the particular video is not suitable for background playback.” [0046], “At 310, process 300 can determine if the requested content is suitable for background playback.” [0048], “In some embodiments, process 300 (and/or any other suitable process) can determine the suitability of a content item for background playback based at least in part on the amount and/or length of silences in audio of the content item.” Fig. 3); and
based at least in part on determining that the played content reaches the section containing visual information that is not reproduced in the audio-only mode ([0046], “At 310, process 300 can determine if the requested content is suitable for background playback.” [0048], “In some embodiments, process 300 (and/or any other suitable process) can determine the suitability of a content item for background playback based at least in part on the amount and/or length of silences in audio of the content item.” Fig. 3):
accessing metadata of the section containing visual information that is not reproduced in the audio-only mode ([0049], “In some embodiments, when a content item is transmitted to a user device and/or when a content item is presented by a user device, a portion at the beginning and/or the end that includes silence or dialogue with no music can be automatically skipped over by the user device based on metadata associated with the content item and/or based on instructions transmitted to the user device to skip a particular portion of a content item.” [0075], “In some embodiments, process 400 can generate metadata that can be used to identify a content item that is to be substituted for the content item that is not suitable for background playback.” Fig. 4);
identifying a supplemental audio portion based at least in part on the metadata of the section containing visual information that is not reproduced in the audio-only mode, wherein the supplemental audio portion comprises at least the visual information that is not reproduced in the audio-only mode ([0059], “if process 300 determines that the requested content it not suitable for background playback (‘NO’ at 312), process 300 can move to 316.” [0060], “At 316, process 300 can cause the requested content to be skipped and/or can cause replacement content that is suitable to be transmitted to the user device instead of the requested content.” [0062], “In some embodiments, if process 300 determines that a content item is not suitable for background playback, process 300 can cause a replacement content item to be transmitted in response to the request. Such a replacement item can include the same or similar content to the requested content item, but be more suitable for background playback.” [0075], “In some embodiments, process 400 can generate metadata that can be used to identify a content item that is to be substituted for the content item that is not suitable for background playback.” Figs. 3-4);
causing to output the supplemental portion instead of the audio-only mode for the section containing visual information that is not reproduced in the audio-only mode ([0059], “if process 300 determines that the requested content it not suitable for background playback (‘NO’ at 312), process 300 can move to 316.” [0060], “At 316, process 300 can cause the requested content to be skipped and/or can cause replacement content that is suitable to be transmitted to the user device instead of the requested content.” [0062], “In some embodiments, if process 300 determines that a content item is not suitable for background playback, process 300 can cause a replacement content item to be transmitted in response to the request. Such a replacement item can include the same or similar content to the requested content item, but be more suitable for background playback.” Fig. 3).
While Lewis teaches identifying the supplemental audio portion, Lewis does not expressly teach generating the supplemental audio portion. Lewis does not expressly teach causing to output the supplemental portion instead of the audio-only mode for the section until completion of the supplemental portion, and causing to resume output of the content in the audio-only mode.
Basson teaches generating supplemental audio portion, wherein the supplemental audio portion comprises at least visual information that is not reproduced in primary audio content ([0028], “FIG. 1 illustrates a block diagram view of a system 2 for generating additional verbal descriptions for specific graphics within video data 5b, in accordance with embodiments of the present invention. The additional verbal descriptions for specific graphics within video data 5b may be generated so that a disabled audience or viewer (e.g., hearing impaired, visually impaired, etc) may better understand the specific graphics. For example, a slide show may comprise basic text or audio descriptions for each slide (e.g., a highway scene) but a specific graphic (e.g., an automobile) in a slide may not be adequately described (e.g., only referred to as an automobile). Therefore, system 2 will generate a detailed audio and/or text description for the automobile (e.g., the automobile is a red convertible of a specific model and brand) and add the detailed audio and/or text description for the automobile to an associated portion of the slide show.”).
In view of Basson’s teaching, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Lewis to include generating the supplemental audio portion. The modification would serve to enable a combined system with an additional means of catering content to users’ requirements or preferences.
The combination teaches the limitations specified above; however, the combination does not expressly teach causing to output the supplemental portion instead of the audio-only mode for the section until completion of the supplemental portion, and causing to resume output of the content in the audio-only mode.
Bhardwaj teaches causing to output supplemental portion until completion of the supplemental portion, and causing to resume output of content ([0078], “The scenario 400 proceeds to the bottom portion where playback of the content stream 402 resumes from the resume point 414 and utilizing the resume state 420. For example, playback skips from the end of the supplementary content 410 to the resume point 414, without playing the segment 404b or portions of the segment 406b that occur before the resume point 414. Utilizing the resume state 420 provides a data context for processing and output of the segment 406b from the resume point 414, as well as subsequent portions of the content stream 402.” [0111], “embodiments enable supplementary content to be inserted at any specified point in a content stream, such as between boundaries of a segment and/or a period. Embodiments further enable playback of a content stream to resume after supplementary content has been inserted.”).
In view of Bhardwaj’s teaching, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination to enable causing to output the supplemental portion instead of the audio-only mode for the section until completion of the supplemental portion, and causing to resume output of the content in the audio-only mode. The modification would serve to facilitate a return to user-preferred content. The modification would thereby improve the user experience.
Regarding claims 3 and 13, the combination further teaches a method comprising:
accessing the metadata of the section containing visual information that is not reproduced in the audio-only mode for supplemental information associated with the content, wherein the supplemental information includes at least one of an actor, a character, music, a commentary, a rating, bonus content, trivia, or social media network information; identifying audio information corresponding with the accessed supplemental information; and causing to output the supplemental portion based at least in part on the accessed supplemental information (Lewis: [0075], “In some embodiments, process 400 can generate metadata that can be used to identify a content item that is to be substituted for the content item that is not suitable for background playback. For example, the metadata can include a content identifier of a song in the content item which can be used to identify a content item that includes the same song but that is more suitable for background playback.”).
Regarding claims 4 and 14, the combination further teaches a method comprising:
accessing a data of the section containing visual information that is not reproduced in the audio-only mode for supplemental information associated with the content, wherein the supplemental information includes at least one of an actor, a character, music, a commentary, a rating, bonus content, trivia, or social media network information; identifying audio information corresponding with the accessed supplemental information; and causing to output the supplemental portion based at least in part on the accessed supplemental information (Lewis: [0075], “In some embodiments, process 400 can generate metadata that can be used to identify a content item that is to be substituted for the content item that is not suitable for background playback. For example, the metadata can include a content identifier of a song in the content item which can be used to identify a content item that includes the same song but that is more suitable for background playback.”).
However, the combination as presently combined does not expressly teach accessing a manifest of the section for supplemental information associated with the content.
Bhardwaj teaches accessing a manifest for supplemental information associated with content ([0045], “in at least some embodiments, the modified manifest 216 is utilized to manage playback of the content stream 202 and the supplementary content 212 ….”) [0047], “The modified manifest 216 further indicates that the supplementary content 212 is inserted between the segment 206a and the segment 204b.”).
In view of Bhardwaj’s teaching, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination to include accessing a manifest of the section containing visual information that is not reproduced in the audio-only mode for supplemental information associated with the content. The modification would produce a combined system enabled with an additional and/or alternative means of identifying supplemental content related to a current program.
Regarding claims 8 and 18, the combination further teaches a method comprising:
determining advertisement-related content associated with the content; and causing to output the supplemental portion based at least in part on the advertisement-related content (Lewis: [0058], “In some embodiments, when an advertisement is determined not to be suitable for background playback, a substitute advertisement can be provided in its place.”).
Regarding claim 11, Lewis teaches a system comprising: circuitry ([0031]-[0034], Fig. 2) configured to execute the method of claim 1. The grounds of rejection of claim 1 under 35 USC §103 are similarly applied to the remaining limitations of claim 11.
Claim(s) 2 and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over a combination of Lewis, Basson, Bhardwaj, and Chang et al. (US 2009/0259473).
Regarding claims 2 and 12, the combination teaches the limitations specified above, and wherein the supplemental audio portion comprises the text in an audio format (Basson: [0028], “For example, a slide show may comprise basic text or audio descriptions for each slide (e.g., a highway scene) but a specific graphic (e.g., an automobile) in a slide may not be adequately described (e.g., only referred to as an automobile). Therefore, system 2 will generate a detailed audio and/or text description for the automobile (e.g., the automobile is a red convertible of a specific model and brand) and add the detailed audio and/or text description for the automobile to an associated portion of the slide show.”).
However, the combination does not expressly teach determining text present in frames of the section containing visual information that is not reproduced in the audio-only mode; and causing to output the text in an audio format.
Chang determining text present in frames of content containing visual information that is not reproduced in audio, and causing to output the text in an audio format ([0012], “FIG. 1 is a schematic illustration of an example STB 100 that, in addition to receiving and presenting video programs to a user, detects textual portions of a video program that are not readily consumable by a visually impaired person, converts said textual portions into corresponding audio data, and presents the thus generated audio data to the visually impaired person to aid in the consumption of the video program. Example textual portions include, but are not limited to a series of video frames presenting a substantially static text-based information screen (e.g., the example screen snapshot of FIG. 2), a series of video frames presenting scrolling text within all or a portion of a screen (e.g., the example screen snapshots of FIGS. 3A and 3B), and/or text-based information that describes a series of video frames that do not have associated spoken words (e.g., when a panoramic scene is being displayed but no person is speaking, and the text-based information describes the scene being displayed). It will be readily understood that such text-based portions and any other type(s) of text-based portions of a video program are not readily consumable by visually impaired persons.”).
In view of Chang’s teaching, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination to include determining text present in frames of the section containing visual information that is not reproduced in the audio-only mode; and causing to output the text in an audio format. The modification would serve to facilitate consumption of text content while consuming content in audio only mode.
Regarding claim 21, the combination further teaches:
determining that a segment of the played content includes video text relevant to subsequent audio of the played content; and
wherein the generating the supplemental audio portion comprises
generating the supplemental audio portion based at least in part on the video text ([0028], “FIG. 1 illustrates a block diagram view of a system 2 for generating additional verbal descriptions for specific graphics within video data 5b, in accordance with embodiments of the present invention. The additional verbal descriptions for specific graphics within video data 5b may be generated so that a disabled audience or viewer (e.g., hearing impaired, visually impaired, etc) may better understand the specific graphics. For example, a slide show may comprise basic text or audio descriptions for each slide (e.g., a highway scene) but a specific graphic (e.g., an automobile) in a slide may not be adequately described (e.g., only referred to as an automobile). Therefore, system 2 will generate a detailed audio and/or text description for the automobile (e.g., the automobile is a red convertible of a specific model and brand) and add the detailed audio and/or text description for the automobile to an associated portion of the slide show.”).
However, the combination does not expressly teach generating, using a text-to-speech generator, the supplemental audio portion.
Chang teaches generating, using a text-to-speech generator, the supplemental audio portion ([0010], “A disclosed example set-top box (STB) includes a network interface to receive a video stream and an associated audio stream, a detector to identify a portion of the video stream that is not readily consumable by a visually impaired person, a text locator to obtain text associated with the portion of the video stream, a text-to-speech (TTS) module to convert the text to a second audio stream, and a mixer to combine the second audio stream with the associated audio stream.”).
In view of Chang’s teaching, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination to include generating, using a text-to-speech generator, the supplemental audio portion. The modification would serve to provide an additional and/or alternative means of generating audio content for distribution to users. The modification would improve the user experience.
Claim(s) 5-7 and 15-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over a combination of Lewis, Basson, Bhardwaj, and Trollope et al. (US 9491522).
Regarding claims 5 and 15, the combination further teaches a method comprising:
accessing data of the section containing visual information that is not reproduced in the audio-only mode for supplemental information associated with the content, wherein the supplemental information includes at least one of an actor, a character, music, a commentary, a rating, bonus content, trivia, or social media network information; identifying audio information corresponding with the accessed supplemental information; and causing to output the supplemental portion based at least in part on the accessed supplemental information (Lewis: [0075], “In some embodiments, process 400 can generate metadata that can be used to identify a content item that is to be substituted for the content item that is not suitable for background playback. For example, the metadata can include a content identifier of a song in the content item which can be used to identify a content item that includes the same song but that is more suitable for background playback.”).
However, the combination does not expressly teach accessing a subtitle of the section for supplemental information associated with the content.
Trollope teaches accessing a subtitle of a section for supplemental information associated with content (Col. 17, line 64 to col. 18, line 12, “In some implementations, a capture module 1122 can receive media data related to a program or a channel, such as video data, audio data, electronic program guide data, metadata, subtitles or captioning content, etc., as described above in connection with, for example, FIGS. 1 and 2. Additionally or alternatively, capture module 1122 can extract various media data from content provided from content sources as described in connection with, for example, FIGS. 1 and 2. Such extracted media data can include, for example, audio fingerprints, subtitles, etc. This information can be stored, for example, in a database (not shown) for use by the search application executing on front-end server 1120 in identifying channels, identifying program and/or other program-related information, obtaining supplemental content items, and/or various other operations.” Col. 18, lines 29-43, “keyword extraction module 1130 can extract keywords from captured audio data, video data, and/or subtitle information and obtain supplemental content items from multiple content sources (e.g., content sources 1114).”).
In view of Trollope’s teaching, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination to include accessing a subtitle of the section containing visual information that is not reproduced in the audio-only mode for supplemental information associated with the content. The modification would produce a combined system enabled with an additional and/or alternative means of identifying supplemental content related to a current program.
Regarding claims 6 and 16, the combination further teaches a method comprising:
accessing a data of the section containing visual information that is not reproduced in the audio-only mode for supplemental information associated with the content, wherein the supplemental information includes at least one of an actor, a character, music, a commentary, a rating, bonus content, trivia, or social media network information; identifying audio information corresponding with the accessed supplemental information; and causing to output the supplemental portion based at least in part on the accessed supplemental information (Lewis: [0075], “In some embodiments, process 400 can generate metadata that can be used to identify a content item that is to be substituted for the content item that is not suitable for background playback. For example, the metadata can include a content identifier of a song in the content item which can be used to identify a content item that includes the same song but that is more suitable for background playback.”).
However, the combination does not expressly teach accessing a closed caption data of the section for supplemental information associated with the content.
Trollope teaches accessing a closed caption data of a section for supplemental information associated with content (Col. 17, line 64 to col. 18, line 12, “In some implementations, a capture module 1122 can receive media data related to a program or a channel, such as video data, audio data, electronic program guide data, metadata, subtitles or captioning content, etc., as described above in connection with, for example, FIGS. 1 and 2. Additionally or alternatively, capture module 1122 can extract various media data from content provided from content sources as described in connection with, for example, FIGS. 1 and 2. Such extracted media data can include, for example, audio fingerprints, subtitles, etc. This information can be stored, for example, in a database (not shown) for use by the search application executing on front-end server 1120 in identifying channels, identifying program and/or other program-related information, obtaining supplemental content items, and/or various other operations.”).
In view of Trollope’s teaching, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination to include accessing a closed caption data of the section containing visual information that is not reproduced in the audio-only mode for supplemental information associated with the content. The modification would produce a combined system enabled with an additional and/or alternative means of identifying supplemental content related to a current program.
Regarding claims 7 and 17, the combination teaches the limitations specified above; however, the combination does not expressly teach:
accessing one or more social media networks to retrieve comments or posts related to the content; and
causing to output the supplemental portion based at least in part on the retrieved comments or posts.
Trollope teaches accessing one or more social media networks to retrieve comments or posts related to content, and causing to output supplemental content based at least in part on the retrieved comments or posts (Col. 13, lines 32-39, “As shown in FIG. 7, recommendation cards 720, 730, and 740 each include a supplemental content item. For example, in response to determining the keywords ‘John Smith’ and ‘hydraulic fracking,’ cards 720 and 730 that provide text snippets, thumbnail images, links, and/or other supplemental content relating to ‘John Smith’ and card 740 that provides social media snippets relating to the topic ‘hydraulic fracking’ can be presented to the user of the client application.”).
In view of Trollope’s teaching, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination to include accessing one or more social media networks to retrieve comments or posts related to the content, and causing to output the supplemental portion based at least in part on the retrieved comments or posts. The modification would serve to enable a combined system to provide supplemental content items based on social media content. The modification would serve to improve the user experience.
Claim(s) 9 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over a combination of Lewis, Basson, Bhardwaj, and Huttner et al. (US 11513759).
Regarding claims 9 and 19, the combination teaches the limitations specified above; however, the combination does not expressly teach preceding the supplemental audio portion with a special sound or tone to inform the user that the supplemental audio portion is not part of the main content.
Huttner teaches preceding supplemental audio with a special sound or tone (Col. 2, lines 31-20, “In one or more embodiments, a listener may be given an option to have a device present supplemental content. The option may be presented in the form of a ‘soundmark’ (e.g., a notification sound such as a tone, beep, series of sounds, etc.) that indicate the presence of available supplemental content. … A soundmark may be distinct for different types of content. For example, one soundmark may indicate to a listener that footnote or endnote content is available. Another soundmark may indicate to a listener that images are available. Another soundmark may indicate to a listener that charts or graphs are available. Another soundmark may indicate to a listener that additional audio (e.g., music) is available. Another soundmark may indicate to a listener that content promotions are available.”).
In view of Huttner’s teaching, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination to include preceding the supplemental audio portion with a special sound or tone to inform the user that the supplemental audio portion is not part of the main content. The modification would serve to facilitate user access to supplemental content while listening to audio content.
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over a combination of Lewis, Basson, Bhardwaj, and Hattery et al. (US 2020/0077147).
Regarding claim 10 , the combination teaches the limitations specified above; however, the combination does not expressly teach including delimiter words before and after the supplemental audio portion to indicate that the audio that follows the delimiter words is not part of the content.
Hattery teaches including delimiter words before supplemental content to indicate that the content that follows the delimiter words is not part of the content ([0019], “FIG. 2B illustrates a scenario showing what may occur when the zombie show is nearing a break 215, according to particular embodiments. The social-networking system 222 may anticipate the start time of a break in the television program. At a predetermined amount of time prior to the anticipated start time of the break, the social-networking system 222 may transmit a notification 231 to the user's device 230. The notification 231 may be surfaced to the user 201 through an application installed on the device 230, such as a social-networking application associated with the social-networking system 222, text message, phone call, e-mail, or any other suitable communication channels. The notification 231 may include a message that informs the user 201 that live streaming content related to the zombie show 215 is about to begin.”).
In view of Hattery’s teaching, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination to include delimiter words before and after the supplemental audio portion to indicate that the audio that follows the delimiter words is not part of the content. The modification would serve to facilitate user recognition of supplemental content. The modification would thereby improve the user experience.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: Bettis et al. (US 2007/0058681) discloses a voice media server interfaced with a text-to-speech server ([0028]).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL R TELAN whose telephone number is (571)270-5940. The examiner can normally be reached 9:30AM-6:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Nasser Goodarzi can be reached at (571) 272-4195. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL R TELAN/Primary Examiner, Art Unit 2426
Prosecution Insights