DETAILED ACTION
This communication is responsive to Amendment, filed 07/29/2026.
Claims 31-50 are pending in this application. In the Amendment, claims 1-30 have been cancelled, claims 31,40, and 49 have been amended. This action is made Final.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 31-48 are rejected on the ground of nonstatutory obviousness-type double patenting as being unpatentable over claims 1 and 2 of U.S. Patent No. 12,050,839 in view of Swerdlow et al. (“Swerdlow”, Pub. No. US 2023/0308709).
Claim 31 Of Instant Application
Claim 2 of U.S. Patent No. 12,050,839
A method, comprising:
-generating for output, at a computing device, a content item;
-determining content of the content item;
-identifying a subset of soundmojis from a plurality of soundmojis that are contextually related to the content of the content item, wherein each soundmoji comprises a predefined association of an emoji component with an audio component;
- generating a soundmoji menu comprising one or more icons, wherein each icon enables a soundmoji of the subset of soundmojis to be selected for output and generating for output, at the computing device, the soundmoji menu while the content item is being generated for output.
A method comprising:
- generating for output, at a first computing device, a first content item;
-determining content of the first content item;
-identifying, based on the determined content, one or more soundmojis;
receiving a first input associated with a selection of a soundmoji menu;
-in response to receiving the first input, generating the soundmoji menu comprising one or more icons, each icon associated with a soundmoji for output, wherein each icon enables a soundmoji comprising a predefined association of an emoji component with an audio component to be selected, generating the soundmoji menu further comprises generating the soundmoji menu comprising the one or more soundmojis identified based on the determined content;
-receiving a second input associated with a selection of a first icon of the one or more icons, the first icon associated with a first soundmoji; identifying a first timestamp of the first content item associated with the selection of the first icon; receiving an indication of a second timestamp of the first content item and a second soundmoji; updating a user interface element associated with the first content item to indicate the second soundmoji when the first content item is being generated for output at the second timestamp; receiving a query comprising an identification of a third soundmoji; transmitting the query to a second computing device; receiving an identification of a plurality of content items, wherein the plurality of content items is based on the query and an aggregated number of times the identified third soundmoji has been selected for each content item of the plurality of content items; generating, for output, a representation of the plurality of content items; and receiving a selection of a second content item from the plurality of content items.
Although the conflicting claims are not identical, claims 1 and 2 of U.S. Patent No. 12,050,839 contains every element of claims 31-48 of the instant application except for some obvious variations of “ identifying a subset of soundmojis from a plurality of soundmojis that are contextually related to the content of the content item”, “wherein a volume level of the audio component is controllable independently of a volume level of an audio component of the content item”, and limitations of claims 32-39 and 41-49. However, the obvious variations are being taught by Swerdlow, Muenster and Franco as shown in the rejection below. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the teaching of Swerdlow, Muenster and Franco in the invention of U.S. Patent No. 12,050,839 because doing so would enhance user experience by improving the communication of viewer reactions and emotions by providing contextually relevant one or more emojis based on key moment and doing so would enhance operability by allowing the user the flexibility to control one or more audio components .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 31-35, 37-38, 40-44, and 46-47 is/are rejected under 35 U.S.C. 103 as being unpatentable over Swerdlow et al. (“Swerdlow”, Pub. No. US 2023/0308709) and Muenster et al. (“Muenster”, Pat. No. US 10,972,682), and Franco et al. (“Franco”, Pub. No. US 2021/0390148).
Per claim 31, Swerdlow teaches a method, comprising:
generating for output, at a computing device, a content item and determining content of the content item ([0064]… It should be noted that a key moment can be associated with a single time position within the media content item, a time window within the media content item, a single frame of the media content item, a subset of frames of the media content item, a chapter of the media content item, etc. In some embodiments, process 100 can identify multiple key moments in the media content item in response to determining that multiple different time windows contain a number of reactions within the time window that are greater than the threshold value of reactions. [0101]…For example, the “heart” emoji in reaction panel interface 430 can transition to present a single “crying” emoji to indicate the popular viewer reaction at that moment in the video content item. In a more particular example, the single graphical icon presented in the reaction panel interface can be animated to transition between representative graphical icons based on the aggregated viewer reaction as the media content item is being played back. It should be noted that, in instances in which the current playback position does not correspond with an identified key moment of the media content item, no default graphical icon can be presented in reaction panel interface 430);
identifying a subset of emojis from a plurality of emojis that are contextually related to the content of the content item ([0094]…In some embodiments, process 300 can continue to display the reaction panel interface within the video sharing application at 306. For example, as shown in FIG. 4A, the reaction panel interface can be initially displayed in a collapsed state with a default graphical icon for providing a reaction to the current playback position of the media content item (e.g., a single “heart” emoji). In another example, the reaction panel interface can be initially displayed in a collapsed state in which a single graphical icon is presented, where the single graphical icon can be modified to a representative graphical icon during the presentation of the media content item to indicate the aggregated viewer reaction at the current playback position of the media content item (e.g., changing a single “heart” emoji to a single “crying” emoji to indicate the popular viewer reaction at that moment in the video content item). [0100]…It should also be noted that reaction panel interface 430 can include any suitable graphic. For example, in some embodiments, a graphical icon presented within reaction panel interface 430 can be a default graphical icon selected from a menu of selectable graphical icons. For example, as shown in FIG. 4A, a “heart” emoji was selected from multiple graphical icons as the default graphical icon for presentation on reaction panel interface 430. [0101]…For example, the “heart” emoji in reaction panel interface 430 can transition to present a single “crying” emoji to indicate the popular viewer reaction at that moment in the video content item. In a more particular example, the single graphical icon presented in the reaction panel interface can be animated to transition between representative graphical icons based on the aggregated viewer reaction as the media content item is being played back. It should be noted that, in instances in which the current playback position does not correspond with an identified key moment of the media content item, no default graphical icon can be presented in reaction panel interface 43; [0107]… each of the graphical icons can be selected based on the most popular graphical icons used in the video sharing application to react to video content items. In another example, in some embodiments, a set of graphical icons from multiple sets of graphical icons can be selected for providing in the reaction panel interface based on video information (e.g., content-related information in video metadata), based on content creator information (e.g., reactions that are commonly provided in videos uploaded by the content creator), based on application information (e.g., reactions that are commonly provided in a messaging application installed on a user device), etc.);
generating a emoji menu comprising one or more icons, wherein each icon enables a emoji of the subset of emojis to be selected for output (figs. 4A-4C; [0094]…In some embodiments, process 300 can continue to display the reaction panel interface within the video sharing application at 306. For example, as shown in FIG. 4A, the reaction panel interface can be initially displayed in a collapsed state with a default graphical icon for providing a reaction to the current playback position of the media content item (e.g., a single “heart” emoji). In another example, the reaction panel interface can be initially displayed in a collapsed state in which a single graphical icon is presented, where the single graphical icon can be modified to a representative graphical icon during the presentation of the media content item to indicate the aggregated viewer reaction at the current playback position of the media content item (e.g., changing a single “heart” emoji to a single “crying” emoji to indicate the popular viewer reaction at that moment in the video content item). [0100]…It should also be noted that reaction panel interface 430 can include any suitable graphic. For example, in some embodiments, a graphical icon presented within reaction panel interface 430 can be a default graphical icon selected from a menu of selectable graphical icons. For example, as shown in FIG. 4A, a “heart” emoji was selected from multiple graphical icons as the default graphical icon for presentation on reaction panel interface 430. [0101]…For example, the “heart” emoji in reaction panel interface 430 can transition to present a single “crying” emoji to indicate the popular viewer reaction at that moment in the video content item. In a more particular example, the single graphical icon presented in the reaction panel interface can be animated to transition between representative graphical icons based on the aggregated viewer reaction as the media content item is being played back. It should be noted that, in instances in which the current playback position does not correspond with an identified key moment of the media content item, no default graphical icon can be presented in reaction panel interface 43; [0107]… each of the graphical icons can be selected based on the most popular graphical icons used in the video sharing application to react to video content items. In another example, in some embodiments, a set of graphical icons from multiple sets of graphical icons can be selected for providing in the reaction panel interface based on video information (e.g., content-related information in video metadata), based on content creator information (e.g., reactions that are commonly provided in videos uploaded by the content creator), based on application information (e.g., reactions that are commonly provided in a messaging application installed on a user device), etc.); and
generating for output, at the computing device, the soundmoji menu while the content item is being generated for output (figs. 4A-4C; [0094]…In some embodiments, process 300 can continue to display the reaction panel interface within the video sharing application at 306. For example, as shown in FIG. 4A, the reaction panel interface can be initially displayed in a collapsed state with a default graphical icon for providing a reaction to the current playback position of the media content item (e.g., a single “heart” emoji). In another example, the reaction panel interface can be initially displayed in a collapsed state in which a single graphical icon is presented, where the single graphical icon can be modified to a representative graphical icon during the presentation of the media content item to indicate the aggregated viewer reaction at the current playback position of the media content item (e.g., changing a single “heart” emoji to a single “crying” emoji to indicate the popular viewer reaction at that moment in the video content item). [0100]…It should also be noted that reaction panel interface 430 can include any suitable graphic. For example, in some embodiments, a graphical icon presented within reaction panel interface 430 can be a default graphical icon selected from a menu of selectable graphical icons. For example, as shown in FIG. 4A, a “heart” emoji was selected from multiple graphical icons as the default graphical icon for presentation on reaction panel interface 430. [0101]…For example, the “heart” emoji in reaction panel interface 430 can transition to present a single “crying” emoji to indicate the popular viewer reaction at that moment in the video content item. In a more particular example, the single graphical icon presented in the reaction panel interface can be animated to transition between representative graphical icons based on the aggregated viewer reaction as the media content item is being played back. It should be noted that, in instances in which the current playback position does not correspond with an identified key moment of the media content item, no default graphical icon can be presented in reaction panel interface 43; [0107]… each of the graphical icons can be selected based on the most popular graphical icons used in the video sharing application to react to video content items. In another example, in some embodiments, a set of graphical icons from multiple sets of graphical icons can be selected for providing in the reaction panel interface based on video information (e.g., content-related information in video metadata), based on content creator information (e.g., reactions that are commonly provided in videos uploaded by the content creator), based on application information (e.g., reactions that are commonly provided in a messaging application installed on a user device), etc.).
Swerdlow does not specifically teach soundmoji comprises a predefined association of an emoji component with an audio component.
However, Muenster teaches soundmoji comprises a predefined association of an emoji component with an audio component (figs. 3-5 and 12-13; col. 4, lines 42-49… a collection of predefined virtual audio stickers may be displayed in an audio sticker panel. In some embodiments, a user may listen to the audio clip of a virtual audio sticker in the virtual audio sticker panel by simply selecting (e.g., touching) that virtual audio sticker. Users may define their own custom virtual audio stickers by pairing images and audio clips, both of which may be predefined or supplied by the user. Col. 10, lines 51-63… a video may be edited to include multiple virtual audio stickers, and two or more of the virtual audio stickers may overlap in time. In the example of FIG. 12, the video represented by timeline 1200 includes three virtual audio stickers vas1, vas2, and vas3. The video begins at time W. The first virtual audio sticker vas1 plays first, beginning at time t1 and concluding at time t2. The second virtual audio sticker vas2 plays next, beginning at time t3 and concluding at time t6. The third virtual audio sticker vas3 starts last, beginning at time t4 and concluding at time t5. The video ends at time t7. But as noted above, the video may repeat, for example in any one of a number of repeating notes that may be selected by a user. Col. 11, lines 4-14… FIG. 13 shows example properties 1300 of a virtual audio sticker, according to some embodiments of the disclosed technologies. Referring to FIG. 13, the example properties 1300 may include both fixed properties 1302 and variable properties 1304. For example, the fixed properties 1302 may include an ID 1306, an image file 1308, and an audio file 1310. The ID 1306 may be a text string for internal use by the tool in referring to particular virtual audio stickers. The image file 1308 and audio file 1310 may be represented by pointers to files for the image and audio clip of the virtual audio sticker.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the teaching of Muenster in the invention of Swerdlow to include audio with emoji because doing so would enhance user experience by improving the communication of viewer reactions and emotions by providing audio components associated with one or more emojis.
The modified Swerdlow does not specifically teach wherein a volume level of the audio component is controllable independently of a volume level of an audio component of the content item. However, Muenster and Franco teach a volume level of the audio component is controllable independently of a volume level of an audio component of the content item. (Munenster, col. 4, lines 32-41… the audio clip of the virtual audio sticker may replace the audio track of the video. In other embodiments, the audio clip and the audio track may play together. In some embodiments, the audio level of the audio track of the video may be reduced while the audio clip of the virtual audio sticker plays. In still other embodiments, the virtual audio sticker may include an audio effect that modifies the audio of the video. For example, the audio effect may cause a voice in the video to echo, sound robotic, and the like. Franco, [0029]… the user may be enabled to provide volume settings 122, based on a scale of 0-10, with 0 being no sound, and 10 being the loudest sound level, for each of the sound types 121. Alternatively, the volume settings may be associated with a different scale, such as, for example, decibel (dB) levels. As shown, and for example, the user has selected a volume setting of 3 for sounds associated with an avatar and a volume setting of 7 for sounds associated with smileys. The user also may be enabled to provide mute settings 123 for each of the sound types 121. [0046]… Output controls window 320 includes audio controls 330, display controls 340, and application controls 350. Audio controls 330 may enable the user to provide particular sound types 331 (e.g., sounds associated with video embedded in the web page or advertisements shown in the web page), as well as volume settings 332 (e.g., on a scale of 0 to 10, as discussed above) and mute settings 333 (e.g., an amount of time after which the web page is loaded that the particular sounds may be muted) to be associated with the particular sound types. For example, and as shown, the user has indicated that video sounds may be played at a volume setting of 8 and are never muted, while sounds associated with advertisements are to be played at a volume setting of 5 and muted 10 minutes after the web page is initially loaded)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the further teaching of Muenster and the teaching of Franco in the invention of the modified Swerdlow to volume setting for different sound types because doing so would enhance operability by allowing the flexibility of controlling volume settings for one or more audio types.
Per claim 32, the modified Swerdlow teaches the method of claim 31, wherein generating the soundmoji menu is based at least on receiving an interaction with an interface element for selecting the soundmoji menu (Swerdlow, figs. 4A-4C; [0103]… In some embodiments, the default graphical icon can be associated with a corresponding reaction and, in response to receiving a user selection of the default graphical icon on reaction panel interface 430, the corresponding reaction can be associated with the current playback position of the media content item (e.g., a “love” reaction by the user at t=1:25 of the video content item). Alternatively, in some embodiments, in response to receiving a user selection of the default graphical icon, reaction panel interface 430 can transition from a collapsed state shown in FIG. 4A to an expanded state 440 shown in FIG. 4B. [0104]… a user input that is deemed to be a short press of the default graphical icon of reaction panel interface 430 can cause the corresponding reaction (e.g., a “love” reaction) to be provided in connection with the current playback position of the media content item and a user input that is deemed to be a long press of the default graphical icon of reaction panel interface 430 can cause reaction panel interface 430 to transition from the collapsed state 430 shown in FIG. 4A to an expanded state 440 as shown in FIG. 4B).
Per claim 33, Swerdlow teaches the method of claim 31, wherein the subset of soundmojis are contextually related to a genre of the content item (Swerdlow, [0060]… In a more particular example, the video content item can be input into one or more risk classifiers that determine (i) whether the video content item is present on a block list, (ii) whether the video content item is associated with a channel identifier or a user identifier that is present on a block list, (iii) whether the video content item is associated with particular content types (e.g., news content), (iv) whether the video content item is associated with topics or other entities that are deemed sensitive or objectionable (e.g., violent content, objectionable language, hate speech, adult content, and/or any other suitable type of objectionable content), etc. In continuing with this example, in response to determining that the one or more risk classifiers have deemed the video content item as having a risk rating greater than a particular threshold value (e.g., a high risk video content item based on the video content item being associated with a news content type), process 100 can end in which key moments are not to be identified for the video content item associated with the high risk rating. Alternatively, process 100 can determine that the video content item is eligible for identifying key moments within the video content item based on viewer reactions that were received during the presentation of the video content item. [0064]… It should be noted that a key moment can be associated with a single time position within the media content item, a time window within the media content item, a single frame of the media content item, a subset of frames of the media content item, a chapter of the media content item, etc. In some embodiments, process 100 can identify multiple key moments in the media content item in response to determining that multiple different time windows contain a number of reactions within the time window that are greater than the threshold value of reactions. [0101]…For example, the “heart” emoji in reaction panel interface 430 can transition to present a single “crying” emoji to indicate the popular viewer reaction at that moment in the video content item. In a more particular example, the single graphical icon presented in the reaction panel interface can be animated to transition between representative graphical icons based on the aggregated viewer reaction as the media content item is being played back. It should be noted that, in instances in which the current playback position does not correspond with an identified key moment of the media content item, no default graphical icon can be presented in reaction panel interface 430).
Per claim 34, Swerdlow The method of claim 31, wherein the subset of soundmojis are contextually related to one or more of a title of the content item, content type of the content item, characters in the content item, or actors in the content item (Swerdlow, [0059]…As another example, reactions can be included in the metadata associated with the media content item. In such an example, process 100 can access, request, and/or retrieve the reactions associated with the media content item concurrently with the metadata associated with the media content item (e.g., the title of the media content item, a description of the media content item, the content creator of the media content item, etc.). [0060]… In a more particular example, the video content item can be input into one or more risk classifiers that determine (i) whether the video content item is present on a block list, (ii) whether the video content item is associated with a channel identifier or a user identifier that is present on a block list, (iii) whether the video content item is associated with particular content types (e.g., news content), (iv) whether the video content item is associated with topics or other entities that are deemed sensitive or objectionable (e.g., violent content, objectionable language, hate speech, adult content, and/or any other suitable type of objectionable content), etc. In continuing with this example, in response to determining that the one or more risk classifiers have deemed the video content item as having a risk rating greater than a particular threshold value (e.g., a high risk video content item based on the video content item being associated with a news content type), process 100 can end in which key moments are not to be identified for the video content item associated with the high risk rating. Alternatively, process 100 can determine that the video content item is eligible for identifying key moments within the video content item based on viewer reactions that were received during the presentation of the video content item. [0064]… It should be noted that a key moment can be associated with a single time position within the media content item, a time window within the media content item, a single frame of the media content item, a subset of frames of the media content item, a chapter of the media content item, etc. In some embodiments, process 100 can identify multiple key moments in the media content item in response to determining that multiple different time windows contain a number of reactions within the time window that are greater than the threshold value of reactions. [0107]… a set of graphical icons from multiple sets of graphical icons can be selected for providing in the reaction panel interface based on video information (e.g., content-related information in video metadata), based on content creator information (e.g., reactions that are commonly provided in videos uploaded by the content creator), based on application information (e.g., reactions that are commonly provided in a messaging application installed on a user device), etc. In yet another example, in some embodiments, the content creator that uploaded the media content item can select the graphical icons or the set of graphical icons for inclusion in the reaction panel interface. ).
Per claim 35, the modified Swerdlow teaches the method of claim 31, wherein the content of the content item is determined using metadata associated with the content item (Swerdlow, [0059]…As another example, reactions can be included in the metadata associated with the media content item. In such an example, process 100 can access, request, and/or retrieve the reactions associated with the media content item concurrently with the metadata associated with the media content item (e.g., the title of the media content item, a description of the media content item, the content creator of the media content item, etc.). [0107]… a set of graphical icons from multiple sets of graphical icons can be selected for providing in the reaction panel interface based on video information (e.g., content-related information in video metadata), based on content creator information (e.g., reactions that are commonly provided in videos uploaded by the content creator), based on application information (e.g., reactions that are commonly provided in a messaging application installed on a user device), etc. In yet another example, in some embodiments, the content creator that uploaded the media content item can select the graphical icons or the set of graphical icons for inclusion in the reaction panel interface.)
Per claim 37, Swerdlow the method of claim 31, wherein identifying the subset of soundmojis that are contextually related to the content of the content item comprises: identifying a category of each of the plurality of soundmojis; and selecting the subset of soundmojis having a category contextually related to the content of the content item (Swerdlow, [0107]… a set of graphical icons from multiple sets of graphical icons can be selected for providing in the reaction panel interface based on video information (e.g., content-related information in video metadata), based on content creator information (e.g., reactions that are commonly provided in videos uploaded by the content creator), based on application information (e.g., reactions that are commonly provided in a messaging application installed on a user device), etc. In yet another example, in some embodiments, the content creator that uploaded the media content item can select the graphical icons or the set of graphical icons for inclusion in the reaction panel interface. ).
Per claim 38, the modified Swerdlow teaches the method of claim 31, further comprising: receiving an interaction with an icon of the one or more icons, the icon for enabling a first soundmoji, identifying a first timestamp of the content item associated with the interaction with the icon, receiving an indication of a second timestamp of the content item and a second soundmoji, and updating a user interface element associated with the content item to indicate the second soundmoji when the content item is being generated for output at the second timestamp (Swerdlow, [0056]… FIGS. 3-5, reactions can be an expression from a viewer that indicates the viewer's feeling(s) at a particular portion of the media content item. For example, in some embodiments, a reaction can be received when a viewer selects a graphical icon (e.g., an emoticon or an emoji) from a menu of graphical icons at a particular timestamp while a video content item is being played back. [0059]… The reaction information from the reactions database can include a reaction identifier that identifies the reaction; a reaction timestamp that identifies the timestamp that the reaction was received from a viewer, a frame number that identifies the frame that the reaction was received from a viewer, and/or any other suitable reference to a time point within the media content item when the reaction was received from a viewer; a user identifier that identifies a user account associated with a user or a group of users that provided the reaction, a creation timestamp that indicates the time when the reaction was created, a media content identifier that identifies the media content item or a portion of the media content item relating to the reaction, etc. [0062]… In some embodiments, process 100 can, at 106, determine that a particular number of reactions has been received at a particular timestamp and/or time window (e.g., a duration of time with a starting timestamp and ending timestamp, a duration of time with a starting frame number and an ending frame number). For example, process 100 can determine that greater than a threshold number of reactions (e.g., 10 reactions) has been received at a particular time position within the media content item. Muenster, Col. 10, lines 51-63… a video may be edited to include multiple virtual audio stickers, and two or more of the virtual audio stickers may overlap in time. In the example of FIG. 12, the video represented by timeline 1200 includes three virtual audio stickers vas1, vas2, and vas3. The video begins at time W. The first virtual audio sticker vas1 plays first, beginning at time t1 and concluding at time t2. The second virtual audio sticker vas2 plays next, beginning at time t3 and concluding at time t6. The third virtual audio sticker vas3 starts last, beginning at time t4 and concluding at time t5. The video ends at time t7. But as noted above, the video may repeat, for example in any one of a number of repeating notes that may be selected by a user).
Claims 40-44 and 46-47 are rejected under the same rationale as claims 31-35 and 37-38 respectively.
Claim(s) 36 and 45 is/are rejected under 35 U.S.C. 103 as being unpatentable over Swerdlow et al. (“Swerdlow”, Pub. No. US 2023/0308709) and Muenster et al. (“Muenster”, Pat. No. US 10,972,682), Franco et al. (“Franco”, Pub. No. US 2021/0390148), and Faulkner et al. (“Faulkner”, Pub. No. US 2018/0331842).
Per claim 36, the modified Swerdlow teaches the method of claim 31, wherein the content of the content item is determined using a trained machine learning algorithm.
However, Faulkner teaches the content of the content item is determined using a trained machine learning algorithm ([0093]… the analysis performed to determine a threshold number of notable events that occur within a threshold time period can focus on activity that is of interest to the user. For example, machine learning algorithm(s) can generate and/or update machine-learned parameters to determine which types of activity hotspots are more important to a user than others and/or to determine reasons why the types of activity hotspots are more important. The machine-learned parameters can be updated based on user feedback (e.g., whether the user performs a computer interaction with an activity hotspot representation on the interactive timeline). )
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the teaching of Faulkner in the invention of Swerdlow to include machine learning model configure to analyze content of a content item because doing so would enhance user experience to provide an interactive timeline activity hotspots that are likely of interest to the user.
Claim 45 is rejected under the same rationale as claim 36.
Claim(s) 39 and 48 is/are rejected under 35 U.S.C. 103 as being unpatentable over Swerdlow et al. (“Swerdlow”, Pub. No. US 2023/0308709) and Muenster et al. (“Muenster”, Pat. No. US 10,972,682), Franco et al. (“Franco”, Pub. No. US 2021/0390148), and Prasad et al. (“Prasad”, Pat. No. US 2021/0382610).
Per claim 39, the modified Swerdlow teaches the method of claim 31, further comprising: receiving an indication of a timestamp of the content item and a particular soundmoji and updating a user interface element associated with the content item to indicate the particular soundmoji when the content item is being generated for output at the timestamp (Swerdlow, [0056]… FIGS. 3-5, reactions can be an expression from a viewer that indicates the viewer's feeling(s) at a particular portion of the media content item. For example, in some embodiments, a reaction can be received when a viewer selects a graphical icon (e.g., an emoticon or an emoji) from a menu of graphical icons at a particular timestamp while a video content item is being played back. [0059]… The reaction information from the reactions database can include a reaction identifier that identifies the reaction; a reaction timestamp that identifies the timestamp that the reaction was received from a viewer, a frame number that identifies the frame that the reaction was received from a viewer, and/or any other suitable reference to a time point within the media content item when the reaction was received from a viewer; a user identifier that identifies a user account associated with a user or a group of users that provided the reaction, a creation timestamp that indicates the time when the reaction was created, a media content identifier that identifies the media content item or a portion of the media content item relating to the reaction, etc. [0062]… In some embodiments, process 100 can, at 106, determine that a particular number of reactions has been received at a particular timestamp and/or time window (e.g., a duration of time with a starting timestamp and ending timestamp, a duration of time with a starting frame number and an ending frame number). For example, process 100 can determine that greater than a threshold number of reactions (e.g., 10 reactions) has been received at a particular time position within the media content item. Muenster, Col. 10, lines 51-63… a video may be edited to include multiple virtual audio stickers, and two or more of the virtual audio stickers may overlap in time. In the example of FIG. 12, the video represented by timeline 1200 includes three virtual audio stickers vas1, vas2, and vas3. The video begins at time W. The first virtual audio sticker vas1 plays first, beginning at time t1 and concluding at time t2. The second virtual audio sticker vas2 plays next, beginning at time t3 and concluding at time t6. The third virtual audio sticker vas3 starts last, beginning at time t4 and concluding at time t5. The video ends at time t7. But as noted above, the video may repeat, for example in any one of a number of repeating notes that may be selected by a user).
However, the modified Swerdlow does not teach identifying an advertisement based on the particular soundmoji and generating for output, at the computing device, the advertisement with the content item.
However, Prasad teaches identifying an advertisement based on the particular soundmoji and generating for output, at the computing device, the advertisement with the content item ([0052]… The media content comprises, for example, textual content, image content, audio content, video content, audiovisual content, multimedia content, etc., or any combination thereof. In an embodiment, the media content comprises branding data and/or advertising data for reconstruction of a digital pictorial image such as a product image into a branded product image. In this embodiment, the image reconstruction module 112 maps the digital pictorial image with brands selling a product in a category. For example, the image reconstruction module 112 enlarges the size of a chocolate emoji and inserts a brand logo to the chocolate emoji to generate a digital sticker.)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the teaching of Prasad in the invention of Swerdlow to include the media content comprises branding data and/or advertising data for reconstruction of a digital pictorial image, because doing so would enhance brand recognition to promote products and/or services.
Claim 48 is rejected under the same rationale as claim 39.
Claim(s) 49 and 50 is/are rejected under 35 U.S.C. 103 as being unpatentable over Leydon et al. (“Leydon”, Pub. No. US 2013/0159919) and Muenster et al. (“Muenster”, Pat. No. US 10,972,682)
Per claim 49, A method comprising:
generating for output, at a computing device, a content item (fig. 5, content item 502; fig. 6, content item: message input 604)
determining first portion of the content item and a second portion of the content item, wherein the second portion is different than the first portion (Fig. 5, first portion “so much fun”, second portion “your city”; Fig. 6; First portion: “football”; second portion: “field goal”);
analyzing the first portion of the content item, based at least in part on analyzing the first portion of the content item, identifying a first subset of emojis from a plurality of emojis that are contextually related to the first portion of the content item (Figs 5, emojis/graphical icons 510 and 508; and fig. 6, emojis/graphical icons 614 and 622; [0091] As the user prepares a message in the message input field 504, various embodiments may suggest emoticons for entry at the current position of the input cursor 516. For example, as the user writes "It was so much fun" in the message input field 504, the embodiment may suggest a "smiley face" graphical emoticon 510 for entry into the input field 504 based on the embodiment's analysis of the segment of interest 512, which recites "so much fun." The embodiment may suggest the "smiley face" graphical emoticon 510 based on an association between the "smiley face" graphical emoticon 510 and the context of the segment of interest 512. Once the user selects the "smiley face" graphical emoticon 510, the embodiment may enter the "smiley face" graphical emoticon 510 into the message input field 504. [0097]… As the user prepares a message in the message input field 604, various embodiments may suggest emoticons for entry at the current position of the input cursor 610. For instance, as the user writes "I never would have thought the football" in the message input field 604, the embodiment may suggest a "football" graphical emoticon 614 for entry into the input field 604 based on the embodiment's analysis of the segment of interest 612, which recites "football." The embodiment may suggest the "football" graphical emoticon 614 based on an association between the "football" graphical emoticon 614 and the context of the segment of interest 612. Once the user selects the "football" graphical emoticon 614, the embodiment may enter the "football" graphical emoticon 614 into the message input field 604.);
analyzing the second portion of the content item, based at least in part on analyzing the second portion of the content item, identifying a second subset of emojis from the plurality of emojis that are contextually related to the second portion of the content item, the second subset comprising at least one emoji not in the first subset (Figs 5, emojis/graphical icons 510 and 508; and fig. 6, emojis/graphical icons 614 and 622; [0092]… Likewise, as the user writes "Thanks again for showing me around your city" in the message input field 504, the embodiment may suggest a plurality of graphical emoticons 506 based on the context analysis of the segment of interest 514. As noted herein, the embodiment may present the suggested, graphical emoticons 506 by displaying the graphical emoticons 506 in a callout box 508 positioned at or near the current position of the input cursor 516. Based on analysis of the segment of interest 514, which recites "your city," the embodiment may suggest the graphical emoticons 506, which relate to cities. [0098]… Similarly, as the user writes "The 50-yard field goal" in the message input field 604, the embodiment may suggest a plurality of "field goal" graphical emoticons based on the context analysis of the segment of interest 608. In particular, the embodiment may present the "field goal" graphical emoticons for entry selection by displaying the graphical emoticons in the emoticon menu 622, which may be displayed as part of the on-screen keyboard 606);
generating a first emoji menu comprising a first set of one or more icons for the first portion of the content item, wherein each icon enables a emoji of the first subset of emojis to be selected for output (Figs 5, emojis/graphical icons 510 and 508; and fig. 6, emojis/graphical icons 614 and 622; [0091] As the user prepares a message in the message input field 504, various embodiments may suggest emoticons for entry at the current position of the input cursor 516. For example, as the user writes "It was so much fun" in the message input field 504, the embodiment may suggest a "smiley face" graphical emoticon 510 for entry into the input field 504 based on the embodiment's analysis of the segment of interest 512, which recites "so much fun." The embodiment may suggest the "smiley face" graphical emoticon 510 based on an association between the "smiley face" graphical emoticon 510 and the context of the segment of interest 512. Once the user selects the "smiley face" graphical emoticon 510, the embodiment may enter the "smiley face" graphical emoticon 510 into the message input field 504. [0097]… As the user prepares a message in the message input field 604, various embodiments may suggest emoticons for entry at the current position of the input cursor 610. For instance, as the user writes "I never would have thought the football" in the message input field 604, the embodiment may suggest a "football" graphical emoticon 614 for entry into the input field 604 based on the embodiment's analysis of the segment of interest 612, which recites "football." The embodiment may suggest the "football" graphical emoticon 614 based on an association between the "football" graphical emoticon 614 and the context of the segment of interest 612. Once the user selects the "football" graphical emoticon 614, the embodiment may enter the "football" graphical emoticon 614 into the message input field 604. );
generating a second emoji menu comprising a second set of one or more icons for the second portion of the content item, wherein each icon enables a emoji of the second subset of emojis to be selected for output (Figs 5, emojis/graphical icons 510 and 508; and fig. 6, emojis/graphical icons 614 and 622; [0092]… Likewise, as the user writes "Thanks again for showing me around your city" in the message input field 504, the embodiment may suggest a plurality of graphical emoticons 506 based on the context analysis of the segment of interest 514. As noted herein, the embodiment may present the suggested, graphical emoticons 506 by displaying the graphical emoticons 506 in a callout box 508 positioned at or near the current position of the input cursor 516. Based on analysis of the segment of interest 514, which recites "your city," the embodiment may suggest the graphical emoticons 506, which relate to cities. [0098]… Similarly, as the user writes "The 50-yard field goal" in the message input field 604, the embodiment may suggest a plurality of "field goal" graphical emoticons based on the context analysis of the segment of interest 608. In particular, the embodiment may present the "field goal" graphical emoticons for entry selection by displaying the graphical emoticons in the emoticon menu 622, which may be displayed as part of the on-screen keyboard 606); and
generating for output, at the computing device, the first emoji menu while the first portion of the content item is being generated for output and the second emoji menu while the second portion of the content item is being generated for output ((Figs 5, emojis/graphical icons 510 and 508; and fig. 6, emojis/graphical icons 614 and 622; [0091]… As the user prepares a message in the message input field 504, various embodiments may suggest emoticons for entry at the current position of the input cursor 516. For example, as the user writes "It was so much fun" in the message input field 504, the embodiment may suggest a "smiley face" graphical emoticon 510 for entry into the input field 504 based on the embodiment's analysis of the segment of interest 512, which recites "so much fun." The embodiment may suggest the "smiley face" graphical emoticon 510 based on an association between the "smiley face" graphical emoticon 510 and the context of the segment of interest 512. Once the user selects the "smiley face" graphical emoticon 510, the embodiment may enter the "smiley face" graphical emoticon 510 into the message input field 504. [0092]… Likewise, as the user writes "Thanks again for showing me around your city" in the message input field 504, the embodiment may suggest a plurality of graphical emoticons 506 based on the context analysis of the segment of interest 514. As noted herein, the embodiment may present the suggested, graphical emoticons 506 by displaying the graphical emoticons 506 in a callout box 508 positioned at or near the current position of the input cursor 516. Based on analysis of the segment of interest 514, which recites "your city," the embodiment may suggest the graphical emoticons 506, which relate to cities. [0097]… As the user prepares a message in the message input field 604, various embodiments may suggest emoticons for entry at the current position of the input cursor 610. For instance, as the user writes "I never would have thought the football" in the message input field 604, the embodiment may suggest a "football" graphical emoticon 614 for entry into the input field 604 based on the embodiment's analysis of the segment of interest 612, which recites "football." The embodiment may suggest the "football" graphical emoticon 614 based on an association between the "football" graphical emoticon 614 and the context of the segment of interest 612. Once the user selects the "football" graphical emoticon 614, the embodiment may enter the "football" graphical emoticon 614 into the message input field 604. [0098]…Similarly, as the user writes "The 50-yard field goal" in the message input field 604, the embodiment may suggest a plurality of "field goal" graphical emoticons based on the context analysis of the segment of interest 608. In particular, the embodiment may present the "field goal" graphical emoticons for entry selection by displaying the graphical emoticons in the emoticon menu 622, which may be displayed as part of the on-screen keyboard 606.).
Leydon does not specifically teach soundmoji comprises a predefined association of an emoji component with an audio component.
However, Muenster teaches soundmoji comprises a predefined association of an emoji component with an audio component (figs. 3-5 and 12-13; col. 4, lines 42-49… a collection of predefined virtual audio stickers may be displayed in an audio sticker panel. In some embodiments, a user may listen to the audio clip of a virtual audio sticker in the virtual audio sticker panel by simply selecting (e.g., touching) that virtual audio sticker. Users may define their own custom virtual audio stickers by pairing images and audio clips, both of which may be predefined or supplied by the user. Col. 10, lines 51-63… a video may be edited to include multiple virtual audio stickers, and two or more of the virtual audio stickers may overlap in time. In the example of FIG. 12, the video represented by timeline 1200 includes three virtual audio stickers vas1, vas2, and vas3. The video begins at time W. The first virtual audio sticker vas1 plays first, beginning at time t1 and concluding at time t2. The second virtual audio sticker vas2 plays next, beginning at time t3 and concluding at time t6. The third virtual audio sticker vas3 starts last, beginning at time t4 and concluding at time t5. The video ends at time t7. But as noted above, the video may repeat, for example in any one of a number of repeating notes that may be selected by a user. Col. 11, lines 4-14… FIG. 13 shows example properties 1300 of a virtual audio sticker, according to some embodiments of the disclosed technologies. Referring to FIG. 13, the example properties 1300 may include both fixed properties 1302 and variable properties 1304. For example, the fixed properties 1302 may include an ID 1306, an image file 1308, and an audio file 1310. The ID 1306 may be a text string for internal use by the tool in referring to particular virtual audio stickers. The image file 1308 and audio file 1310 may be represented by pointers to files for the image and audio clip of the virtual audio sticker.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the teaching of Muenster in the invention of Leydon to include audio with emoji because doing so would enhance user experience by improving the communication of viewer reactions and emotions with associated audio clips.
Per claim 50, the modified Leydon teaches The method of claim 49, further comprising identifying a category of each of the plurality of soundmojis, wherein: identifying the first subset of soundmojis that are contextually related to the first content of the content item comprises selecting soundmojis having a first category contextually related to the first content (Figs 5, emojis/graphical icons 510 and 508; and fig. 6, emojis/graphical icons 614 and 622; [0091] As the user prepares a message in the message input field 504, various embodiments may suggest emoticons for entry at the current position of the input cursor 516. For example, as the user writes "It was so much fun" in the message input field 504, the embodiment may suggest a "smiley face" graphical emoticon 510 for entry into the input field 504 based on the embodiment's analysis of the segment of interest 512, which recites "so much fun." The embodiment may suggest the "smiley face" graphical emoticon 510 based on an association between the "smiley face" graphical emoticon 510 and the context of the segment of interest 512. Once the user selects the "smiley face" graphical emoticon 510, the embodiment may enter the "smiley face" graphical emoticon 510 into the message input field 504. [0097]… As the user prepares a message in the message input field 604, various embodiments may suggest emoticons for entry at the current position of the input cursor 610. For instance, as the user writes "I never would have thought the football" in the message input field 604, the embodiment may suggest a "football" graphical emoticon 614 for entry into the input field 604 based on the embodiment's analysis of the segment of interest 612, which recites "football." The embodiment may suggest the "football" graphical emoticon 614 based on an association between the "football" graphical emoticon 614 and the context of the segment of interest 612. Once the user selects the "football" graphical emoticon 614, the embodiment may enter the "football" graphical emoticon 614 into the message input field 604.); and
identifying the second subset of soundmojis that are contextually related to the second content of the content item comprises selecting soundmojis having a second category contextually related to the second content (Figs 5, emojis/graphical icons 510 and 508; and fig. 6, emojis/graphical icons 614 and 622; [0092]… Likewise, as the user writes "Thanks again for showing me around your city" in the message input field 504, the embodiment may suggest a plurality of graphical emoticons 506 based on the context analysis of the segment of interest 514. As noted herein, the embodiment may present the suggested, graphical emoticons 506 by displaying the graphical emoticons 506 in a callout box 508 positioned at or near the current position of the input cursor 516. Based on analysis of the segment of interest 514, which recites "your city," the embodiment may suggest the graphical emoticons 506, which relate to cities. [0098]… Similarly, as the user writes "The 50-yard field goal" in the message input field 604, the embodiment may suggest a plurality of "field goal" graphical emoticons based on the context analysis of the segment of interest 608. In particular, the embodiment may present the "field goal" graphical emoticons for entry selection by displaying the graphical emoticons in the emoticon menu 622, which may be displayed as part of the on-screen keyboard 606).
Response to Arguments
Applicant’s arguments with respect to the amendment have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Inquiries
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THANH T VU whose telephone number is (571)272-4073. The examiner can normally be reached M-F: 7AM - 3:30PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Fred Ehichioya can be reached at (571) 272-4034. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/THANH T VU/Primary Examiner, Art Unit 2179
