DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
2. A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 12/05/2025 has been entered.
Response to Amendment
3. Acknowledgement is made of amendment filed on December 05, 2025, in which claims 1, 8 and 15 are amended, and claims 1-20 are still pending.
Response to Arguments
4. Applicant's arguments, filed on December 05, 2025, with respect to Claims 1, 8 and 15 have been fully considered but they are not persuasive.
5. With regards to arguments for independent claims 1, 8 and 15, applicants argue that Zadeh et al. (US 2022/0261128 A1), Clawges et al. (US 2019/0371267 A1) and Voss (US 2016/0227134 A1) fail to disclose generate, for individual overlaid content items, respective improved color values; use of outputs of a machine learning model to determine an improve color value for overlaid content items within a video. The examiner respectfully agrees and also moot in view of the new grounds of rejections regarding claim 1, 8 and 15, since in Rykhliuk et al. (US 2021/0390745 A1) teaches (“the augmentation system 206 provides functions related to the generation and publishing of media overlays for messages processed by the messaging system 100. The augmentation system 206 operatively supplies a media overlay or augmentation (e.g., an image filter) to the messaging client 104 based on a geolocation of the client device 102. In another example, the augmentation system 206 operatively supplies a media overlay to the messaging client 104 based on other information, such as social network information of the user of the client device 102. A media overlay may include audio and visual content and visual effects. Examples of audio and visual content include pictures, texts, logos, animations, and sound effects. An example of a visual effect includes color overlaying.” [0037] “The custom augmentation system 124 uses machine learning to generate an image augmentation decision and provides the image augmentation as input to an augmented reality content item.” [0042] “transformations changing some areas of an object using its elements can be performed by calculating characteristic points for each element of an object and generating a mesh based on the calculated characteristic points. Points are generated on the mesh, and then various areas based on the points are generated. The elements of the object are then tracked by aligning the area for each element with a position for each of the at least one element, and properties of the areas can be modified based on the request for modification, thus transforming the frames of the video stream. Depending on the specific request for modification properties of the mentioned areas can be transformed in different ways. Such modifications may involve changing color of areas;” [0058] “the custom augmentation system 124 generates an image augmentation decision using a machine learning model. The image augmentation decision is based on the received image. In some examples, the machine learning model is the machine learning model 606. In one example, the machine learning model 606 is used to decide whether a user has warm or cool skin tones. … The augmentation reality content item is configured to modify image content of the received image. The augmentation reality content item may be accessed from the augmentation system 206. The augmentation content item may be a real-time special effect and sound that may be added to an image or a video. For example, an augmentation reality content item may be configured to change the color of an image.” [0113-0114] “the custom augmentation system 124 may include a machine learning model 606 that is trained to analyze the received image and generate a decision on whether a person in the image has a warm skin tone or a cool skin tone. The custom augmentation system 124 associates that image augmentation decision (e.g., whether person has a warm skin tone or a cool skin tone) with an augmentation reality content item. For example, the augmentation reality content item may be configured to apply virtual makeup to the person's face. When the custom augmentation system 124 associates the image augmentation decision with the augmentation reality content item, the custom augmentation system 124 enhances the effect of the augmentation reality content item by providing the image augmentation decision to the augmentation reality content item. For example, if the image augmentation decision is that the person in the image has a warm skin tone, the augmentation reality content item will apply virtual makeup using color shades that are best suited for people with warm skin tones. If the image augmentation decision is that the person in the image has a cool skin tone, the augmentation reality content will apply virtual makeup using color shades that are best suited for people with cool skin tones.” [0116]) Rykhliuk teaches generates an image augmentation decision using a machine learning model, decide whether a user has warm or cool skin tones which the improved color value, and modify image content of the received image that includes color overlaying / changing for the result of a customized image augmentation. Therefore, Rykhliuk teaches the arguments of the limitations for claims 1, 8 and 15 as it is recited.
Claim Rejections - 35 USC § 103
6. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
7. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
8. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
9. Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zadeh et al. (US 2022/0261128 A1) in view of Clawges et al. (US 2019/0371267 A1) and Rykhliuk et al. (US 2021/0390745 A1)).
10. With reference to claim 1, Zadeh teaches A method comprising: receiving, via a user interface of a mobile application, a video for upload, wherein the video comprises a plurality of frames each having primary video content one or more overlaid content items including additional information about the primary video content; (“Described herein are systems and methods that search for objects, faces, or other items selected by users in media content items to find media content items (such as videos or frames of video) where the media content preferred by users appear.” [0004] “The user may upload the set of media content items to the media detection system 140. Alternatively, the user may browse media content items stored in the media detection system 140 and select from among the browsed media content items. For example, the user browses media content items used to train detectors. The user can upload and/or select the set of media content items via one or more user interfaces provided by the user interface module 150. Via the one or more user interfaces, the user can also identify portions of media content items such as a video segment, a section of a video frame, or a section of an image that include the visual features.” [0030] “the user interface module 150 generates user interfaces that allow users to configure example media content items (e.g., images, videos, portions of images, portions of videos such as video segments, portions of video frames) for training and/or updating detectors.” [0039] “the video frame 452 is presented in the display area 451 of the user interface 450. A user interface element 458 (e.g., a box) indicates to the user the portion of the video frame 452 that is determined to include the preferred content. The user interface element 458 is overlaid over the video frame 452.” [0076]) Zadeh also teaches generating data related to the video based on a location of each overlaid content item of the one or more overlaid content items within a corresponding frame and an initial color of each overlaid content item within the corresponding frame; providing the data related to the video as input to a machine learning model; (“the user interface module 150 generates user interfaces that allow users to configure example media content items (e.g., images, videos, portions of images, portions of videos such as video segments, portions of video frames) for training and/or updating detectors.” [0039] “The user interface element module 150 further generates user interface elements (e.g., a border box) configured to surround the portion of the video frame where the preferred content is determined as being present. The user interface elements are overlaid on top of the region displaying the video. The location and dimension of the user interface elements can be determined according to the search engine's output. The user interface elements track the preferred content over time. To improve user experience, the user interface element module 150 may interpolate locations of the user interface elements across consecutive video frames to avoid abrupt location changes.” [0041] “the detector management module 152 creates the detectors by training one or more machine learning models using training data. The training data include example media content items provided by a user. The example media content items can be obtained from a variety of sources. For example, a user specifies media content items by selecting media content items stored in the media content store 158, uploading media content items to the media detection system 140, selecting portions of a media content item, providing locations of media content items to the media detection system 140, and the like.” [0043] “the user interface element 240 is a progress bar, a segment of the bar is of a first color to indicate an amount of the video 202 that has been played and the remaining segment of the bar is of a different color to indicate the remaining amount of the video 202 to be played.” [0051] “the video frame 452 is presented in the display area 451 of the user interface 450. A user interface element 458 (e.g., a box) indicates to the user the portion of the video frame 452 that is determined to include the preferred content. The user interface element 458 is overlaid over the video frame 452.” [0076] “A particular user interface element 610 illustrates one or more particular events. For example, the user interface element 610a represents events between 4:00 AM and 5:00 AM on Monday October 23. The user interface elements 610 are configured such that they present analytics of events, for example, a total count of events in a particular time interval corresponding to a row as illustrated. The user interface elements 610 may be configured to be overlaid on top of or embedded in the calendar view. A user interface elements 610 is positioned to be aligned to a cell on the calendar view that corresponds to the specific time interval. The user interface elements 610 are further configured to allow users to review original footage of events. For example, if the user clicks on the user interface element 610a, a signal is generated to trigger the user interface module 150 to present footage of the events between 4:00 AM and 5:00 AM on Monday October 23.” [0086])
PNG
media_image1.png
678
538
media_image1.png
Greyscale
Zadeh does not explicitly teach trained to generate, for each of the one or more overlaid content items, a respective improved color value based at least on the generated data; obtaining one or more outputs of the machine learning model, the one or more outputs comprising, for each of the one or more overlaid content items, the respective improved color value; for each overlaid content item of the one or more overlaid content items, determining, based on the one or more outputs of the machine learning model, an improved color value that improves one or more presentation characteristics of the one or more overlaid content items within the video; and providing, via the user interface of the mobile application, a modified color for each of the one or more overlaid content items based on a respective improved color value. This is what Clawges teaches Clawges teaches obtaining one or more outputs of the machine learning model, the one or more outputs comprising, for each of the one or more overlaid content items, the respective color value; for each overlaid content item of the one or more overlaid content items, and providing, via the user interface of the mobile application, a color for each of the one or more overlaid content items. (“process 100 can cluster the pixels into any suitable number of pixels (e.g., two, five, ten, and/or any other suitable number). Note that, in some embodiments, the number of clusters can vary based on any suitable information (e.g., a genre of the video content item, a resolution of the video content item, and/or any other suitable information). In some embodiments, process 100 can cluster the pixels of the identified frame using any suitable technique or combination of techniques. For example, in some embodiments, process 100 can cluster the pixels using a group of thresholds corresponding to each cluster. As a more particular example, in some embodiments, process 100 can compare color values (e.g., hue, saturation, and value, or HSV, parameters for each pixel, and/or any other suitable values) for a pixel to predetermined threshold values corresponding to different clusters to assign the pixel to a particular cluster. As a specific example, process 100 can determine that a pixel with a hue parameter between 26 degrees and 52 degrees, a saturation parameter between 0.3 and 0.6, and a value parameter between 0.3 and 0.6 is to be assigned to a first cluster. As another example, in some embodiments, process 100 can use any suitable statistical classification or machine learning technique(s) to cluster the pixels (e.g., a K-nearest neighbors algorithm, and/or any other suitable clustering algorithm).” [0030] "user devices 306 can be implemented as a mobile device, such as a smartphone, mobile phone, a tablet computer, a laptop computer, a vehicle (e.g., a car, a boat, an airplane, or any other suitable vehicle) entertainment system, a portable media player, and/or any other suitable mobile device." [0049] “process 600 can identify initial appearance parameters for the user interface based on the color palette information at 604. For example, as described above, the initial appearance parameters can include colors for different sections or components of the user interface. As a more particular example, the parameters can include a first color for video player controls of the user interface when in an active state (e.g., when the video player is tapped, clicked or otherwise selected) and a second color for the video player controls of the user interface when in an idle state (e.g., when the video player has not been selected in more than a predetermined duration of time). As another more particular example, in some embodiments, the parameters can include colors for different sections of the user interface, for example, a first color for a first section of the user interface that provides information about the video content item, a second color for a second section of the user interface that provides recommendations of other videos to a user of the user interface, etc. As yet another more particular example, in some embodiments, the parameters can include colors for graphic overlays (e.g., graphics that indicate a selectable input to view other video content items created by the creator of the video content item, graphics that indicate a channel or group of videos associated with the video content item, and/or any other suitable graphic overlays) presented over the video content item during presentation of the video content item. In instances where the color palette information is transmitted to the user device as an array of arrays or matrix, in some embodiments, process 600 can identify the color palette information corresponding to a particular frame or portion of the video content item based on information associated with the first dimension of the array of arrays.” [0063]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Clawges into Zadeh, in order to improve the viewing experience.
The combination of Zadeh and Clawges does not explicitly teach trained to generate, for each of the one or more overlaid content items, a respective improved color value based at least on the generated data; determining, based on the one or more outputs of the machine learning model, an improved color value that improves one or more presentation characteristics of the one or more overlaid content items within the video; a modified color based on a respective improved color value. This is what Rykhliuk teaches (“the augmentation system 206 provides functions related to the generation and publishing of media overlays for messages processed by the messaging system 100. The augmentation system 206 operatively supplies a media overlay or augmentation (e.g., an image filter) to the messaging client 104 based on a geolocation of the client device 102. In another example, the augmentation system 206 operatively supplies a media overlay to the messaging client 104 based on other information, such as social network information of the user of the client device 102. A media overlay may include audio and visual content and visual effects. Examples of audio and visual content include pictures, texts, logos, animations, and sound effects. An example of a visual effect includes color overlaying.” [0037] “The custom augmentation system 124 uses machine learning to generate an image augmentation decision and provides the image augmentation as input to an augmented reality content item.” [0042] “transformations changing some areas of an object using its elements can be performed by calculating characteristic points for each element of an object and generating a mesh based on the calculated characteristic points. Points are generated on the mesh, and then various areas based on the points are generated. The elements of the object are then tracked by aligning the area for each element with a position for each of the at least one element, and properties of the areas can be modified based on the request for modification, thus transforming the frames of the video stream. Depending on the specific request for modification properties of the mentioned areas can be transformed in different ways. Such modifications may involve changing color of areas;” [0058] “the custom augmentation system 124 generates an image augmentation decision using a machine learning model. The image augmentation decision is based on the received image. In some examples, the machine learning model is the machine learning model 606. In one example, the machine learning model 606 is used to decide whether a user has warm or cool skin tones. … The augmentation reality content item is configured to modify image content of the received image. The augmentation reality content item may be accessed from the augmentation system 206. The augmentation content item may be a real-time special effect and sound that may be added to an image or a video. For example, an augmentation reality content item may be configured to change the color of an image.” [0113-0114] “the custom augmentation system 124 may include a machine learning model 606 that is trained to analyze the received image and generate a decision on whether a person in the image has a warm skin tone or a cool skin tone. The custom augmentation system 124 associates that image augmentation decision (e.g., whether person has a warm skin tone or a cool skin tone) with an augmentation reality content item. For example, the augmentation reality content item may be configured to apply virtual makeup to the person's face. When the custom augmentation system 124 associates the image augmentation decision with the augmentation reality content item, the custom augmentation system 124 enhances the effect of the augmentation reality content item by providing the image augmentation decision to the augmentation reality content item. For example, if the image augmentation decision is that the person in the image has a warm skin tone, the augmentation reality content item will apply virtual makeup using color shades that are best suited for people with warm skin tones. If the image augmentation decision is that the person in the image has a cool skin tone, the augmentation reality content will apply virtual makeup using color shades that are best suited for people with cool skin tones.” [0116]) Rykhliuk teaches generates an image augmentation decision using a machine learning model, decide whether a user has warm or cool skin tones which the improved color value, and modify image content of the received image that includes color overlaying / changing for the result of a customized image augmentation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Rykhliuk into the combination of Zadeh and Clawges, in order to provide a creative tool for users to enhance the aesthetic of their content.
11. With reference to claim 2, Zadeh teaches generating the data related to the video comprises: generating a bounding box around each overlaid content item within a frame of the video; identifying a first color of initial colors within each bounding box within the frame; identifying, based on the first color associated with each bounding box within the frame, a second color of first color of bounding boxes within the frame; and creating a feature vector of the plurality of frames of the video, wherein the data provided as input to the machine learning model comprises the feature vector. (“A detector configured to identify an object within media content items can be trained using a machine learned model (e.g., a convolutional neural network) as applied to a set of example media content items that include the object. The search engine 148 can select a set of detectors stored in the detector store 156 to conduct the search. A detector can detect a predetermined set of visual features representing items. For example, a detector can detect one or more entities (e.g., an object, a particular individual, a human, etc.), an entity of a particular characteristic (e.g., a fawn pug puppy, an Asian female wearing blue jacket, etc.), an action (e.g., flying, sailing, etc.), a color, a texture, a shape, a pattern, and the like.” [0035] “A starting location of a user interface element is determined based on a starting time point of the video segment. A length of the user interface element is determined according to a duration of the video segment. The user interface module 150 can obtain the confidence scores from the search store 156. The starting time point of the video segment can be determined according to a timestamp (absolute or relative to the beginning of the video) of a beginning video frame of the video segment. The duration can be determined in a similar manner. In some embodiments, the user interface module 150 generates a user interface element if the corresponding video segment lasts at least a threshold duration. The user interface module 150 configures the user interface such that a user can input a threshold confidence value. A user can input the threshold confidence value concurrently when the video is being played. The user interface module 150 dynamically configures the user interface elements for illustrating the relevant video segments according to the user's input of the threshold confidence value.” [0038] “The user interface element module 150 further generates user interface elements (e.g., a border box) configured to surround the portion of the video frame where the preferred content is determined as being present. The user interface elements are overlaid on top of the region displaying the video.” [0041] “the detector management module 152 creates the detectors by training one or more machine learning models using training data. The training data include example media content items provided by a user. The example media content items can be obtained from a variety of sources. For example, a user specifies media content items by selecting media content items stored in the media content store 158, uploading media content items to the media detection system 140, selecting portions of a media content item, providing locations of media content items to the media detection system 140, and the like.” [0043] “The user interface element 240 uses visually distinguished elements to illustrate the progression. For example, the user interface element 240 is a progress bar, a segment of the bar is of a first color to indicate an amount of the video 202 that has been played and the remaining segment of the bar is of a different color to indicate the remaining amount of the video 202 to be played.” [0051] “the video frame 452 is presented in the display area 451 of the user interface 450. A user interface element 458 (e.g., a box) indicates to the user the portion of the video frame 452 that is determined to include the preferred content. The user interface element 458 is overlaid over the video frame 452. In the illustrated example, the box 458 is positioned to surround the individual's image to indicate to the user that the individual's image is likely to be Saylor Twift's image.” [0076] “A particular user interface element 610 illustrates one or more particular events. For example, the user interface element 610a represents events between 4:00 AM and 5:00 AM on Monday October 23. The user interface elements 610 are configured such that they present analytics of events, for example, a total count of events in a particular time interval corresponding to a row as illustrated. The user interface elements 610 may be configured to be overlaid on top of or embedded in the calendar view. A user interface elements 610 is positioned to be aligned to a cell on the calendar view that corresponds to the specific time interval. The user interface elements 610 are further configured to allow users to review original footage of events. For example, if the user clicks on the user interface element 610a, a signal is generated to trigger the user interface module 150 to present footage of the events between 4:00 AM and 5:00 AM on Monday October 23.” [0086])
Zadeh does not explicitly teach a first average color value of pixels, based on the first average color value, a second average color value of first average color values, and using second average color values. This is what Clawges teaches (“upon obtaining multiple clusters of pixels, process 100 can calculate an average color for each cluster in the top N clusters with the most pixels assigned to the cluster. For example, in an instance where process 100 clustered the pixels into ten clusters at block 104, N can be any suitable number less than ten (e.g., three, five, and/or any other suitable number). In some embodiments, process 100 can rank the clusters from block 104 based on the number of pixels assigned to each cluster and can calculate the average color for each of the N top-ranked clusters. In some embodiments, process 100 can calculate an average color using any suitable technique or combination of techniques. For example, in some embodiments, process 100 can average individual components of a color indicating scale (e.g., Red, Green, Blue, or RGB, values, HSV values, and/or any other suitable color indicating scale). … process 100 can select a cluster of the top N clusters with an average color most similar to content color data of a previously analyzed frame at 108. … process 100 can select the main color or the dominant color of the first frame using any suitable technique or combination of techniques. For example, in some embodiments, process 100 can use the clustering technique described above in connection with block 104 to partition the first frame into any suitable number of clusters, and can use the technique described above in connection with block 106 to determine an average color of each of the clusters. In some embodiments, process 100 can select the main color or the dominant color of the first frame based on the average colors of the clusters. For example, in some embodiments, process 100 can select the main color or the dominant color to be the average color of the largest cluster. As another example, in some embodiments, process 100 can select the main color or the dominant color to be the average color of a cluster in a particular spatial region of the first frame. As a more particular example, in some embodiments, process 100 can select the main color or the dominant color to be the average color of a cluster that spans at least a center portion of the first frame. … at block 110, process 100 determines that the average color is not to be selected as the content color data of the current frame (“no” at 110), process 100 can loop back to 108 and select an average color of a different cluster. For example, process 100 can determine that a cluster having the next highest number of pixels is used to determine the color content data. If, at block 110, process 100 determines that the average color is to be selected as the content color data of the current frame (“yes” at 110), process 100 can proceed to block 112 and can determine a color palette for the current frame based on the content color data. In some embodiments, the color palette can specify colors for any suitable portions of a user interface in which the video content item is being presented.” [0031-0038]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Clawges into Zadeh, in order to improve the viewing experience.
12. With reference to claim 3, Zadeh teaches generating a bounding box around each overlaid content item within a frame of the video comprises: for each overlaid content item within the frame, drawing a bounding box around a respective overlaid content item, within the bounding box associated with displaying the video in the respective frame (“The user interface element module 150 further generates user interface elements (e.g., a border box) configured to surround the portion of the video frame where the preferred content is determined as being present. The user interface elements are overlaid on top of the region displaying the video.” [0041] “The user interface element 240 uses visually distinguished elements to illustrate the progression. For example, the user interface element 240 is a progress bar, a segment of the bar is of a first color to indicate an amount of the video 202 that has been played and the remaining segment of the bar is of a different color to indicate the remaining amount of the video 202 to be played.” [0051] “the video frame 452 is presented in the display area 451 of the user interface 450. A user interface element 458 (e.g., a box) indicates to the user the portion of the video frame 452 that is determined to include the preferred content. The user interface element 458 is overlaid over the video frame 452. In the illustrated example, the box 458 is positioned to surround the individual's image to indicate to the user that the individual's image is likely to be Saylor Twift's image.” [0076] “A particular user interface element 610 illustrates one or more particular events. For example, the user interface element 610a represents events between 4:00 AM and 5:00 AM on Monday October 23. The user interface elements 610 are configured such that they present analytics of events, for example, a total count of events in a particular time interval corresponding to a row as illustrated. The user interface elements 610 may be configured to be overlaid on top of or embedded in the calendar view. A user interface elements 610 is positioned to be aligned to a cell on the calendar view that corresponds to the specific time interval. The user interface elements 610 are further configured to allow users to review original footage of events. For example, if the user clicks on the user interface element 610a, a signal is generated to trigger the user interface module 150 to present footage of the events between 4:00 AM and 5:00 AM on Monday October 23.” [0086])
Zadeh does not explicitly teach a number of pixels exceeds a number of pixels used to display the respective overlaid content in the frame. This is what Clawges teaches (“the mechanisms can determine the content color data using any suitable technique or combination of techniques. For example, in some embodiments, the mechanisms can cluster pixels within a frame based on a color of each pixel, and can identify the cluster having the most pixels. In another example, in some embodiments, the mechanisms can cluster pixels within a particular area of a frame by grouping or connecting pixels within a particular color range and can identify the color range or color ranges having a number of pixels greater than a particular threshold number of pixels. The mechanisms can then determine that a main color or a dominant color for the frame is the average color, or other suitable combination, of the pixels within the cluster having the most pixels or the average color, or other suitable combination, of the pixels within the cluster or clusters having a number of pixels greater than a threshold number of pixels.” [0025] “process 600 can identify initial appearance parameters for the user interface based on the color palette information at 604. For example, as described above, the initial appearance parameters can include colors for different sections or components of the user interface. As a more particular example, the parameters can include a first color for video player controls of the user interface when in an active state (e.g., when the video player is tapped, clicked or otherwise selected) and a second color for the video player controls of the user interface when in an idle state (e.g., when the video player has not been selected in more than a predetermined duration of time). As another more particular example, in some embodiments, the parameters can include colors for different sections of the user interface, for example, a first color for a first section of the user interface that provides information about the video content item, a second color for a second section of the user interface that provides recommendations of other videos to a user of the user interface, etc. As yet another more particular example, in some embodiments, the parameters can include colors for graphic overlays (e.g., graphics that indicate a selectable input to view other video content items created by the creator of the video content item, graphics that indicate a channel or group of videos associated with the video content item, and/or any other suitable graphic overlays) presented over the video content item during presentation of the video content item. In instances where the color palette information is transmitted to the user device as an array of arrays or matrix, in some embodiments, process 600 can identify the color palette information corresponding to a particular frame or portion of the video content item based on information associated with the first dimension of the array of arrays.” [0063]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Clawges into Zadeh, in order to improve the viewing experience.
13. With reference to claim 4, Zadeh teaches the feature vector is of a predefined length. (“A detector configured to identify an object within media content items can be trained using a machine learned model (e.g., a convolutional neural network) as applied to a set of example media content items that include the object. The search engine 148 can select a set of detectors stored in the detector store 156 to conduct the search. A detector can detect a predetermined set of visual features representing items. For example, a detector can detect one or more entities (e.g., an object, a particular individual, a human, etc.), an entity of a particular characteristic (e.g., a fawn pug puppy, an Asian female wearing blue jacket, etc.), an action (e.g., flying, sailing, etc.), a color, a texture, a shape, a pattern, and the like.” [0035] A starting location of a user interface element is determined based on a starting time point of the video segment. A length of the user interface element is determined according to a duration of the video segment. The user interface module 150 can obtain the confidence scores from the search store 156. The starting time point of the video segment can be determined according to a timestamp (absolute or relative to the beginning of the video) of a beginning video frame of the video segment. The duration can be determined in a similar manner. In some embodiments, the user interface module 150 generates a user interface element if the corresponding video segment lasts at least a threshold duration. The user interface module 150 configures the user interface such that a user can input a threshold confidence value. A user can input the threshold confidence value concurrently when the video is being played. The user interface module 150 dynamically configures the user interface elements for illustrating the relevant video segments according to the user's input of the threshold confidence value.” [0038])
14. With reference to claim 5, Zadeh teaches the one or more overlaid content items of the video comprise at least one of a title of the video, a description of the video or one or more comments provided by viewers of the video. (“The preferred content definition module 144 may associate text descriptors with defined preferred content. A text descriptor describes characteristics of the preferred content.” [0031] “the video frame 452 is presented in the display area 451 of the user interface 450. A user interface element 458 (e.g., a box) indicates to the user the portion of the video frame 452 that is determined to include the preferred content. The user interface element 458 is overlaid over the video frame 452. In the illustrated example, the box 458 is positioned to surround the individual's image to indicate to the user that the individual's image is likely to be Saylor Twift's image.” [0076] “A particular user interface element 610 illustrates one or more particular events. For example, the user interface element 610a represents events between 4:00 AM and 5:00 AM on Monday October 23. The user interface elements 610 are configured such that they present analytics of events, for example, a total count of events in a particular time interval corresponding to a row as illustrated. The user interface elements 610 may be configured to be overlaid on top of or embedded in the calendar view. A user interface elements 610 is positioned to be aligned to a cell on the calendar view that corresponds to the specific time interval. The user interface elements 610 are further configured to allow users to review original footage of events. For example, if the user clicks on the user interface element 610a, a signal is generated to trigger the user interface module 150 to present footage of the events between 4:00 AM and 5:00 AM on Monday October 23.” [0086])
15. With reference to claim 6, Zadeh teaches identifying the one or more overlaid content items in the plurality of frames of the video using optical character recognition (OCR). (“A detector configured to identify an object within media content items can be trained using a machine learned model (e.g., a convolutional neural network) as applied to a set of example media content items that include the object. The search engine 148 can select a set of detectors stored in the detector store 156 to conduct the search. A detector can detect a predetermined set of visual features representing items. For example, a detector can detect one or more entities (e.g., an object, a particular individual, a human, etc.), an entity of a particular characteristic (e.g., a fawn pug puppy, an Asian female wearing blue jacket, etc.), an action (e.g., flying, sailing, etc.), a color, a texture, a shape, a pattern, and the like.” [0035] “the user interface module 150 generates user interfaces that allow users to configure example media content items (e.g., images, videos, portions of images, portions of videos such as video segments, portions of video frames) for training and/or updating detectors.” [0039] “The user interface element 405 is configured to allow a user to configure characteristics of detectors. The characteristics can be features of preferred content desired and/or undesired by the user. Example characteristics include a detector type such as natural imagery, facial recognition, facial characteristics, and the like. Referring now to FIG. 4C, the user interface element 405 is activated, which triggers the user interface 400 to provide user interface elements 430 through 432 that allow users to configure a detector type of the new detector designated by “Saylor Twift.” A detector of a detector type is configured to search for specific features in media content.” [0074] “the video frame 452 is presented in the display area 451 of the user interface 450. A user interface element 458 (e.g., a box) indicates to the user the portion of the video frame 452 that is determined to include the preferred content. The user interface element 458 is overlaid over the video frame 452. In the illustrated example, the box 458 is positioned to surround the individual's image to indicate to the user that the individual's image is likely to be Saylor Twift's image.” [0076] “A particular user interface element 610 illustrates one or more particular events. For example, the user interface element 610a represents events between 4:00 AM and 5:00 AM on Monday October 23. The user interface elements 610 are configured such that they present analytics of events, for example, a total count of events in a particular time interval corresponding to a row as illustrated. The user interface elements 610 may be configured to be overlaid on top of or embedded in the calendar view. A user interface elements 610 is positioned to be aligned to a cell on the calendar view that corresponds to the specific time interval. The user interface elements 610 are further configured to allow users to review original footage of events. For example, if the user clicks on the user interface element 610a, a signal is generated to trigger the user interface module 150 to present footage of the events between 4:00 AM and 5:00 AM on Monday October 23.” [0086])
16. With reference to claim 7, Zadeh does not explicitly teach receiving, via the user interface, a color selection, wherein the color selection may be one of: a modified color, a default color, or an arbitrary color; and updating, based on the color selection, an initial color of a respective overlaid content item within the video. This is what Clawges teaches (“process 100 can select a cluster of the top N clusters with an average color most similar to content color data of a previously analyzed frame at 108. For example, in some embodiments, process 100 can select the cluster with an average color most similar to a main color or dominant color of a frame corresponding to the previous second of video. As another example, in some embodiments, process 100 can select the cluster with an average color most similar to a main color or dominant color of the frame that was previously analyzed by process 100. … process 100 can select the main color or the dominant color of the first frame using any suitable technique or combination of techniques. For example, in some embodiments, process 100 can use the clustering technique described above in connection with block 104 to partition the first frame into any suitable number of clusters, and can use the technique described above in connection with block 106 to determine an average color of each of the clusters. In some embodiments, process 100 can select the main color or the dominant color of the first frame based on the average colors of the clusters. For example, in some embodiments, process 100 can select the main color or the dominant color to be the average color of the largest cluster.” [0032-0033] “Input device controller 406 can be any suitable circuitry for controlling and receiving input from one or more input devices 408 in some embodiments. For example, input device controller 406 can be circuitry for receiving input from a touchscreen, from a keyboard, from a mouse, from one or more buttons, from a voice recognition circuit, from a microphone, from a camera, from an optical sensor, from an accelerometer, from a temperature sensor, from a near field sensor, and/or any other type of input device.” [0055] “process 600 can begin by receiving, at a user device, a video content item and color palette information corresponding to the video at 602. In some embodiments, the user device can receive the video content item and the color palette information in response to a request for the video content item. For example, in some embodiments, the user device can transmit a request for the video content item in response to receiving, from a user of the user device, a selection of an identifier of the video content item for presentation on the user device. In some embodiments, the color palette information can be received in any suitable format. … process 600 can identify initial appearance parameters for the user interface based on the color palette information at 604. For example, as described above, the initial appearance parameters can include colors for different sections or components of the user interface. As a more particular example, the parameters can include a first color for video player controls of the user interface when in an active state (e.g., when the video player is tapped, clicked or otherwise selected) and a second color for the video player controls of the user interface when in an idle state (e.g., when the video player has not been selected in more than a predetermined duration of time). As another more particular example, in some embodiments, the parameters can include colors for different sections of the user interface, for example, a first color for a first section of the user interface that provides information about the video content item, a second color for a second section of the user interface that provides recommendations of other videos to a user of the user interface, etc. As yet another more particular example, in some embodiments, the parameters can include colors for graphic overlays (e.g., graphics that indicate a selectable input to view other video content items created by the creator of the video content item, graphics that indicate a channel or group of videos associated with the video content item, and/or any other suitable graphic overlays) presented over the video content item during presentation of the video content item. In instances where the color palette information is transmitted to the user device as an array of arrays or matrix, in some embodiments, process 600 can identify the color palette information corresponding to a particular frame or portion of the video content item based on information associated with the first dimension of the array of arrays.” [0062-0063]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Clawges into Zadeh, in order to improve the viewing experience.
17. Claim 8 is similar in scope to the claim 1, and thus is rejected under similar rationale. Zadeh additionally teaches A system comprising: a memory device; and a processing device coupled to the memory device, wherein the processing device is to perform operations (“Described herein are systems and methods that search for objects, faces, or other items selected by users in media content items to find media content items (such as videos or frames of video) where the media content preferred by users appear.” [0004] “The computer 1100 includes at least one processor 1102 coupled to a chipset 1104. Also coupled to the chipset 1104 are a memory 1106, a storage device 1108, a keyboard 1110, a graphics adapter 1112, a pointing device 1114, and a network adapter 1116. … The storage device 1108 is any non-transitory computer-readable storage medium, such as a hard drive, compact disk read-only memory (CD-ROM), DVD, or a solid-state memory device. The memory 1106 holds instructions and data used by the processor 1102. The pointing device 1114 may be a mouse, track ball, or other type of pointing device, and is used in combination with the keyboard 1110 to input data into the computer system 1100.” [0106-0107] “A system, comprising: a computer processor; and a non-transitory memory storing executable computer instructions that when executed by the computer processor are configured to cause the computer processor to perform steps” claim 15)
18. Claims 9-14 are similar in scope to the claims 2-7, and they are rejected under similar rationale.
19. Claim 15 is similar in scope to the claim 1, and thus is rejected under similar rationale. Zadeh additionally teaches A non-transitory computer-readable medium comprising instructions that, responsive to execution by a processing device, cause the processing device to perform operations (“The computer 1100 includes at least one processor 1102 coupled to a chipset 1104. Also coupled to the chipset 1104 are a memory 1106, a storage device 1108, a keyboard 1110, a graphics adapter 1112, a pointing device 1114, and a network adapter 1116. … The storage device 1108 is any non-transitory computer-readable storage medium, such as a hard drive, compact disk read-only memory (CD-ROM), DVD, or a solid-state memory device. The memory 1106 holds instructions and data used by the processor 1102. The pointing device 1114 may be a mouse, track ball, or other type of pointing device, and is used in combination with the keyboard 1110 to input data into the computer system 1100.” [0106-0107] “A non-transitory computer-readable storage medium storing executable computer instructions that when executed by a hardware processor are configured to cause the hardware processor to perform steps” claim 8)
20. Claims 16-20 are similar in scope to the claims 2-6, and they are rejected under similar rationale.
Conclusion
21. Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michelle Chin whose telephone number is (571)270-3697. The examiner can normally be reached on Monday-Friday 8:00 AM-4:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http:/Awww.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Kent Chang can be reached on (571)272-7667. The fax phone number for the organization where this application or proceeding is assigned is (571)273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https:/Awww.uspto.gov/patents/apply/patent- center for more information about Patent Center and https:/Awww.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE CHIN/
Primary Examiner, Art Unit 2614