DETAILED ACTION
The present office action is responsive to the applicant’s filling the application on 10/08/2024.
The application has claims 1-20 present. All present claims have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner Notes
Examiner cites particular columns, paragraphs, figures and line numbers in the references as applied to the claims below for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested that, in preparing responses, the applicant fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner. The entire reference is considered to provide disclosure relating to the claimed invention. The claims & only the claims form the metes & bounds of the invention. Office personnel are to give the claims their broadest reasonable interpretation in light of the supporting disclosure. Unclaimed limitations appearing in the specification are not read into the claim. Prior art was referenced using terminology familiar to one of ordinary skill in the art. Such an approach is broad in concept and can be either explicit or implicit in meaning. Examiner's Notes are provided with the cited references to assist the applicant to better understand how the examiner interprets the applied prior art. Such comments are entirely consistent with the intent & spirit of compact prosecution.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 9-12, and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Cheng et al (US 20140328570), in view of Spreitzer et al. (US 20220293133).
In regards to claims (1, 9 and 17), Cheng teaches a non-transitory computer readable storage medium including instructions that, when executed by a computing device, cause the computing device to: present, via a communicatively coupled display, a video editing assistant user interface (UI) including: a request from a user to create adaptive video content that satisfies a set of characteristics identified based on the request (searching and obtaining video segments that meet search from user: See abstract and at least para 15, 50, 55: “The salient event criteria 138 may also include salient event criteria that is specified by or derived from user inputs 256. For example, the interactive storyboard module 124 may determine salient event criteria based on user inputs received by the editing module 126.”. Also in para 55: “For example, the system 100 may receive the event type (e.g., an event description 106) from an end user, e.g., as a text string, search query, or meta tag. The system 100 may extract meta tags previously associated with the input and use those meta tags to classify the input. Thus, in some embodiments, feature detection algorithms may be used to determine the salient event segments as described herein”): analyze, using a machine-learning model, existing video content to identify portions of the existing video content that satisfy the set of characteristics; for each portion of the existing video content that satisfies the set of characteristics, create adaptive video content using a respective portion of the existing video content that satisfies the set of characteristics (after receiving a request from a user searching storage and generating a montage based on the characteristics: see at least para 48: “indicators of evidentiary significance are determined using machine learning techniques. For example, a machine learning analysis of training videos depicting a "person making a sandwich" (a complex event) may indicate that semantic descriptions such as "kitchen" (scene), "hands visible" (actor), "placing fillings on bread" (action) and "spreading creamy substance" (action) are highly likely to be associated with a person making a sandwich, while other semantic descriptions such as "outdoor event," (scene), "vehicle moving" (action) or "person jumping" (action) are unlikely to be associated with that particular event. Such indicators can be used by the multimedia content understanding module 104, or the event detection module 228 more specifically, to perform semantic reasoning.” See also para 50-51, 59. “The system 100 analyzes the video using feature detection algorithms and semantic reasoning as described above. Time-dependent output of the system 100's semantic analysis of the detected features is shown by the graphics 512, 514, 516, 518. In the graphic 512, the portion 520 represents a salient event segment 112 of the video 510. The system 100 has determined, using the techniques described above, that the salient event segment 520 depicts the salient activity of blowing out candles. Similarly, the system 100 has identified, using the techniques described above, salient event segments 522, 524, each of which depicts a person singing, and a salient event segment 526, which depicts a person opening a box. The system 100 did not detect any segments depicting the activity of cutting a cake, in the video 510. The system 100 can extract the segments 520, 522, 524, 526 from the video 510 and incorporate the segments 520, 522 524, 526 into a video clip that includes only the most interesting or salient portions of the video 510.”); generate, using a machine-learning model, textual descriptions of the adaptive video content; and update the video editing assistant UI, presented via the communicatively coupled display, to include: an adaptive video content UI element for each created adaptive video content, the adaptive video content UI element including textual descriptions of the adaptive video content and a representation of the adaptive video content (teaches usage scenarios including generating a natural language description of the generated clip that can be use and presented to the user. Providing the generated video and description to a user in an interface. See at least para 60-64: “One usage scenario of the technology disclosed herein provides a fully-automated video creation service with a "one-click" sharing capability. In this embodiment, the service takes one or more input video files, which can be e.g. raw footage captured by smartphone, and identifies a most-relevant event type for the uploaded video, e.g. birthday party, wedding, football game. The service identifies the event using feature recognition algorithms, such as described above and in the aforementioned priority patent applications of SRI International.”. Para 63: “In some embodiments, the event description 106 or the NL description 122 generated automatically by the system 100 can take the form of, e.g., meta data that can be used for indexing, search and retrieval, and/or for advertising (e.g., as ad words). The meta data can include keywords that are derived from the algorithmically performed complex activity recognition and other semantic video analysis (e.g. face, location, object recognition; text OCR; voice recognition), performed by the system 100 using the feature detection algorithms 130 as described above.”).
Cheng doesn’t specifically teach using a first machine-learning model, using a second machine-learning model.
However, Cheng teaches learning module and the use of one or more machine learning algorithms for various aspects which accomplish the limitations for the first and second machine-learning model and using machine learning techniques to accomplish steps within the system (see FIG. 2 and at least para 37:”computing system 100 also includes a user preference learning module 148. The user preference learning module 148 is embodied as software, firmware, hardware, or a combination thereof. The user preference learning module 148 monitors implicit and/or explicit user interactions with the presentation 120 (user feedback 146) and executes, e.g., machine learning algorithms to learn user-specific specifications and/or preferences as to, for example, the types of activities that the user considers to be "salient" with respect to particular events, the user's specifications or preferences as to the ordering of salient events in various types of different presentations 120, and/or other aspects of the creation of the presentation 120 and/or the NL description 122. The user preference learning module 148 updates the templates 142, 144 and/or portions of the knowledge base 132 (e.g., the salient event criteria 138) based on its analysis of the user feedback 146.”. Also see para 43-44, 48 and 77 which provides examples for using one or more machine learning algorithms).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the teachings of Cheng to use multiple machine learning models to accomplish different aspects of the editing process, since it provides means to facilitated managing the task among models that are trained for each of different task and thus improving speed and system resources.
Cheng teaches monitoring for user input, which allows a user to enter a search query to initiate the process to generate and provide an edited montage to the user (see para 35, 55, 63), but doesn’t specifically teach a video editing assistant dialog for requesting user instructions, and a textual prompt UI element for receiving user input; in response to receiving, via the textual prompt UI element.
Spreitzer teaches a video editing assistant dialog for requesting user instructions, and a textual prompt UI element for receiving user input; in response to receiving, via the textual prompt UI element (see para 77: a video editing system where a notification is provided to a user asking for the users desired action associated to a video “The notification may be in the form of a prompt stating that an edited version of an existing video is available and asking if the user would like to view it, save it, edit it, discard it or provide a reminder at a later time.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Spreitzer with the teachings of Cheng in order to have an interface requesting user instructions associated to video editing and editing steps, since it provides means to facilitated beginning the process when a user desired to generate a video montage.
In regards to claims (2, 10 and 18), Cheng teaches wherein the video editing assistant UI, adaptive video content, and/or the generated textual descriptions are presented to a user via a video-editing platform (see at least para 33: “The interactive storyboard module 124 presents the salient event segments 112 using a storyboard format that enables the user to intuitively review, rearrange, add and delete segments of the presentation 120 (e.g. by tapping on a touchscreen of the HCI subsystem 638). When the user's interaction with the presentation 120 is complete, the interactive storyboard module 124 stores the updated version of the presentation 120 in computer memory (e.g., a data storage 620)”).
In regards to claims (3 11 and 19), Cheng doesn’t specifically teaches wherein the first machine-learning algorithm and the second machine-learning algorithm are a same machine-learning algorithm.
However, Cheng teaches learning module and that one or more machine learning algorithms can be implemented for various aspects which accomplish the limitations for the first and second machine-learning model and using machine learning techniques to accomplish steps within the system (see FIG. 2 and at least para 37:”computing system 100 also includes a user preference learning module 148. The user preference learning module 148 is embodied as software, firmware, hardware, or a combination thereof. The user preference learning module 148 monitors implicit and/or explicit user interactions with the presentation 120 (user feedback 146) and executes, e.g., machine learning algorithms to learn user-specific specifications and/or preferences as to, for example, the types of activities that the user considers to be "salient" with respect to particular events, the user's specifications or preferences as to the ordering of salient events in various types of different presentations 120, and/or other aspects of the creation of the presentation 120 and/or the NL description 122. The user preference learning module 148 updates the templates 142, 144 and/or portions of the knowledge base 132 (e.g., the salient event criteria 138) based on its analysis of the user feedback 146.”. Also see para 43-44, 48 and 75-78 which provides examples for using one or more machine learning algorithms).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the teachings of Cheng to use a single or multiple machine learning algorithms to accomplish different aspects of the editing process, since by having that option improves the system and provides means to facilitated managing system resources depending on the user’s device.
In regards to claims (4, 12 and 20), Cheng teaches, wherein the first machine-learning algorithm and/or the second machine-learning algorithm are large language models (see at least para 44: “Portions of the text model 240 may be embodied as a language model or vocabulary, for example”).
Claims (7, 15 and 16), are rejected under 35 U.S.C. 103 as being unpatentable over Cheng et al (US 20140328570), in view of Spreitzer et al. (US 20220293133) as applied to claims above, and further in view of Jindal et al. (US 20230360271).
In regards to claim (7 and 15), Cheng teaches tag which are the characteristics or descriptions for the events and presenting the events based on user request with associated tags and descriptions after generating the montage as taught above (see at least para 19, 48-50, 60-63), but doesn’t specifically teach wherein the instructions, when executed by the computing device, further cause the computing device to: update the video editing assistant UI to include descriptive reasoning, within the video editing assistant dialog, of one or more characteristics of the set of characteristics used in creating the respective adaptive video content.
Jindal teaches wherein the instructions, when executed by the computing device, further cause the computing device to: update the video editing assistant UI to include descriptive reasoning, within the video editing assistant dialog, of one or more characteristics of the set of characteristics used in creating the respective adaptive video content (teaches providing a user a summary for a media modification based on edits don’t to the media. See at least para 33: “provides a summary to the user relating to those modifications. In doing so, the image modification tracking system efficiently presents to the client device with a summary of image modifications to a selected point of interest”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the teachings of Jindal in combination with the teaches of Cheng to provide the user with a summary of the characteristics used when the montage was created, since by providing the information to the user, it allows the user to understand the characteristics that were focused on when generating the montage requested by the user.
In regards to claim 16, Cheng teaches wherein the characteristic of the set of characteristics include one or more of a location, a capture time, a capture sequence, sensor data, subject matter of video data, theme of video data, video data overview, and time of day (tags that are used to search “as characteristics” include location and subject matter. See at least para 19: “The event description 106 semantically describes an event depicted by the multimedia input 102, as determined by the multimedia content understanding module 104. In the illustrative embodiments, the event description 106 is determined algorithmically by the computing system 100 analyzing the multimedia input 102. In other embodiments, the event description 106 may be user-supplied or determined by the system 100 based on meta data or other descriptive information associated with the input 102. The illustrative event description 106 generated by the understanding module 104 indicates an event type or category, such as "birthday party," "wedding," "soccer game," "hiking trip," or "family activity." The event description 106 may be embodied as, for example, a natural language word or phrase that is encoded in a tag or label, which the computing system 100 associates with the multimedia input 102 (e.g., as an extensible markup language or XML tag). Alternatively or in addition, the event description 106 may be embodied as structured data, e.g., a data type or data structure including semantics, such as "Party(retirement)," "Party(birthday)," "Sports_Event(soccer)," "Performance(singing)," or "Performance(dancing)." Also see 48-50: “Each or any of the models 134, 136 and/or the mapping 140 can maintain (e.g., probabilistic or statistical) indicators of the determined evidentiary significance of the relationships between features, concepts, events, and salient activities. In some embodiments, indicators of evidentiary significance are determined using machine learning techniques. For example, a machine learning analysis of training videos depicting a "person making a sandwich" (a complex event) may indicate that semantic descriptions such as "kitchen" (scene), "hands visible" (actor), "placing fillings on bread" (action) and "spreading creamy substance" (action) are highly likely to be associated with a person making a sandwich, while other semantic descriptions such as "outdoor event," (scene), "vehicle moving" (action) or "person jumping" (action) are unlikely to be associated with that particular event. Such indicators can be used by the multimedia content understanding module 104, or the event detection module 228 more specifically, to perform semantic reasoning”).
Claims 5-6, 8, 13 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Cheng et al (US 20140328570), in view of Spreitzer et al. (US 20220293133) as applied to claims above, and further in view of Dombrowski et al. (US 20170220568).
In regards to claims (5 and 13), Cheng teaches using meta-tags to determine characteristics that meet the user request (see at least para 49, 54), but doesn’t specifically teach wherein a respective characteristic of the set of characteristics is determined to be satisfied based on sensor data from a wearable device.
Dombrowski teaches wherein a respective characteristic of the set of characteristics is determined to be satisfied based on sensor data from a wearable device (teaches the use of wearable devices to generate meta-tags for events. See at least para 98: “the example embodiments can automatically associate a fantag and a corresponding synchronized videostream (captured asynchronously) with particular highlights of a specific event. In addition to statistics/scoring applications, the example embodiments can support #fantag tweets, SMS messages, sensors, wearable devices, and a variety of other means for generating the implicit meta-tag generation signal to trigger a fantagged moment.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Dombrowski with the teachings of Cheng in order to include meta-tags for events generated by wearable devices among other sensors, since it provides means to facilitated generating tagged moments that can be accessed and searched for (see para 98).
In regards to claim (6 and 14), Chen teaches using machine learning techniques to determine and identify complex events that can be annotated tagged which the system can then search for (see at least para 19: “The event description 106 semantically describes an event depicted by the multimedia input 102, as determined by the multimedia content understanding module 104. In the illustrative embodiments, the event description 106 is determined algorithmically by the computing system 100 analyzing the multimedia input 102. In other embodiments, the event description 106 may be user-supplied or determined by the system 100 based on meta data or other descriptive information associated with the input 102. The illustrative event description 106 generated by the understanding module 104 indicates an event type or category, such as "birthday party," "wedding," "soccer game," "hiking trip," or "family activity." The event description 106 may be embodied as, for example, a natural language word or phrase that is encoded in a tag or label, which the computing system 100 associates with the multimedia input 102 (e.g., as an extensible markup language or XML tag). Alternatively or in addition, the event description 106 may be embodied as structured data, e.g., a data type or data structure including semantics, such as "Party(retirement)," "Party(birthday)," "Sports_Event(soccer)," "Performance(singing)," or "Performance(dancing)." Also see 48-50: “Each or any of the models 134, 136 and/or the mapping 140 can maintain (e.g., probabilistic or statistical) indicators of the determined evidentiary significance of the relationships between features, concepts, events, and salient activities. In some embodiments, indicators of evidentiary significance are determined using machine learning techniques. For example, a machine learning analysis of training videos depicting a "person making a sandwich" (a complex event) may indicate that semantic descriptions such as "kitchen" (scene), "hands visible" (actor), "placing fillings on bread" (action) and "spreading creamy substance" (action) are highly likely to be associated with a person making a sandwich, while other semantic descriptions such as "outdoor event," (scene), "vehicle moving" (action) or "person jumping" (action) are unlikely to be associated with that particular event. Such indicators can be used by the multimedia content understanding module 104, or the event detection module 228 more specifically, to perform semantic reasoning”). However, Cheng doesn’t specifically teach, wherein analyzing existing video content to identify portions of the existing video content that satisfy the set of characteristics includes: detecting existing video content that are associated with sensor data indicative of a user gesture; determining that the existing video content associated with the sensor data indicative of the user gesture satisfy the set of characteristics; and a portion of the existing video content that satisfies the set of characteristics includes the existing video content associated with the sensor data indicative of the user gesture.
Dombrowski teaches wherein analyzing existing video content to identify portions of the existing video content that satisfy the set of characteristics includes: detecting existing video content that are associated with sensor data indicative of a user gesture; determining that the existing video content associated with the sensor data indicative of the user gesture satisfy the set of characteristics; and a portion of the existing video content that satisfies the set of characteristics includes the existing video content associated with the sensor data indicative of the user gesture (teaches the use of wearable devices and other sensors to generate meta-tags for events. See at least para 98: “the example embodiments can automatically associate a fantag and a corresponding synchronized videostream (captured asynchronously) with particular highlights of a specific event. In addition to statistics/scoring applications, the example embodiments can support #fantag tweets, SMS messages, sensors, wearable devices, and a variety of other means for generating the implicit meta-tag generation signal to trigger a fantagged moment.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Dombrowski with the teachings of Cheng in order to include meta-tags for events generated by wearable devices among other sensors which would annotate when an event occurs and allow the system as taught by Cheng to retrieve a video event which has a tag generated by a wearable, since by incorporating these feature, it provides means to facilitate generating tagged moments that can be accessed and searched for (see para 98).
In regards to claim 8, Cheng teaches wherein the characteristic of the set of characteristics include one or more of a location, a capture time, a capture sequence, sensor data, subject matter of video data, theme of video data, video data overview, and time of day (tags that are used to search “as characteristics” include location and subject matter. See at least para 19: “The event description 106 semantically describes an event depicted by the multimedia input 102, as determined by the multimedia content understanding module 104. In the illustrative embodiments, the event description 106 is determined algorithmically by the computing system 100 analyzing the multimedia input 102. In other embodiments, the event description 106 may be user-supplied or determined by the system 100 based on meta data or other descriptive information associated with the input 102. The illustrative event description 106 generated by the understanding module 104 indicates an event type or category, such as "birthday party," "wedding," "soccer game," "hiking trip," or "family activity." The event description 106 may be embodied as, for example, a natural language word or phrase that is encoded in a tag or label, which the computing system 100 associates with the multimedia input 102 (e.g., as an extensible markup language or XML tag). Alternatively or in addition, the event description 106 may be embodied as structured data, e.g., a data type or data structure including semantics, such as "Party(retirement)," "Party(birthday)," "Sports_Event(soccer)," "Performance(singing)," or "Performance(dancing)." Also see 48-50: “Each or any of the models 134, 136 and/or the mapping 140 can maintain (e.g., probabilistic or statistical) indicators of the determined evidentiary significance of the relationships between features, concepts, events, and salient activities. In some embodiments, indicators of evidentiary significance are determined using machine learning techniques. For example, a machine learning analysis of training videos depicting a "person making a sandwich" (a complex event) may indicate that semantic descriptions such as "kitchen" (scene), "hands visible" (actor), "placing fillings on bread" (action) and "spreading creamy substance" (action) are highly likely to be associated with a person making a sandwich, while other semantic descriptions such as "outdoor event," (scene), "vehicle moving" (action) or "person jumping" (action) are unlikely to be associated with that particular event. Such indicators can be used by the multimedia content understanding module 104, or the event detection module 228 more specifically, to perform semantic reasoning”).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Sull et al. (US 20110093492): teaches system are provided for tagging, indexing, searching, retrieving, manipulating, and editing video images.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MARIO M VELEZ-LOPEZ whose telephone number is (571)270-7971. The examiner can normally be reached on M-F 10:30am-5:30pm ET.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Scott Baderman, can be reached at telephone number 571-272-3644. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center and the Private Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from Patent Center or Private PAIR. Status information for unpublished applications is available through Patent Center and Private PAIR for authorized users only. Should you have questions about access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/MARIO M VELEZ-LOPEZ/
Examiner, Art Unit 2118
/SCOTT T BADERMAN/Supervisory Patent Examiner, Art Unit 2118
Prosecution Insights