DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 14 March 2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 USC 101 because the claimed invention is directed to an abstract idea without significantly more. The claims recite steps for detecting key segments in audio or video. The limitations of claims 1-20 for detecting key segments in audio or video, as drafted, describe a method that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. The limitations of claims 1-20 for detecting key segments in audio or video, as drafted, are a computer program product or apparatus that, under their broadest reasonable interpretation, cover performance of the limitations in the mind but for the recitation of generic computer components. That is, other than “instructions” “computer”, “processor”, and “memory” nothing in the claim element precludes the step from practically being performed in the mind. For example, but for the computer hardware language, claim 1 and the other independent claims encompass steps than may be performed manually by the user. Specifically, a person could view/listen to segments of video/audio and write a list of keywords. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claims recite an abstract idea.
This judicial exception is not integrated into a practical application. In particular, the independent claims only recite the additional elements “computer readable storage medium”, “instructions” “computer”, “processor”, and “memory” to perform the aforementioned steps. The processor and other hardware are recited at a high-level of generality (i.e., as a generic processor performing a generic computer function for computer-based systematic literature review such that they amount to no more than mere instructions to apply the exception using generic computer components. Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional hardware elements to perform both the aforementioned steps amount to no more than mere instructions to apply the exception using generic computer components. Mere instructions to apply an exception using generic computer components cannot provide an inventive concept. The claims are not patent eligible.
A similar analysis applies to the dependent claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 9-10, 12-14, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20220020361, hereinafter referred to as Wintrode, in view of US 20190294668, hereinafter referred to as Goel et al.
Regarding claim 1, Wintrode discloses a method comprising:
obtaining multi-modal features of an audio or video, wherein the multi-modal features comprise a visual feature, an acoustic feature, and a natural language feature (“Receiver 308 de-modulates and/or extracts an audio signal from signal 322 which includes at least one audio segment 324 that is output to SAD 310. In one implementation, SAD 310 uses a deep neural network (DNN) model 326 to detect whether audio segment 324 includes speech. Those segments 324 determined to include speech are passed on and output to ASR 306,” Wintrode, para [0049].);
determining candidate key segments in the audio or video based on the multi-modal features (“By including KWS module 302 and KWS filter 304 in ASR 306 that pre-filters out or omits those audio segments that do not include a keyword of list 330 and/or keywords 320, ASR 306 is able to perform automatic speech recognition substantially more efficiently, more rapidly, and by utilizing substantially less processing power of a device such as devices 102, 104, 106 and server 112,” Wintrode, para [0052]. Here, key (i.e., non-omitted) segments are determined based on whether or not they contain a keyword.);
obtaining a keyword list based on automatic speech recognition (ASR) text of the candidate key segments (“By including KWS module 302 and KWS filter 304 in ASR 306 that pre-filters out or omits those audio segments that do not include a keyword of list 330 and/or keywords 320,” Wintrode, para [0052]. And, “…output, from the first automatic speech recognition engine, the one or more phonemes to a keyword filter such as filter 304 or 406 (Step 906),” Wintrode, para [0064]. Here, a “filtered list” (i.e., subset of the original keyword list) of keywords is obtained from the original keyword list based on ASR of the of key segments.); and
determining a key segment in the audio or video based on the keyword list (“Segment labels 334 may include labels indicating whether each audio segment of audio segments 332 includes a keyword in keyword list 330 or does not include a keyword of keyword list 330,” Wintrode, para [0050].).
Wintrode, though, does not disclose obtaining multi-modal features of an audio or video, wherein the multi-modal features comprise a visual.
Goel et al. is cited to disclose obtaining multi-modal features of an audio or video, wherein the multi-modal features comprise a visual feature (“The multimedia analysis engine 110 generates the emotions based on at least one of an object detection technique, a face detection method, a facial feature extraction and analysis method, a speech recognition and text analysis method, a keyword/keyphrase analysis method, a speech tone analysis method and a background audio score analysis method,” Goel et al., para [0064].). Goel et al. benefits Wintrode by generating contextual data elements from a comprehensive analysis of multimedia to identify intent of a user when consuming the multimedia (Goel et al., para [0024]). Therefore, it would be obvious for one skilled in the art to combine the teachings of Wintrode with those of Goel et al. to enhance the media analysis techniques of Wintrode.
As to claim 12, device claim 12 and method claim 1 are related as method and device of using same, with each claimed element’s function corresponding to the method step. Accordingly claim 12 is similarly rejected under the same rationale as applied above with respect to method claim. And, Wintrode, para [0037] and fig. 2, teaches processor, CRM, storage medium, and memory.
As to claim 20, CRM claim 20 and method claim 1 are related as method and CRM of using same, with each claimed element’s function corresponding to the method step. Accordingly claim 20 is similarly rejected under the same rationale as applied above with respect to method claim. And, Wintrode, para [0037] and [0065] and fig. 2, teaches processor, CRM, storage medium, and memory.
Regarding claim 2, Wintrode, as modified by Goel et al., discloses the method according to claim 1, wherein determining candidate key segments in the audio or video based on the multi-modal features comprises:
determining a plurality of segments in the audio or video based on the multi-modal features (Wintrode, para [0052].);
recognizing ASR text of the plurality of segments (Wintrode, para [0052] and [0064].); and
obtaining the candidate key segments by filtering out segments without ASR text from the plurality of segments (Wintrode, para [0052] and [0064].).
As to claim 13, device claim 13 and method claim 2 are related as method and device of using same, with each claimed element’s function corresponding to the method step. Accordingly claim 13 is similarly rejected under the same rationale as applied above with respect to method claim. And, Wintrode, para [0037] and fig. 2, teaches processor, CRM, storage medium, and memory.
Regarding claim 3, Wintrode, as modified by Goel et al., discloses the method according to claim 2, wherein determining a plurality of segments in the audio or video comprises:
classifying and scoring the multi-modal features (“Keyword classifier 408 may execute a machine learning technique to determine a filter function based on a keyword list such as keyword list 330 and one or more posteriorgram features. Keyword classifier 408 may define a single filter function that is used by KW filter 406 to filter out and/or omit segments such as segment 420 that do not contain a keyword such as in keyword list 330 which would be an input into learning module 402,” Wintrode, para [0056]. This excerpt shows the classifying of keywords associated with audio segments. “…decision threshold function 316 may assign a score 318 related to a probability of the presence of any keywords 320. If the score 318 is determined to be greater than or equal to the threshold value, the pipeline and/or system 300 determines that the audio segment 324 includes a keyword of keywords 320,” Wintrode, para [0052]. And, “Word lattice search module 418 may perform one or more of the operations of search module 314 and decision threshold function 316 to determine a score associated with each audio segment 420 such as score 318,” Wintrode, para [0058]. These last two excerpts are included to show scoring of audio segments.); and
determining the plurality of segments in the audio or video in response to scoring results exceeding a threshold (Wintrode, para [0052] and [0058].).
As to claim 14, device claim 14 and method claim 3 are related as method and device of using same, with each claimed element’s function corresponding to the method step. Accordingly claim 14 is similarly rejected under the same rationale as applied above with respect to method claim. And, Wintrode, para [0037] and fig. 2, teaches processor, CRM, storage medium, and memory.
Regarding claim 9, Wintrode, as modified by Goel et al., discloses the method according to claim 1, wherein obtaining multi-modal features of the audio or video comprises:
obtaining the visual feature of the audio or video from the audio or video through object detection, wherein the visual feature comprises a picture feature of the audio or video (“Embodiments herein disclose methods and systems for generating the keywords and/or keyphrases by analyzing the audio portions, text portions and objects and/or actions of the video portions of the multimedia,” Goel et al., para [0026]. And, “The multimedia analysis engine 110 analyzes the image frames of the video portions and detects the objects and/or actions using the CV technique for object detection/recognition (OD/R), action detection/recognition (AD/R), optical character recognition (OCR) and coverts the detected objects and/or actions into a textual list, which in turn processed for identifying the keywords and/or keyphrases that represent the context of the contents presented in the image frames of the multimedia,” Goel et al., para [0068].).
Regarding claim 10, Wintrode, as modified by Goel et al., discloses the method according to claim 1, wherein obtaining multi-modal features of the audio or video comprises:
obtaining the acoustic feature from the audio or video through audio event detection, wherein the acoustic feature comprises an audio event (Wintrode, para [0052] and [0064]. Here, an audio event detection is the keyword detection in the audio segment.).
Claim(s) 4-5 and 15-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20220020361, hereinafter referred to as Wintrode, in view of US 20190294668, hereinafter referred to as Goel et al., and further in view of US 20230317069, hereinafter referred to as Nakano et al.
Regarding claim 4, Wintrode, as modified by Goel et al., discloses the method according to claim 1, wherein obtaining the keyword list based on the ASR text of the candidate key segments comprises:
obtaining candidate keywords based on the ASR text of the candidate key segments (Wintrode, para [0052] and [0064].); and
ranking the candidate keywords based on labels of the candidate key segments (As previously noted, Wintrode teaches that audio segments are labeled according to whether or not the segment contains one or more keywords. The segments are then scored according to the probability of the presence of any keywords. Thus, the ranking of the candidate keywords is based on the labeled candidate key segments.).
Neither Wintrode nor Goel et al., though, disclose the labels of the candidate key segments are related to a knowledge graph used to obtain the keyword list.
Nakano et al. is cited to disclose the labels of the candidate key segments are related to a knowledge graph used to obtain the keyword list (“The speech corpora and/or portions thereof may be sorted into groups based on relevance to one or more target domains. If the target domain is not known from the outset, machine learning (e.g., named-entity recognition (NER), knowledge graph (KG), etc.) may be used in a cold-start process to infer the target domain(s) of the speech corpora from the inclusion and/or frequency of various predetermined keywords,” Nakano et al. [0031].). Nakano et al. benefits Wintrode by providing a method for context aware speech transcription and editing misused words in the speech corpora with correct words for the target domain (Nakano et al., para [0004]). Therefore, it would be obvious for one skilled in the art to combine the teachings of Wintrode with those of Nakano et al. to enhance the media analysis techniques of Wintrode.
As to claim 15, device claim 15 and method claim 4 are related as method and device of using same, with each claimed element’s function corresponding to the method step. Accordingly claim 15 is similarly rejected under the same rationale as applied above with respect to method claim. And, Wintrode, para [0037] and fig. 2, teaches processor, CRM, storage medium, and memory.
Regarding claim 5, Wintrode, as modified by Goel et al. and Nakano et al., discloses the method according to claim 4, wherein obtaining candidate keywords comprises:
recalling the candidate keywords from the ASR text of the candidate key segments, wherein the recalling comprises at least one of the following: model-based recalling; recalling based on vocabulary matching; or recalling based on data pattern matching (As previously noted, Wintrode matches the ASR results of the audio segments against a keyword (i.e., vocabulary).).
As to claim 16, device claim 16 and method claim 5 are related as method and device of using same, with each claimed element’s function corresponding to the method step. Accordingly claim 16 is similarly rejected under the same rationale as applied above with respect to method claim. And, Wintrode, para [0037] and fig. 2, teaches processor, CRM, storage medium, and memory.
Claim(s) 6-8, and 17-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20220020361, hereinafter referred to as Wintrode, in view of US 20190294668, hereinafter referred to as Goel et al., further in view of US 20230317069, hereinafter referred to as Nakano et al., and further in view of US US8904420, hereinafter referred to as Mountain.
Regarding claim 6, Wintrode, as modified by Goel et al. and Nakano et al., discloses the method according to claim 4, wherein ranking the candidate keywords to obtain the keyword list comprises:
ranking the candidate keywords based on the labels of the candidate key segments in which the candidate keywords are located and label priorities (As previously noted, Wintrode teaches that audio segments are labeled according to whether or not the segment contains one or more keywords. The segments are then scored according to the probability of the presence of any keywords. Thus, the ranking of the candidate keywords is based on the labeled candidate key segments.).
Neither Wintrode nor Goel et al. nor Nokano et al., though, disclose obtaining the keyword list from the ranked candidate keywords based on a keyword frequency condition (“The number of identified undesired words may be limited to a maximum number such that the step of analysing the media content using a media content recognition unit to identify one or more portions that contain undesired content includes comparing received audio content with data stored on a memory, the comparisons being performed with a maximum number of words, or up to a maximum number of words. This can be achieved, for example, by limiting the number of comparisons performed by the system, or limiting the number of words that are stored on the memory itself. The maximum number of words may be selected such that the recognition unit can compare the received audio with the maximum number of identified words within a specified time period, in particular the maximum number of words may be selected based on the time taken for a decoder to decode a portion of media content, the maximum number of words being such that the recognition unit can compare the portion of received audio content with the identified number of undesired words within the time taken for the decoder to decode the portion of media content,” Mountain, col. 2, line 57 – col. 3, line 15.). Mountain benefits Wintrode by selecting the maximum number of words such that the comparison of words with received audio content can be performed at the same rate, or a greater rate, as the received audio content is decoded by the decoder, thereby allowing the audio content to be checked in real time (Mountain, col. 3, lines 9-14). Therefore, it would be obvious for one skilled in the art to combine the teachings of Wintrode with those of Mountain to enhance the media analysis techniques of Wintrode.
As to claim 17, device claim 17 and method claim 6 are related as method and device of using same, with each claimed element’s function corresponding to the method step. Accordingly claim 17 is similarly rejected under the same rationale as applied above with respect to method claim. And, Wintrode, para [0037] and fig. 2, teaches processor, CRM, storage medium, and memory.
Regarding claim 7, Wintrode, as modified by Goel et al., Nakano et al., and Mountain, discloses the method according to claim 6, wherein the keyword frequency condition specifies a maximum allowed number or proportion of keywords in a time interval (Mountain, p. 3, highlighted section.).
As to claim 18, device claim 18 and method claim 7 are related as method and device of using same, with each claimed element’s function corresponding to the method step. Accordingly claim 18 is similarly rejected under the same rationale as applied above with respect to method claim. And, Wintrode, para [0037] and fig. 2, teaches processor, CRM, storage medium, and memory.
Regarding claim 8, Wintrode, as modified by Goel et al., Nakano et al., and Mountain, discloses the method according to claim 6, wherein the label priorities are determined based on a knowledge graph (Nakano et al. [0031] ).
As to claim 19, device claim 19 and method claim 8 are related as method and device of using same, with each claimed element’s function corresponding to the method step. Accordingly claim 19 is similarly rejected under the same rationale as applied above with respect to method claim. And, Wintrode, para [0037] and fig. 2, teaches processor, CRM, storage medium, and memory.
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20220020361, hereinafter referred to as Wintrode, in view of US 20190294668, hereinafter referred to as Goel et al., and further in view of US 20190348041, hereinafter referred to as Cella et al.
Regarding claim 11, Wintrode, as modified by Goel et al., discloses the method according to claim 1, but not wherein the obtaining multi-modal features of the audio or video comprises: obtaining the natural language feature from the audio or video based on a knowledge graph and a pre-trained text detection model, wherein the natural language feature comprises ASR text.
Cella et al. is cited to disclose obtaining the natural language feature from the audio or video based on a knowledge graph and a pre-trained text detection model, wherein the natural language feature comprises ASR text (“In embodiments a database may include one or more knowledge graphs, such as representing one or more a priori structures representing an understanding of one or more topics, such as a knowledge graph of physical activities, a knowledge graph of social relationships, a knowledge graph of business relationships, a knowledge graph of content (e.g., music, audio, video or the like), a knowledge graph of psychographic or demographic categories, a map or other location-based knowledge graph, or the like. In embodiments, sensor data, audio signal data, speech, processed output, and other information handled by the systems described herein may be associated with one or more nodes in a knowledge graph, such as by machine processing, such as by supervised learning on a labeled training set or by deep learning based on feedback to the system. For example, the system may learn to augment a social graph for a user or a group by adding speech or other content that is spoken by or to a user (or an annotation about such speech, or a summary thereof) to a relevant node of the knowledge graph, such as to capture a conversation over time involving one or more topics that are contained in the speech and that node of the knowledge graph,” Cella et al., para [0226].). Cella et al. benefits Wintrode by incorporating a knowledge graph to represent a priori structures representing an understanding of topics, thereby improving the pre-training of an audio text search system. Therefore, it would be obvious for one skilled in the art to combine the teachings of Wintrode with those of Cella et al. to improve the filtering of audio keyword search as described by Wintrode.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. See attached PTO-892.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANNE L THOMAS-HOMESCU whose telephone number is (571)272-0899. The examiner can normally be reached Mon-Fri 8-6.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bhavesh M Mehta can be reached on 5712727453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANNE L THOMAS-HOMESCU/Primary Examiner, Art Unit 2656
Prosecution Insights