Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after allowance or after an Office action under Ex Parte Quayle, 25 USPQ 74, 453 O.G. 213 (Comm'r Pat. 1935). Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, prosecution in this application has been reopened pursuant to 37 CFR 1.114. Applicant's submission filed on 4/22/2026 has been entered.
The IDS filed 4/22/2026 has been examined.
Claims 2, 4-6, 19 and 21 are cancelled.
Thus, Claims 1, 3, 7-18, 20, 22-23 are currently pending.
Response to Arguments
Applicant’s arguments with respect to claim(s) and the previous prior art rejection have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 7-10 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over
Berglund US Pub. No. 2024/0403341 A1, in view of Martigny et al., US Pub. No. 2024/0273105 A1, in view of Miller et al., US Pub. No. 2022/0253871 A1, in view of Newman et al., US Patent No. 11,768,837, in view of Holt et al., US Patent No. US 7,840,589 B1.
As to claim 1, Berglund discloses:
computer-implemented method
(Berglund Fig. 5 and [0015, 0083, 0090])
for including:
obtaining a content item having text and an image;
(Berglund teaches obtaining various content with text, i.e. “obtaining a content item having text and an image” see [0014] In some aspects, the techniques described herein include receiving a user query associated with a plurality of content items stored in a content repository maintained by the content management system.;
see also [0014] In some aspects, the techniques described herein include receiving a user query associated with a plurality of content items stored in a content repository maintained by the content management system. A set of text chunks corresponding to the plurality of content items are retrieved, where each text chunk has an associated text embedding. See also [0017] The content repository 150 stores content items such as documents, videos, images, audio recordings, 3D renderings, 3D models, or immersive content files (e.g., metaverse files). Documents stored in the content repository can include, for example, technical reports, sales brochures, books, web pages, transcriptions of video or audio recordings, presentations, or any other type of document.;
see also [0071] The transformer 412 can be trained to perform certain functions on a natural language input… the transformer 412 is trained to perform certain functions on other input formats than natural language input. For example, the input can include objects, images, audio content, or video content, or a combination thereof.)
causing a first machine learning model to generate a text summary that summarizes the content item;
(Berglund teaches machine learning summarizations of content see [0071] The transformer 412 can be trained to perform certain functions on a natural language input. Examples of the functions include summarizing existing content, brainstorming ideas, writing a rough draft, fixing spelling and grammar, and translating content. Summarizing can include extracting key points or themes from an existing content in a high-level summary.;
see also [0069] FIG. 4 is a block diagram of an example transformer 412. A transformer is a type of neural network architecture that uses self-attention mechanisms to generate predicted output based on input data that has some sequential meaning)
obtaining a search query through user interface;
(Berglund teaches obtaining a query see [0014] In some aspects, the techniques described
herein include receiving a user query associated with a plurality of content items stored in a
content repository maintained by the content management system.;
see also [0014] In some aspects, the techniques described herein include receiving a user
query associated with a plurality of content items stored in a content repository maintained by
the content management system. A set of text chunks corresponding to the plurality of content
items are retrieved, where each text chunk has an associated text embedding. See also [0017]
The content repository 150 stores content items such as documents, videos, images, audio recordings, 3D renderings, 3D models, or immersive content files (e.g., metaverse files). Documents stored in the content repository can include, for example, technical reports, sales brochures, books, web pages, transcriptions of video or audio recordings, presentations, or any other type of document.)
Berglund does not disclose:
causing an image captioning model to generate an image caption corresponding to the image of the content item;
causing a text embedding model to generate semantic search data by at least providing the text of the content item, the text summary, and the image caption as a first input to the text embedding model, the semantic search data including text embeddings representing semantic meaning corresponding to the text of the content item, the text summary, and the image caption;
causing a lexical data model to generate lexical search data by at least providing the first input to the lexical data model, the lexical search data including a set of strings extracted from the text of the content item, the text summary, and the image caption;
and
performing a semantic search using the semantic search data to obtain a first set of content items are semantically relevant to the search query and a lexical search using the lexical search data to determine a second set of content items that are lexically relevant to the search query, where the content item is a member of the first set of content items or the second set of content items; and
however, Martigny discloses:
causing an image captioning model to generate an image caption corresponding to the image of the content item;
(Martigny teaches extracting relevant features/summaries/ or contextual items of image from the images, i.e. generate an image caption corresponding to an image see [0046] For example, according to some aspects, a first vector search performed by a first predictive model on image data may use an image recognition technique to extract relevant features from the images associated with different content items in the repository (and/or database) and match them with the features in the request and/or query input. [0046] responsive to the request and/or query, multimodal content analysis and identification module 132 may cause independently trained predictive models ( e.g., machine learning models, neural networks, etc.) included, configured with, and/or in communication with (and/or the like) the multimodal content analysis and identification module 132 to concurrently run a first vector search on a first data type (e.g., image data for content items, etc.) and a second vector search on second data type ( e.g., a text description for content items, etc.) using the request and/or query input as a search parameter.
[0055] The term "feature," as used herein, may refer to any characteristic of an item of data that may be used to determine whether the item of data falls within one or more specific categories. For example, the features described herein may comprise indications of content items relevant to a query based on semantic text similarity, lexical similarities, attributes, and/or contextual items of image/depictions that indicate similarities in image data, sonic attributes, tones, pitches, vocal patterns, rhythms/beats, etc. that indicate similarities in audio content, ancillary information ( e.g., indicating or related to a writer, director, producer, composer, artist, actor, summary, chapters, production, history, year, trailers, alternate versions, related content, applications, objects depicted in content items, object types, etc.). According to some aspects of this disclosure, features may include any other information pertaining or relating to content items, as well as queries/requests for content items.;
See also [0050] the multimodal content analysis and identification module 132 may concurrently engage multiple predictive models to determine correspondences and/or similarities between
semantic information multiple modalities of data determined from a request and/or query and semantic information associated with content items.)
causing a text embedding model to generate semantic search data by at least providing the text of the content item, the text summary, and the image caption as a first input to the text embedding model,
(Martigny teaches using feature sets and Multimodal Analysis for Content Item Semantic Retrieval and Identification using embedding spaces, i.e. “causing a text embedding model to generate semantic search data by at least providing the text of the content item, the text summary, and the image caption as a first input to the text embedding model, ”
see [0058] embedded methods may include textual data, image data, audio data, ancillary content item data, and/or the like being mapped to an embedding space to enable similarity between content items within a repository and content items requested and/or search/queried for to be identified.
See also [0059] According to some aspects of this disclosure, after multimodal content analysis and identification module 132 generates a feature set(s), the multimodal content analysis
and identification module 132 may generate a machine learning-based predictive model 340 based on the feature set(s ).
see [0051] The machine learning-based classifier 330 may be configured to classify features for a specific modality and/or data type ( e.g., textual data, image data, audio data, ancillary content item data, etc.) extracted from requests and/or queries for content and/or content, as well as content items stored and/or available within a repository, catalog, database, via a service, and/or the like.
see also [0045] According to some aspects of this disclosure, system server(s) 126 (e.g. multimodal content analysis and identification module 132, etc.) operate to facilitate multimodal
analysis for content item semantic retrieval and identification. According to some aspects of this disclosure, system server(s) 126 may receive a request and/or query, for example, from media 106 and/or the like, for a content item and/or a type of content item. According to some aspects of this disclosure, the receive a request and/or query may include textual data ( e.g., typed data, natural language converted to text, etc.), image/graphics data (e.g., an image, an image banner, emoji, emoticons, screenshots, etc.) and/or the like.;
see also [0015] For example, according to some aspects of this disclosure, the content retrieval system may infer from image data ( e.g., an image, video, graphical depictions, etc.), textual data ( e.g., content descriptive data, closed captioning data, audio description data, etc.), audio data (e.g., voice/ audio tracks, soundtracks, sound effects, etc.), ancillary content item data ( e.g., information indicating or related to writer, director, producer, composer, artist, actor, summary, chapters, production, history, year, trailers, alternate versions, etc.), or the like whether a movie, show, program, or the like will be about a specific topic (e.g., racing cars,
sharks, dinosaurs, aliens, British wizards, etc.) or belong to a particular category (e.g., children shows, mature content, etc.), and decide when best to provide indication of the movie, show, program, and/or the like responsive to a query.)
the semantic search data including text embeddings representing semantic meaning corresponding to the text of the content item, the text summary, and the image caption;
(Martigny teaches multimodal semantic search using multiple predictive models, i.e. “semantic search data including text embeddings representing semantic meaning corresponding to the text of the content item, the text summary, and the image caption”
See [0013] Provided herein are system, apparatus, device, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for multimodal analysis for content item semantic retrieval and identification.
See also [0050] As described herein, the multimodal content analysis and identification module
132 may concurrently engage multiple predictive models to determine correspondences and/or similarities between semantic information multiple modalities of data determined from a request and/or query and semantic information associated with content items.
See also [0076] Returning to FIG. 1, as described, responsive to a request and/or query, multimodal content analysis and identification module 132 may cause independently trained
predictive models to concurrently run a first vector search on a first data type (e.g., image data for content items, etc.) and a second vector search on second data type (e.g., a text
description for content items, etc.) using the request and/or query input as a search parameter to identify relevant content items and output a final search result (to the media device 106 and/or user 134 to the user) indicative of the relevant content items. According to some aspects of this disclosure, the result may be presented in the form of a list of content items that match the request and/or query input, ranked according to their normalized similarity scores, or as
a single, aggregated result that combines information from both the second and first data types for identified content items.)
causing a lexical data model to generate lexical search data by at least providing the first input to the lexical data model, the lexical search data including a set of strings extracted from the text of the content item, the text summary, and the image caption;
(Martigny teaches multiple predictive models to performing a lexical search, i.e. “causing a lexical data model to generate lexical search data by at least providing the text of the content item, the text summary, and the image caption as a second input to the lexical data model” see [0077] According to some aspects of this disclosure, multimodal content analysis and identification module 132 may further refine a search result determined from the similarity scores generated by the first and second predictive models by incorporating search results determined from an exact match and/or lexical search performed responsive to the request and/or query input;
see also [0083] According to some aspects of this disclosure, content items identified using exact-match and/or lexical techniques may be combined with content items identified via
semantic analysis to provide indication of the best and/or most relevant content items responsive to a request and/or query input.;
see also [0087] In 508, system server(s) 126 utilizes one or more predictive models to perform a lexical search of the repository, catalog, database, and/or the like to retrieve relevant content items. The resulting content items for each modality of data may be ranked according to how closely they match the lexical attributes of the query.)
and
performing a semantic search using the semantic search data to obtain a first set of content items are semantically relevant to the search query and a lexical search using the lexical search data to determine a second set of content items that are lexically relevant to the search query, where the content item is a member of the first set of content items or the second set of content items; and
(Martigny teaches both lexical and semantic search results, i.e. “performing a semantic search using the semantic search data to obtain a first set of content items are semantically relevant to the search query and a lexical search using the lexical search data to determine a second set of content item that are lexically relevant to the search query” [0089-0090] [0089] In 512, system server(s) 126 combines the lexical and semantic search results. According to some aspects of this disclosure, the combined search results may include content items with both high-ranking similarity scores and high-ranking relevancy scores. [0090] In 514, system server(s) 126 outputs the combined search results responsive to the query. According to some aspects of this disclosure, the combined search results may be presented in the form of a list of content items that best match the query. According to some aspects of this disclosure, display device 108 may display the combined search results.)
where the content item is a member of the first set of content items or the second set of content items;
(Martigny teaches retrieving relevant content items of different modalities, i.e. “content item is a member of the first set of content items or the second set of content items” see [0014] According to some aspects of this disclosure, the content retrieval system is trained to infer semantics and relevancy from different modalities of data ( e.g., textual data, image data,
audio data, etc.) associated with the content items to provide an indication of the most relevant content items responsive to a query.; see also [0066] According to some aspects of this
disclosure, content item-related information may be used to generate one or more datasets, each dataset associated with a different modality of data.;
See also [0085] In 504, system server(s) 126 utilizes one or more predictive models to perform a semantic search of a repository, catalog, database, and/or the like according to one or more modalities of data to retrieve relevant content items. The resulting content items for each modality of data may be ranked according to how relevant they are to the query.)
It would have been obvious to one having ordinary skill in the art at the time the time of the effective filing date to applying combining the lexical and semantic search results as taught by Martigny, to the system of Berglund, since it was known in the art that search systems provide a system server(s) may include a multimodal content analysis and identification module where a multimodal content analysis is provided with an identification module which may use semantic search items to improve user engagement (e.g., click-through rates, launch rates, streaming hours, etc.) with categorical search and/or query results where for example, multimodal content analysis and identification module which may use processing techniques, such as artificial intelligence, semantic analysis, lexical analysis, exact-match retrieval, statistical models, logical
processing algorithms, and/or the like to indicate the most relevant content items responsive to a query. (Martigny [0040])
Berglund/Martigny do not disclose:
storing the semantic search data and the lexical search data in association with the content item;
However, Miller discloses:
storing the semantic search data and the lexical search data in association with the content item;
(Miller teaches PIDC product related datasets (stored product-related datasets)comprising one or more semantic term vectors and lexical term vectors i.e. “storing the semantic search data and the lexical search data in association with the content item” see [0061] In aspects, the invention provides a method carried out by a computer system, wherein the computer system
comprises a process component and a memory component. In aspects, the memory component of the computer system used in the method comprises a product information data
collection ("PIDC"). In aspects, the PIDC comprises stored product-related records/ datasets (PRRs) or other product relevant DSs ( e.g., product manufacturer-related DS s ), each stored product-related dataset comprising PIDC alphanumeric records contained in or derived from PIDC source data, a collection of PIDC semantic term vectors, and a collection of PIDC lexical term vectors; see also [0292] In aspects, query-type analyses performed herein
comprise lexical similarity comparisons. E.g., the comparison of lexical vectors of one data collection ( e.g., lexical vectors generated from a CPIS) to another (e.g., lexical vectors in a PIDC).)
It would have been obvious to one having ordinary skill in the art at the time the time of the effective filing date to apply lexical and semantic vector storage as taught by Miller, to the system of Berglund/Martigny, since it was known in the art that search systems provide for semantic terms are identified and from those identified semantic terms, semantic vectors are created; and also or alternatively, from that submission, lexical terms are identified and from those identified lexical terms, lexical vectors are created where an assessment of the similarity of such identified datasets can then lead to the direction of the system to analyze particular components of such datasets, or, e.g., indices related to such datasets, etc. where such aspects can, e.g., improve upon efficiencies of systems and methods described herein; and also allow for a proposed optimized evaluation submission is established and the optimized evaluation submission is presented to the end user, and feedback from the end user can be collected where such an evaluation by a human can include, e.g., an assessment of whether the results are flawed in any way, or, e.g., a search was flawed in any way and based on the receipt of end user feedback, and evaluation by the processor is performed to determine if further modification of the evaluation submission is required. (Miller [0439-0441, 0458]).
Berglund/Martigny/Miller do not disclose:
causing a user interface to display the first set of content items in a first
portion of the user interface associated with results that are semantically relevant
to the search query and the second set of content items in a second portion of the
user interface associated with results that are lexically relevant to the search query;
However, Newman discloses:
causing a user interface to display the first set of content items in a first
portion of the user interface associated with results that are semantically relevant
to the search query and the second set of content items in a second portion of the
user interface associated with results that are lexically relevant to the search query.
(Newman teaches a lexical and semantic searching interface for both types of results, i.e. a first/second portions associated with semantic/lexical results see Fig. 5 A with “not found try this instead”/” TAXPAYER IDENTIFICATION NUMBER” i.e. lexical portions and “concepts that are semantically related to the search input”, i.e. sematic portions
see col. 11 ln. 51-61: FIG. 5A illustrates a lexical and semantic searching interface, according to various examples. FIG. 5A includes input portion 502, an execute search element 504, and search suggestions 506. Without a semantic search model-but instead string matching-a user searching for "taxpayer identification number" would receive an incomplete list of results. For example, the results may include column names that start with "tax" but would not include a column labeled "Social_Security NBR" because there are no overlapping 60 characters.;
see also col. 12 ln. 22-31: As illustrated, the displayed selectable search suggestions
506 do not only include a concept based on a lexical correction of the search input-the taxpayer identification number- but concepts that are semantically related to the search input. Thus, search suggestions 506 are a blend of a lexical and semantic search. For instance, FIG. 5A also
shows selectable concepts that include "card identification number", "bank identification number", "issuer identification number", "national identification number, and "employer identification number".
see also col. 11 ln. 45-51: In various examples, a tree-style visualization of data connections between the data assets and ontology may be generated based on the retrieved data. The
columns, tables, databases, and data stores may be the nodes and the vertices the relationships between the nodes. Different icons may be used for each type (e.g., column, table, 50 etc.) of data asset.)
It would have been obvious to one having ordinary skill in the art at the time the time of the effective filing date to apply a lexical and semantic searching interface as taught by Newman, to the system of Berglund/Martigny/Miller, since it was known in the art that search systems provide for a lexical and semantic searching interface, according to various examples where a user may select one of the search suggestions presented in search suggestions and upon receiving the selection, input portion may be updated to display the selected search and after the user selects the execute search element a query may be made to the graph database to retrieve a number of results dependent on the visualization options selected by the user. (Newman [col. 12 ln. 50-59).
Berglund/Martigny/Miller/Newman do not disclose:
wherein displaying of content items of the first set of content items includes an indication of whether the content items are semantically or lexically relevant;
However, Holt discloses:
wherein displaying of content items of the first set of content items includes an indication of whether the content items are semantically or lexically relevant;
(Holt teaches possible matches/result groupings having a different semantic or lexical relationship to the query element, using “mouse over” AJAX-based windows or other pop-up objects, word-wheel-type arrangements, visual models, graphical clusters, price arrays, and the like, depending on the needs of any given application, i.e. “displaying of content items of the first set of content items includes an indication of whether the content items are semantically or lexically relevant”
See col. 12 ln. 19-: 34: It will be appreciated that the column/row format for displaying multiple relationships/groupings, be they between languages such as English or German or Chinese, or between broader, narrower terms, etc., is just one format possible for displaying content within the search query element viewer. Other formats or methods to designate such “groupings” might include “mouse over” AJAX-based windows or other pop-up objects, word-wheel-type arrangements, visual models, graphical clusters, price arrays, and the like, depending on the needs of any given application.
The notion of displaying different “groupings” of lexically-related alternatives, such as in multiple columns (for example, 508 and 512 in FIG. 7 and 548 and 550 in FIG. 11), wherein each group might have a different semantic or lexical relationship to the query element (e.g., 540 in FIG. 540), is represented in the example process flow of FIG. 2 b.;
See also abstract: A user enters their search query into a search query receiver. As the user enters their search query, they see, in real-time in a dynamically-generated object, such
as a drop-down menu, iFrame, or browser window, possible matches to their search query string, and more specifically, the user receives within the dynamic object alternative
semantically- and lexically-related search elements that relate to the search query string and from which the user can either make a selection to further refine their search query, or
the user can proceed to view search results based on the selected query element. The relation of alternate lexical elements is based on a controlled or structured vocabulary (for example a thesaurus);
see also col. 9 ln. 7-13: Such computer-generated content may be the result of clustering,
extraction, or similar summarization techniques wherein vocabulary elements may be related by virtue of parent/child (broader/narrower), synonym (associative), tags, definitions, labels, categories or similar such semantically related associations.);
see also col. 8 ln. 51-59: By way of example and not limitation, lexically-related alternatives can include corresponding broader terms, narrower terms, associative terms or synonyms, homograms, associatively-related terms, tags, sub-strings, symbols, portions of titles or captions, content of knowledge bases or meta-words and/or other controlled vocabulary pre-defined or pre-structured informational elements contained within the controlled vocabulary database. Lexically-related alternatives can include complete sentences or portions of sentences)
It would have been obvious to one having ordinary skill in the art at the time the time of the effective filing date to dynamical lexical and semantic groupings for search as taught by Holt, to the system of Berglund/Martigny/Miller/Newman, since it was known in the art that search systems provide a search query element viewer where the user can select any of the alternative query elements to further refine the initial query by resubmitting the selection to the search query receiver and identifying lexical alternatives to that submitted query and alternatively, the user may select one of the alternatives and submit that as a search term for use in searching the information system of interest where the semantically and lexically-related elements dynamically displayed via the search query element viewer preferably semantically and/or lexically relate both to the user's initially entered search criteria, and to the information system that is being searched or otherwise navigated. (Holt col. 8 ln. 3-27).
As to claim 3, Berglund as modified discloses the computer-implemented method of claim 1,
wherein the text summary is generated, via a large language model (LLM), by providing, to the
LLM, a model prompt including at least a portion of the text of the content item
(Berglund teaches using prompts and relevant context text chunks to create summaries/answers, where queries and the viewed document are sent to an LLM to generate answers, i.e. “wherein the text summary is generated, via a large language model (LLM), by providing, to the LLM, a model prompt including at least a portion of the text” see [0034] In some implementations, the query service 112 processes the search query to evaluate a type of output that should be returned in response to the search query. For example, when a search query includes question indicators such as question words (such as "how" or "what") and/or a question mark, the query service 112 predicts that the intent of the user is to receive an answer to a question. When the search query includes action indicators (such as "summarize" or "create"), the query service 112 predicts that the intent of the user is to perform a corresponding action on a content item or corpus.;
see also [0057] In another example, a user can submit queries related to a specific content item. For example, a document viewer provided by the content management system can include functionality to invoke a chatbot to answer queries related to a document currently being viewed in the viewer. The queries and the viewed document are sent to an LLM to generate answers for output by the chatbot. Such queries can include, for example, "summarize this document and create three questions that evaluate a reader's understanding of it" or "create a sales pitch about this document." In some implementations, queries directed to a particular document
can also reference external content, which is retrieved by the content management system before submitting the query to the LLM. For example, a user may ask the chatbot to
summarize the differences between a currently viewed version of a document and a previous version of the document;
see also [0014] From the set of text chunks, a subset of relevant text chunks is identified based at least in part on similarity between the associated text embeddings and a query embedding that represents the user query. At least a portion of the subset of relevant text chunks are sent to a large language model (LLM) to cause the LLM to generate an answer description
for the user query based on the text chunks. The answer description can be returned to the user, optionally with the content items themselves that contained the relevant text chunks.;
see also Berglund [0080] In some implementations, the input provided to the transformer 412 includes instructions to perform a function on an existing text. The output can include, for example, a modified version of the input text and instructions to modify the text. The modification can include summarizing, translating, correcting grammar or spelling, changing the style of the input text, lengthening or shortening the text, or changing the format of the text (e.g., adding bullet points or checkboxes ). As an example, the input text can include
meeting notes prepared by a user and the output can include a high-level summary of the meeting notes.).
As to claim 7, Berglund as modified discloses the computer-implemented method of claim 1 further obtaining the search query;
(Berglund [0014] In some aspects, the techniques described herein include receiving a user query associated with a plurality of content items stored in a content repository maintained by the content management system.)
generating a query text embedding, via the text embedding model, that represents the search query;
(Berglund teaches query embedding see [0014] A set of text chunks corresponding
to the plurality of content items are retrieved, where each text chunk has an associated text embedding. From the set of text chunks, a subset of relevant text chunks is identified based at least in part on similarity between the associated text embeddings and a query embedding that
represents the user query.;
see also [0038] The search engine 114 can match content items to the search queries based on text embeddings of the text chunks within a content item (which can be generated as
described with respect to FIG. 2A, for example). For example, the search engine 114 generates an embedding similarity score between a query embedding that represents the search query and a respective text embedding associated with a text chunk. A text chunk can be identified as relevant to the query when its embedding similarity score satisfies a similarity threshold;
see also [0028] For each of the sentence-length portions of text, the system 110 sends, at 214, the portion to the LLM 140 to generate a sentence embedding that represents the corresponding portion. Embeddings can alternatively be generated by models or algorithmic methods other than the LLM, for example by using techniques such as word2vec..)
performing the semantic search to determine that the content item is relevant to the search query by comparing the query text embedding to a text embedding representing the content item to analyze similarity between the query text embedding and the text embedding representing the content item;
(Berglund teaches determining similarity between the query and text embeddings see [0039] In some implementations, the search engine 114 filters content items based on other relevancy criteria, before or after determining similarity between the query and text embeddings. An example relevancy criterion is based on content metadata that can include, for example, an author of a content item, a time stamp indicating when the content item was created or most recently updated, tags or categorization labels applied to the content item, or a description for the content item. The search engine 114 can filter content items in the content repository to identify a set of content items that match an explicit content metadata item that is
specified in a search query. The filtered set of content items can then be processed to identify semantic matches to the search query (such as content items with text embeddings that are similar to the query embedding).;
see also [0042] The search engine 114 can furthermore filter content items based on a CRM record maintained by the CRM system 130. For example, the search engine can identify content items that are related to an account object within the CRM 130, prior to semantically matching the search query to text embeddings within the related content items.;
See also [0014] From the set of text chunks, a subset of relevant text chunks is identified based at least in part on similarity between the associated text embeddings and a query embedding that represents the user query;).
and
providing a search result corresponding with the content item for presentation in response to the search query
(Berglund teaches determining similarity between the query and text embeddings see [0039] In some implementations, the search engine 114 filters content items based on other relevancy criteria, before or after determining similarity between the query and text embeddings. An example relevancy criterion is based on content metadata that can include, for example, an author of a content item, a time stamp indicating when the content item was created or most recently updated, tags or categorization labels applied to the content item, or a description for the content item. The search engine 114 can filter content items in the content repository to identify a set of content items that match an explicit content metadata item that is
specified in a search query. The filtered set of content items can then be processed to identify semantic matches to the search query (such as content items with text embeddings that are similar to the query embedding;
see also [0044] Once content items related to the search query have been identified, the query service 112 can send, at 224, a list of the search results to the user interface for display while an answer to the user's search query is generated. The query service 112 can rank content items in the set returned by the search engine 114 before providing the search results to the
user interface, for example based on a degree of match between the content item and the query or based on the date of the content items. Additionally or alternatively, the query service 112 can apply additional filters or processing to the set of matching content items, for example to remove duplicate items.).
As to claim 8, Berglund as modified discloses the computer-implemented method of claim 7,
wherein the search result includes a result context that indicates a portion of the text summary that corresponds with the search query
(Berglund teaches output can include a response to the question, text associated with the request, or a list of ideas associated with the request, i.e. “a result context that indicates a portion of the text summary that corresponds with the search query” see [0080] As an example, the input text can include meeting notes prepared by a user and the output can include
a high-level summary of the meeting notes. In other examples, the input provided to the transformer includes a question or a request to generate text. The output can include a response to the question, text associated with the request, or a list of ideas associated with the request. For example, the input can include the question "What is the weather like in San Francisco?" and the output can include a description of the weather in San Francisco. As another example, the input can include a request to brainstorm names for a flower shop and the output can include a list of relevant names.;
see also [0021] The answers output by the content management system can include natural language answers which, for example, directly respond to the query (e.g., answering a user's question as a natural language response, rather than only linking to a content item that contains the answer), or summarize one or more content items (e.g., providing a bulleted list of key points from a slide deck).).
As to claim 9, Berglund as modified discloses the computer-implemented method of claim 8
further comprising:
obtaining a user feedback modifying the portion of the text summary;
(Berglund [0053] The user can provide feedback on the answer, for example to indicate that the user liked/disliked the answer or that the answer was helpful/ unhelpful. The feedback can additionally or alternatively include more granular signals, such as the answer not matching what the user was looking for, the answer being confusing, etc. The feedback received from the user can be used by the content management system to improve the operations of the content management system or the LLMs described herein.) and
updating the text summary to incorporate the user feedback
(Berglund [0053] The user can provide feedback on the answer, for example to indicate that the user liked/disliked the answer or that the answer was helpful/ unhelpful. The feedback can additionally or alternatively include more granular signals, such as the answer not matching what the user was looking for, the answer being confusing, etc. The feedback received from the user can be used by the content management system to improve the operations of the content management system or the LLMs described herein.).
As to claim 10, Berglund as modified discloses the computer-implemented method of claim 1 further performing the semantic search using the semantic search data to determine that the content item is relevant to the search query;
(Berglund teaches
See [0013] When a user submits a search query, the search platform identifies
relevant content and leverages the relevant content as a knowledge base or context for an LLM to generate an answer to the search query. For example, users can submit queries such as "What are the top product differentiators for FY 2023 ?" "Summarize issues faced by the finserv industry and how Company A helps address these issues," or "key takeaways from product overview deck." A semantic search can be performed to identify content that are relevant to the
submitted queries. The queries and the relevant content items are sent to the LLM to formulate an answer to the queries based on the content items. The output returned in response to the query can therefore include a portion of text generated by the LLM (such as a bulleted list of key
takeaways from a product overview deck), in addition to, or instead of, a list of semantically matched content items.)
and Martigny as modified discloses:
performing the lexical search using the lexical search data to determine that another content item is relevant to the search query;
(Martigny teaches multiple predictive models to performing a lexical search, i.e. “performing the lexical search using the lexical search data to determine that another content item is relevant to the search query” see [0077] According to some aspects of this disclosure, multimodal content analysis and identification module 132 may further refine a search result determined from the similarity scores generated by the first and second predictive models by incorporating search results determined from an exact match and/or lexical search performed responsive to the request and/or query input;
see also [0083] According to some aspects of this disclosure, content items identified using exact-match and/or lexical techniques may be combined with content items identified via
semantic analysis to provide indication of the best and/or most relevant content items responsive to a request and/or query input.;
see also [0087] In 508, system server(s) 126 utilizes one or more predictive models to perform a lexical search of the repository, catalog, database, and/or the like to retrieve relevant content items. The resulting content items for each modality of data may be ranked according to how closely they match the lexical attributes of the query.)
providing, for display, a first search result indicating the content item and a second search result indicating the another content item, wherein the first search result and the second search result are concurrently presented via a graphical user interface in response to the search query
(Martigny teaches combined search results may be presented in the form of a list of content items, i.e. “providing, for display, a first search result indicating the content item and a second search result indicating the another content item” see [0089-0090] [0089] In 512, system server(s) 126 combines the lexical and semantic search results. According to some aspects of this disclosure, the combined search results may include content items with both high-ranking similarity scores and high-ranking relevancy scores. [0090] In 514, system server(s) 126 outputs the combined search results responsive to the query. According to some aspects of this disclosure, the combined search results may be presented in the form of a list of content items that best match the query. According to some aspects of this disclosure, display device 108 may display the combined search results.).
As to claim 22, Berglund as modified discloses the computer-implemented method of claim 1, wherein storing the semantic search data further comprises storing the text embeddings in a vector store that stores a plurality of text embeddings as multi-dimensional vectors, where at least one multi-dimensional vector corresponding to the content item
(Berglund [0077-0078] [0077] The vector space can be defined by the dimensions
and values of the embedding vectors. Various techniques can be used to convert a token 402 to an embedding 406. For example, another trained ML model can be used to convert the token 402 into an embedding 406. In particular, another trained ML model can be used to convert the token 402 into an embedding 406 in a way that encodes additional information
into the embedding 406 ( e.g., a trained ML model can encode positional information about the position of the token 402 in the text sequence into the embedding 406). In some implementations, the numerical value of the token 402 can be used to look up the corresponding embedding in an embedding matrix 404, which can be learned during training of the transformer 412. [0078] The generated embeddings 406 are input into the encoder 408. The encoder 408 serves to encode the embeddings 406 into feature vectors 414 that represent the latent features of the embeddings 406. The encoder 408 can encode positional information (i.e., information about the sequence of the input) in the feature vectors 414. The feature vectors 414 can have very high dimensionality (e.g., on the order of thousands or tens of thousands), with each element in a feature vector 414 corresponding to a respective feature. The numerical weight of each element in a feature vector 414 represents the importance of the corresponding feature. The space of all possible feature vectors 414 that can be generated by the encoder 408 can be referred to as a latent space or feature space.).
Claims 11-15 and 23 are rejected under 35 U.S.C. 103 as being unpatentable over
Berglund US Pub. No. 2024/0403341 A1, in view of Martigny et al., US Pub. No. 2024/0273105 A1, in view of Pell et al., US Pub. No. 2009/0063472 A1, in view of Costello et al., US Pub. No. 2009/0241044 A1.
As to claim 11, Berglund discloses:
discloses a computer-implemented method
(Berglund Fig. 5 and [0015, 0083, 0090])
comprising:
obtaining a search query;
(Berglund teaches obtaining a query see [0014] In some aspects, the techniques described herein include receiving a user query associated with a plurality of content items stored in a content repository maintained by the content management system.;
see also [0014] In some aspects, the techniques described herein include receiving a user
query associated with a plurality of content items stored in a content repository maintained by
the content management system. A set of text chunks corresponding to the plurality of content
items are retrieved, where each text chunk has an associated text embedding. See also [0017]
The content repository 150 stores content items
such as documents, videos, images, audio recordings, 3D renderings, 3D models, or immersive
content files (e.g., metaverse files). Documents stored in the content repository can include, for
example, technical reports, sales brochures, books, web pages, transcriptions of video or audio
recordings, presentations, or any other type of document.)
causing a first machine learning model to generate a text summary that
summarizes a content item of a set of content items, the content item including text
and an image;
(Berglund teaches machine learning summarizations of content see [0071] The transformer 412
can be trained to perform certain functions on a natural language input. Examples of the
functions include summarizing existing content, brainstorming ideas, writing a rough draft, fixing
spelling and grammar, and translating content. Summarizing can include extracting key points
or themes from an existing content in a high-level summary.;
see also [0069] FIG. 4 is a block diagram of an example transformer 412. A transformer is a
type of neural network architecture that uses self-attention mechanisms to generate predicted
output based on input data that has some sequential meaning;
see also [0014] In some aspects, the techniques described herein include receiving a user query associated with a plurality of content items stored in a content repository maintained by the content management system. A set of text chunks corresponding to the plurality of content items are retrieved, where each text chunk has an associated text embedding. See also [0017] The content repository 150 stores content items such as documents, videos, images, audio recordings, 3D renderings, 3D models, or immersive content files (e.g., metaverse files). Documents stored in the content repository can include, for example, technical reports, sales brochures, books, web pages, transcriptions of video or audio recordings, presentations, or any other type of document.;
see also [0071] The transformer 412 can be trained to perform certain functions on a natural language input… the transformer 412 is trained to perform certain functions on other input formats than natural language input. For example, the input can include objects, images, audio content, or video content, or a combination thereof.)
Berglund does not disclose:
causing an image captioning model to generate an image caption corresponding to an image caption of the content item;
performing a semantic search including searching a set of text embeddings,
representing content items of the set of content items, to identify a first subset of
content items of the set of content items semantically similar to the search query, where
the set of text embeddings are generated by a text embedding model taking as an input
the text summary and the image caption;
performing a lexical search including searching a set of lexical search data,
representing the content items, to identify a second subset of content items of the
set of content items lexically similar to the search query by at least providing as an input
to a lexical data model the text summary, the image caption, and the search query;
and providing, for display, a set of search results including indications of the first subset of content items semantically similar to the search query and the second subset of content items lexically similar to the search query;
however, Martigny discloses:
causing an image captioning model to generate an image caption corresponding to an image caption of the content item
(Martigny teaches extracting relevant features/summaries/ or contextual items of image from the
images, i.e. generate an image caption corresponding to an image see [0046] For example,
according to some aspects, a first vector search performed by a first predictive model on image
data may use an image recognition technique to extract relevant features from the images
associated with different content items in the repository (and/or database) and match them with
the features in the request and/or query input. [0046] responsive to the request and/or query,
multimodal content analysis and identification module 132 may cause independently trained
predictive models ( e.g., machine learning models, neural networks, etc.) included, configured
with, and/or in communication with (and/or the like) the multimodal content analysis and
identification module 132 to concurrently run a first vector search on a first data type (e.g.,
image data for content items, etc.) and a second vector search on second data type ( e.g., a text
description for content items, etc.) using the request and/or query input as a search parameter.
[0055] The term "feature," as used herein, may refer to any characteristic of an item of data that
may be used to determine whether the item of data falls within one or more specific categories.
For example, the features described herein may comprise indications of content items relevant
to a query based on semantic text similarity, lexical similarities, attributes, and/or contextual
items of image/depictions that indicate similarities in image data, sonic attributes, tones, pitches,
vocal patterns, rhythms/beats, etc. that indicate similarities in audio content, ancillary
information ( e.g., indicating or related to a writer, director, producer, composer, artist, actor,
summary, chapters, production, history, year, trailers, alternate versions, related content,
applications, objects depicted in content items, object types, etc.). According to some aspects of
this disclosure, features may include any other information pertaining or relating to content
items, as well as queries/requests for content items;
see also [0050] the multimodal content analysis and identification module 132 may concurrently engage multiple predictive models to determine correspondences and/or similarities between
semantic information multiple modalities of data determined from a request and/or query and semantic information associated with content items.)
performing a semantic search including searching a set of text embeddings,
representing content items of the set of content items, to identify a first subset of
content items of the set of content items semantically similar to the search query,
(Martigny teaches multimodal semantic search using multiple predictive models to determine correspondences and/or similarities between semantic information multiple modalities of data determined from a request and/or query and semantic information associated with content items, i.e. “performing a semantic search including searching a set of text embeddings”
See [0013] Provided herein are system, apparatus, device, method and/or computer program
product embodiments, and/or combinations and sub-combinations thereof, for multimodal
analysis for content item semantic retrieval and identification.
See also [0050] As described herein, the multimodal content analysis and identification module
132 may concurrently engage multiple predictive models to determine correspondences and/or
similarities between semantic information multiple modalities of data determined from a request
and/or query and semantic information associated with content items.
See also [0076] Returning to FIG. 1, as described, responsive to a request and/or query,
multimodal content analysis and identification module 132 may cause independently trained
predictive models to concurrently run a first vector search on a first data type (e.g., image data
for content items, etc.) and a second vector search on second data type (e.g., a text
description for content items, etc.) using the request and/or query input as a search parameter
to identify relevant content items and output a final search result (to the media device 106
and/or user 134 to the user) indicative of the relevant content items. According to some aspects
of this disclosure, the result may be presented in the form of a list of content items that match
the request and/or query input, ranked according to their normalized similarity scores, or as
a single, aggregated result that combines information from both the second and first data types
for identified content items.)
where the set of text embeddings are generated by a text embedding model taking as an input
the text summary and the image caption;
(Martigny teaches using feature sets and Multimodal Analysis for Content Item Semantic
Retrieval and Identification using embedding spaces, i.e. “where the set of text embeddings are generated by a text embedding model taking as an input the text summary and the image caption, ”
see [0058] embedded methods may include textual data, image data, audio data, ancillary
content item data, and/or the like being mapped to an embedding space to enable similarity
between content items within a repository and content items requested and/or search/queried
for to be identified.
See also [0059] According to some aspects of this disclosure, after multimodal content analysis
and identification module 132 generates a feature set(s), the multimodal content analysis
and identification module 132 may generate a machine learning-based predictive model 340
based on the feature set(s ).
see [0051] The machine learning-based classifier 330 may be configured to classify features for
a specific modality and/or data type ( e.g., textual data, image data, audio data, ancillary content
item data, etc.) extracted from requests and/or queries for content and/or content, as well as
content items stored and/or available within a repository, catalog, database, via a service,
and/or the like.
see also [0045] According to some aspects of this disclosure, system server(s) 126 (e.g.
multimodal content analysis and identification module 132, etc.) operate to facilitate multimodal
analysis for content item semantic retrieval and identification. According to some aspects of this
disclosure, system server(s) 126 may receive a request and/or query, for example, from media
106 and/or the like, for a content item and/or a type of content item. According to some aspects
of this disclosure, the receive a request and/or query may include textual data ( e.g., typed data,
natural language converted to text, etc.), image/graphics data (e.g., an image, an image banner,
emoji, emoticons, screenshots, etc.) and/or the like.;
see also [0015] For example, according to some aspects of this disclosure, the content retrieval
system may infer from image data ( e.g., an image, video, graphical depictions, etc.), textual
data ( e.g., content descriptive data, closed captioning data, audio description data, etc.), audio
data (e.g., voice/ audio tracks, soundtracks, sound effects, etc.), ancillary content item data (
e.g., information indicating or related to writer, director, producer, composer, artist, actor,
summary, chapters, production, history, year, trailers, alternate versions, etc.), or the like
whether a movie, show, program, or the like will be about a specific topic (e.g., racing cars,
sharks, dinosaurs, aliens, British wizards, etc.) or belong to a particular category (e.g., children
shows, mature content, etc.), and decide when best to provide indication of the movie, show,
program, and/or the like responsive to a query.)
performing a lexical search including searching a set of lexical search data,
representing the content items, to identify a second subset of content items of the
set of content items lexically similar to the search query by at least providing as an input
to a lexical data model the text summary, the image caption, and the search query;
(Martigny teaches combining the lexical and semantic search results, i.e. “performing a lexical search including searching a set of lexical search data, representing the content items, to identify a second subset of content items of the set of content item lexically similar to the search query” [0089-0090] [0089] In 512, system server(s) 126 combines the lexical and semantic
search results. According to some aspects of this disclosure, the combined search results may
include content items with both high-ranking similarity scores and high-ranking relevancy
scores. [0090] In 514, system server(s) 126 outputs the combined search results responsive to
the query. According to some aspects of this disclosure, the combined search results may
be presented in the form of a list of content items that best match the query. According to some
aspects of this disclosure, display device 108 may display the combined search results.).
It would have been obvious to one having ordinary skill in the art at the time the time of the
effective filing date to applying combining the lexical and semantic search results as taught by
Martigny, to the system of Berglund, since it was known in the art that search systems provide a
system server(s) may include a multimodal content analysis and identification module where a
multimodal content analysis is provided with an identification module which may use semantic
search items to improve user engagement (e.g., click-through rates, launch rates, streaming
hours, etc.) with categorical search and/or query results where for example, multimodal content
analysis and identification module which may use processing techniques, such as artificial
intelligence, semantic analysis, lexical analysis, exact-match retrieval, statistical models, logical
processing algorithms, and/or the like to indicate the most relevant content items responsive to
a query. (Martigny [0040]).
Berglund / Martigny do not disclose:
and providing, for display in a user interface, a set of search results including indications of the first subset of content items semantically similar to the search query and the second subset of content items lexically similar to the search query;
However, Pell discloses
and providing, for display in a user interface, a set of search results including indications of the first subset of content items semantically similar to the search query and the second subset of content items lexically similar to the search query
(Pell teaches different styles of emphasizing/differentiating/highlighting search result regions based on semantic/conceptual matching or literal matching, i.e. providing indications of semantically similar and lexically similar items see para. [0073]. Further, the emphasized region 430 encompasses words that literally match at least one (e.g., “criticized' and “obama'), but not all ("who'), of the search terms included in the query. Further yet, the emphasized region encompasses words (“Clinton') that are absent from the search terms of the query. This is mainly due to the process by which regions are identified for highlighting. In contrast to conventional search engines that provide literal matches only, thus, offering irrelevant search results; the natural language engine 290 of FIGS. 2 and 3 compares the conceptual meanings of the query against the passages of the searched documents to arrive upon a relevant search result. The relevant search result 425, comprising the region 430 and the text adjacent thereto, includes emphasis on the sequence of words 435 in order to draw attention to the most relevant portion of the search result 430. Accordingly, the user may make a determination of the applicability of the search result 425 to the query. Also, the emphasis on the sequence of words 435 provides justification as to why the search result 425 is included in the search results 420.;
See also semantic highlighting via para. [0072] The search results 420 are typically listed in a prioritized order based on their relevance to the query. However, the search results may be listed according to any ranking scheme utilized in the data-gathering industry. Further, regions mapped to the matched semantic representations may be emphasized when presented to the user. In one embodiment, the emphasized regions of the identified passages are presented to the user Such that the regions are positioned within actual text of the identified passages as the text appears in the document from which the identified passages are extracted. By way of example, the search result 425 includes a full sentence of content extracted verbatim from a document. A region 430, targeted by mapping a matching semantic representation to the content, includes a sequence of words 435. In this instance, the sequence of words 435 is contiguous; however, in other instances, the sequence of words may be disconnected. Further, the sequence of words 435 included within the highlighted region 430 is emphasized in the form of an answer (“Clinton criticized Obama') that is relevant to, and satisfies, the query, which is in the form of a question (“who criticized obama'). In other instances, emphasized words may be judged as relevant, even though they don't comprise a direct answer to the query.;
See also teachings of 1st/ second type of typographic techniques for different types of highlighting at [0075]. In another embodiment, with continued reference to search result 440, employing the highlighting scheme includes imposing a first type of typographic technique to
emphasize a sequence of words 450 within the region 445. As illustrated, the sequence of words 450 includes “author . . .columnist... criticized obama. Accordingly, the sequence of words 450, encompassed by the region 445, offers a direct and relevant response to the question posed by the query. Also, employing the highlighting scheme may include imposing a second type of typographic technique to emphasize words 455 that provide context for, or modify, the sequence of words 450 within the region 445. As illustrated the words 455 that are included in the search result 440, and possibly adjacent to the sequence of words 450, include Ann Coulter and “heavily, and provide context for, or modify, the sequence of words 450. By way of example, a first type of typographic technique to emphasize a sequence of words 450 may include applying a dark highlighting to the sequence of words 450, while the second type of typographic technique to emphasize words 455 may include applying a light highlighting to the words 455.).
It would have been obvious to one having ordinary skill in the art at the time the time of the effective filing date to apply highlighting search result regions based on semantic/conceptual matching or literal matching as taught by Pell, since it was known in the art that search systems provide for employing a procedure to recognize and highlight words in searched documents corresponding to search terms, or keywords, in a query beyond the exact matches produced by the matching process where in particular, a natural language engine may be implemented to recognize semantic relations between the search terms of the query and content within the searched documents, and to employ techniques for highlighting these recognized words when being presented to a user as search results and accordingly, the accuracy of the search results is increased and the highlighting advantageously directs the user's attention to text in the
searched documents that is most responsive to the query where for instance, the search terms of the query may be written in a format that poses a question, while the highlighted portion of
a search result, which is relevant to the query, may be format ted as an answer that satisfies the question; and when attempting to present search results that include highlighted regions that are relevant to a conceptual meaning of a query, several semantic-related processes are
invoked where a query conditioning pipeline is employed to derive a proposition from a query.
(Pell [0008-0009]).
Berglund / Martigny / Pell do not disclose:
causing the user interface to display the first set of content items including a
first indication that a first content item of the first set of content items is semantically relevant to the search query and the second set of content items including a second indication that a second content item of the second set of content items is lexically relevant to the search query, where the first set of content items and the second set of content items are interleaved;
however, Costello discloses
causing the user interface to display the first set of content items including a
first indication that a first content item of the first set of content items is semantically relevant to the search query and the second set of content items including a second indication that a second content item of the second set of content items is lexically relevant to the search query, where the first set of content items and the second set of content items are interleaved;
(Costello teaches groups of search results with a search result "drill down list” of category information including stems, abbreviations, word grouping, spelling variations, semantic relationships, and search result snippets, i.e. causing a user interface to display the first set of content items including a first indication that the first set of content items are lexically/semantically relevant [0047] FIG. 3 illustrates drill down results displayed for an exemplary search query. In one embodiment, the drill down results include a menu of refining search terms that are dynamically derived in response to the processing of a query. In one embodiment, the menu is a multi-level pull-down menu. In the illustrated example, a drill down menu 40 is displayed that lists search term refinements for the search query, "thunderbird" 10. The determination of the categories (i.e., the search term refinements) to be displayed in the drill down menu may be determined based on criteria including, but not limited to, stems, abbreviations, word grouping, spelling variations, semantic relationships, synonyms, acronym expansion, terms that divide a search space into substantially non-overlapping subsets, capitalization, and Markov techniques that consider preceding and subsequent terms for a related query.;
See also [0049] FIG. 4 illustrates tabs, stacks and drill down categories displayed for an exemplary search query. In the illustrated example, search results for an exemplary search query term, "jaguar" 11 are displayed. In an exemplary operation, upon execution of the search term, "jaguar" 11, a user is presented with one or more tabs representative of the different classes of search results associated with the search query. Upon selection of a particular tab, for example, "All Results" 13, the user is presented with one or more "stacks" that organize the
different classes of search results corresponding to the selected tab. As discussed above, the "stacks" may include web documents with similar contexual propositions associated
with the search query term "jaguar" 11. As further illustrated, the user is also presented with a "drill down list" 17 of category information corresponding to the selected tab, "All Results" 13 derived in response to the processing of the query term, "jaguar" 11.;
see also [0070] FIG. 14 illustrates a search results page with displayed snippets. The snippets are also referred to as textual summaries. The two paragraphs 108 and 110 illustrate two
possible snippets from one or more webpages. If the user chooses to display medium length snippets, then the snippet from the second paragraph 110, where the search term "term
A" appears twice (the snippet consists of the 4 highlighted sentences) is displayed.).
It would have been obvious to one having ordinary skill in the art at the time the time of the effective filing date to apply a drill down menu interface listing search term refinements interface as taught by Costello, to the system of Berglund / Martigny / Pell, since it was known in the art that search systems provide for a drill down technique for analyzing the results of a query is disclosed where a Search Result Drill Down Module includes executable instructions to
display a listing of results derived from processing a query where the Search Result Drill Down Module further includes executable instructions to display a menu of refining search terms that is dynamically derived in response to the processing of the query where the analysis includes inferring a set of terms that are good refinements for the query and providing them as guidance to the user for refinement of the current search or for a future search query where these terms may be grouped into meaningful labeled lists and selecting one of the terms from one of these lists executes the more precise query. (Costello [0046]).
As to claim 12, Berglund as modified discloses the method of claim 11, wherein the first subset of content items is identified as semantically similar to the search query based on a similarity distance between a query text embedding representing the search query and text embeddings representing the first set of content items
(Berglund [0051] For example, the confidence score can indicate a degree of match between the answer generated by the LLM and the related content, which the query service 112 can
generate by identifying a degree of overlap between keywords in the answer and related content, by computing a distance between embeddings that represent the answer and
the related content, or by other methods. In another example, the confidence score can indicate a degree of match between multiple answers output by the same LLM or by different
LLMs.;
See also [0014] From the set of text chunks, a subset of relevant text chunks is identified based at least in part on similarity between the associated text embeddings and a query embedding that represents the user query;).
As to claim 13, Pell as modified discloses the method of claim 11, wherein search results, of the set of search results, corresponding with the first subset of content items semantically similar to the search query and search results, of the set of search results, corresponding with the second subset of content items lexically similar to the search query are interleaved with one another based on a relevance ranking indicating relevance of the corresponding content item to the search query
(Pell [0054] These matching semantic representations may be mapped back to the documents 230 from which they were extracted by associating the documents 230, and the locations
therein, from which the semantic representations were derived. These documents 230, targeted by the associated locations, are collected and sorted by the ranking component
270. Sorting may be performed in any known method within the relevant field, and may include without limitation, ranking according to closeness of match, listing based on popularity
of the returned documents 230, or sorting based on attributes of the user submitting the query 225. These ranked documents 230 comprise the search result 285 and are conveyed to
the presentation device 275 for surfacing in an appropriate format on the UI display 295.).
As to claim 14, Berglund as modified discloses the method of claim 11, wherein the set of text
embeddings representing content items are generated by:
generating text summaries representing the content items;
(Berglund teaches machine learning summarizations of content see [0071] The transformer 412
can be trained to perform certain functions on a natural language input. Examples of the
functions include summarizing existing content, brainstorming ideas, writing a rough draft, fixing
spelling and grammar, and translating content. Summarizing can include extracting key points
or themes from an existing content in a high-level summary.;
see also [0069] FIG. 4 is a block diagram of an example transformer 412. A transformer is a
type of neural network architecture that uses self-attention mechanisms to generate predicted
output based on input data that has some sequential meaning)
and
applying a text embedding model in association with the text summaries to
generate the set of text embeddings
(Berglund teaches determining similarity between the query and text embeddings using description for the content item, i.e. “applying a text embedding model in association with the text summaries to generate the set of text embeddings” see [0039] In some implementations, the search engine 114 filters content items based on other relevancy criteria, before or after determining similarity between the query and text embeddings. An example relevancy criterion is based on content metadata that can include, for example, an author of a content item, a time stamp indicating when the content item was created or most recently updated, tags or categorization labels applied to the content item, or a description for the content item. The search engine 114 can filter content items in the content repository to identify a set of content items that match an explicit content metadata item that is specified in a search query. The filtered set of content items can then be processed to identify semantic matches to the search query (such as content items with text embeddings that are similar to the query embedding).;
see also [0042] The search engine 114 can furthermore filter content items based on a CRM
record maintained by the CRM system 130. For example, the search engine can identify content
items that are related to an account object within the CRM 130, prior to semantically matching
the search query to text embeddings within the related content items.)
As to claim 15, Martigny as modified discloses the method of claim 11, wherein at least one content item includes an image, and wherein a text embedding generated for the at least one content item is based on an image caption generated for an image of the at least one content item
(Martigny teaches multimodal semantic search using multiple predictive models, i.e. “semantic search data including text embeddings representing semantic meaning corresponding to the text of the content item, the text summary, and the image caption”
See [0013] Provided herein are system, apparatus, device, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for multimodal analysis for content item semantic retrieval and identification.
See also [0050] As described herein, the multimodal content analysis and identification module
132 may concurrently engage multiple predictive models to determine correspondences and/or similarities between semantic information multiple modalities of data determined from a request and/or query and semantic information associated with content items.
See also [0076] Returning to FIG. 1, as described, responsive to a request and/or query, multimodal content analysis and identification module 132 may cause independently trained
predictive models to concurrently run a first vector search on a first data type (e.g., image data for content items, etc.) and a second vector search on second data type (e.g., a text
description for content items, etc.) using the request and/or query input as a search parameter to identify relevant content items and output a final search result (to the media device 106 and/or user 134 to the user) indicative of the relevant content items. According to some aspects of this disclosure, the result may be presented in the form of a list of content items that match the request and/or query input, ranked according to their normalized similarity scores, or as
a single, aggregated result that combines information from both the second and first data types for identified content items.).
As to claim 23, Costello as modified, discloses the method of claim 11, wherein the first set of content items and the second set of content items are interleaved are interleaved based on relevance to the search query
(Costello teaches degree of relevance ranking, i.e. “interleaved based on relevance to the search query” see [0037] Stacks of documents may also be formed in accordance with semantic and statistical criteria which determine the relationship between terms that may be used to quantify which parts of the page are relevant and their degree of relevance by inducing a metric on areas identified by a metric on the contents. Stacks of documents may be formed in accordance with clustering criteria, induced metrics, lexical criteria, ontological criteria or mention frequency based on identifying the additional notions referenced on a subset of the pages in the stack that are related to the search query under consideration.;
see also [0048] The order of search term refinements may be based upon page rankings, the number of web pages selected, the overlap of web pages, the percentage of documents selected, a quality metric, or the relevance between list items and specified concepts.).
Claims 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over
Berglund US Pub. No. 2024/0403341 A1, in view of Martigny et al., US Pub. No. 2024/0273105 A1, in view of Peleg et al., US Pub. No. 2022/0198135, in view of Pell et al., US Pub. No. 2009/0063472 A1, in view of Costello et al. US Pub. No. 2009/0241044 A1.
As to claim 16, Berglund discloses
one or more computer storage media having computer-executable instructions embodied thereon that, when executed by one or more processors, cause the one or more processors to perform a method,
(Berglund Fig. 5 and [0015, 0083, 0090])
the method comprising:
obtaining a search query including text and an image;
(Berglund teaches obtaining a query and an image see [0014] In some aspects, the techniques described herein include receiving a user query associated with a plurality of content items stored in a content repository maintained by the content management system.;
see also [0014] In some aspects, the techniques described herein include receiving a user
query associated with a plurality of content items stored in a content repository maintained by
the content management system. A set of text chunks corresponding to the plurality of content
items are retrieved, where each text chunk has an associated text embedding. See also [0017]
The content repository 150 stores content items such as documents, videos, images, audio recordings, 3D renderings, 3D models, or immersive content files (e.g., metaverse files). Documents stored in the content repository can include, for example, technical reports, sales brochures, books, web pages, transcriptions of video or audio recordings, presentations, or any other type of document.
see also [0071] The transformer 412 can be trained to perform certain functions on a natural language input… the transformer 412 is trained to perform certain functions on other input formats than natural language input. For example, the input can include objects, images, audio content, or video content, or a combination thereof.)
causing a first machine learning model to generate a text summary that summarizes a content item of a set of content items;
(Berglund teaches machine learning summarizations of content see [0071] The transformer 412
can be trained to perform certain functions on a natural language input. Examples of the
functions include summarizing existing content, brainstorming ideas, writing a rough draft, fixing
spelling and grammar, and translating content. Summarizing can include extracting key points
or themes from an existing content in a high-level summary.;
see also [0069] FIG. 4 is a block diagram of an example transformer 412. A transformer is a
type of neural network architecture that uses self-attention mechanisms to generate predicted
output based on input data that has some sequential meaning)
Berglund does not disclose:
causing an image captioning model to generate an image caption corresponding to the image of the content item;
performing a semantic search including analyzing a set of text embeddings, representing text summaries of the set of content items, to identify a first subset of content items of the set of content items semantically similar to the search query including the text, where the set of text embeddings are generated by a text embedding model taking as an input the text summary and the image caption;
however, Martigny discloses
causing an image captioning model to generate an image caption corresponding to the image of the content item;
(Martigny teaches extracting relevant features/summaries/ or contextual items of image from the
images, i.e. generate an image caption corresponding to an image see [0046] For example,
according to some aspects, a first vector search performed by a first predictive model on image
data may use an image recognition technique to extract relevant features from the images
associated with different content items in the repository (and/or database) and match them with
the features in the request and/or query input. [0046] responsive to the request and/or query,
multimodal content analysis and identification module 132 may cause independently trained
predictive models ( e.g., machine learning models, neural networks, etc.) included, configured
with, and/or in communication with (and/or the like) the multimodal content analysis and
identification module 132 to concurrently run a first vector search on a first data type (e.g.,
image data for content items, etc.) and a second vector search on second data type ( e.g., a text
description for content items, etc.) using the request and/or query input as a search parameter.
[0055] The term "feature," as used herein, may refer to any characteristic of an item of data that
may be used to determine whether the item of data falls within one or more specific categories.
For example, the features described herein may comprise indications of content items relevant
to a query based on semantic text similarity, lexical similarities, attributes, and/or contextual
items of image/depictions that indicate similarities in image data, sonic attributes, tones, pitches,
vocal patterns, rhythms/beats, etc. that indicate similarities in audio content, ancillary
information ( e.g., indicating or related to a writer, director, producer, composer, artist, actor,
summary, chapters, production, history, year, trailers, alternate versions, related content,
applications, objects depicted in content items, object types, etc.). According to some aspects of
this disclosure, features may include any other information pertaining or relating to content
items, as well as queries/requests for content items.; )
performing a semantic search including analyzing a set of text embeddings, representing text summaries of the set of content items, to identify a first subset of content items of the set of content items semantically similar to the search query including the text,
(Martigny teaches multimodal semantic search using multiple predictive models to determine correspondences and/or similarities between semantic information multiple modalities of data determined from a request and/or query and semantic information associated with content items, i.e. “performing a semantic search including searching a set of text embeddings”
See [0013] Provided herein are system, apparatus, device, method and/or computer program
product embodiments, and/or combinations and sub-combinations thereof, for multimodal
analysis for content item semantic retrieval and identification.
See also [0050] As described herein, the multimodal content analysis and identification module
132 may concurrently engage multiple predictive models to determine correspondences and/or
similarities between semantic information multiple modalities of data determined from a request
and/or query and semantic information associated with content items.
See also [0076] Returning to FIG. 1, as described, responsive to a request and/or query,
multimodal content analysis and identification module 132 may cause independently trained
predictive models to concurrently run a first vector search on a first data type (e.g., image data
for content items, etc.) and a second vector search on second data type (e.g., a text
description for content items, etc.) using the request and/or query input as a search parameter
to identify relevant content items and output a final search result (to the media device 106
and/or user 134 to the user) indicative of the relevant content items. According to some aspects
of this disclosure, the result may be presented in the form of a list of content items that match
the request and/or query input, ranked according to their normalized similarity scores, or as
a single, aggregated result that combines information from both the second and first data types
for identified content items.)
where the set of text embeddings are generated by a text embedding model taking as an input the text summary and the image caption;
(Martigny teaches using feature sets and Multimodal Analysis for Content Item Semantic
Retrieval and Identification using embedding spaces, i.e. “where the set of text embeddings are generated by a text embedding model taking as an input the text summary and the image caption”
see [0058] embedded methods may include textual data, image data, audio data, ancillary
content item data, and/or the like being mapped to an embedding space to enable similarity
between content items within a repository and content items requested and/or search/queried
for to be identified.
See also [0059] According to some aspects of this disclosure, after multimodal content analysis
and identification module 132 generates a feature set(s), the multimodal content analysis
and identification module 132 may generate a machine learning-based predictive model 340
based on the feature set(s ).
see [0051] The machine learning-based classifier 330 may be configured to classify features for
a specific modality and/or data type ( e.g., textual data, image data, audio data, ancillary content
item data, etc.) extracted from requests and/or queries for content and/or content, as well as
content items stored and/or available within a repository, catalog, database, via a service,
and/or the like.
see also [0045] According to some aspects of this disclosure, system server(s) 126 (e.g.
multimodal content analysis and identification module 132, etc.) operate to facilitate multimodal
analysis for content item semantic retrieval and identification. According to some aspects of this
disclosure, system server(s) 126 may receive a request and/or query, for example, from media
106 and/or the like, for a content item and/or a type of content item. According to some aspects
of this disclosure, the receive a request and/or query may include textual data ( e.g., typed data,
natural language converted to text, etc.), image/graphics data (e.g., an image, an image banner,
emoji, emoticons, screenshots, etc.) and/or the like.;
see also [0015] For example, according to some aspects of this disclosure, the content retrieval
system may infer from image data ( e.g., an image, video, graphical depictions, etc.), textual
data ( e.g., content descriptive data, closed captioning data, audio description data, etc.), audio
data (e.g., voice/ audio tracks, soundtracks, sound effects, etc.), ancillary content item data (
e.g., information indicating or related to writer, director, producer, composer, artist, actor,
summary, chapters, production, history, year, trailers, alternate versions, etc.), or the like
whether a movie, show, program, or the like will be about a specific topic (e.g., racing cars,
sharks, dinosaurs, aliens, British wizards, etc.) or belong to a particular category (e.g., children
shows, mature content, etc.), and decide when best to provide indication of the movie, show,
program, and/or the like responsive to a query.)
Berglund/ Martigny do not disclose:
performing a prefix search including analyzing a set of search data, representing full-text of the content items, to identify a second sub set of content items that match the search query;
however, Peleg discloses:
performing a prefix search including analyzing a set of search data, representing full-text of the content items, to identify a second set of content items that match the search query;
(Peleg teaches wildcard search prompts for response/text generation, i.e. “performing a prefix search”/” , to identify a second set of content items that match” see [0104] In another example, a user may call the writing assistant and write "Bono's age is ?", using the symbol '?' to specify where a piece of information should be retrieved and inserted in the sentence. In response, the writing assistant may generate sentences such as "Bono is 60 years old."
See also [0098] Additionally or alternatively, the externally available information may also be used to augment the generated text output options. For example, when a user input refers to an entity, externally available information about that entity can be acquired and, where appropriate, incorporated into generated text output options to enhance the depth and quality of the generated text. Acquisition of information from external sources may be automatic as the user inputs information, or may be triggered by user input. For example, the inclusion of a wildcard symbol such as a "?" may prompt the writing assistant to acquire externally available information from an external source, generate text based on the acquired information, and insert the text in place of the wildcard symbol ( or at least provide text output options to
the user for potential selection and insertion at the site of the wildcard symbol).;
see also [0101] In some embodiments, the writing assistant may receive user input including one or more words and, in response, retrieve information from an external source based
on attributes associated with the user input. The attributes associated with the user input can be, for example, a name of a person, a place name, or an entity name. This list of
attributes is not meant to be limiting and could include any relevant attribute associated with the user input. The user input may also include a wildcard symbol. Common wildcard symbols include, but are not limited to an asterisk (*), a question mark (?), etc.)
It would have been obvious to one having ordinary skill in the art at the time the time of the effective filing date to apply wildcard search as taught by Peleg, since it was known in the art that search systems provide for externally available information may also be used to augment the generated text output options where for example, when a user input refers to an entity, externally available information about that entity can be acquired and, where appropriate, incorporated into generated text output options to enhance the depth and quality of the generated text where acquisition of information from external sources may be automatic as the user inputs information, or may be triggered by user input; for example, the inclusion of a wildcard symbol such as a "?" may prompt the writing assistant to acquire externally available information from an external source, generate text based on the acquired information, and insert the text in place of the wildcard symbol ( or at least provide text output options to the user for potential selection and insertion at the site of the wildcard symbol). (Peleg [0098]).
Berglund/ Martigny/ Peleg do not disclose:
providing, for concurrent display in a user interface, a first set of search results indicating the first subset of content items semantically similar to the search query and a second set of search results indicating the second subset of content items that match the search query;
however, Pell discloses:
providing, for concurrent display in a user interface, a first set of search results indicating the first subset of content items semantically similar to the search query and a second set of search results indicating the second subset of content items that match the search query;
(Pell teaches different styles of emphasizing/differentiating/highlighting search result regions based on semantic/conceptual matching or literal matching, i.e. providing indications of semantically similar and lexically similar items see para. [0073]. Further, the emphasized region 430 encompasses words that literally match at least one (e.g., “criticized' and “obama'), but not all ("who'), of the search terms included in the query. Further yet, the emphasized region encompasses words (“Clinton') that are absent from the search terms of the query. This is mainly due to the process by which regions are identified for highlighting. In contrast to conventional search engines that provide literal matches only, thus, offering irrelevant search results; the natural language engine 290 of FIGS. 2 and 3 compares the conceptual meanings of the query against the passages of the searched documents to arrive upon a relevant search result. The relevant search result 425, comprising the region 430 and the text adjacent thereto, includes emphasis on the sequence of words 435 in order to draw attention to the most relevant portion of the search result 430. Accordingly, the user may make a determination of the applicability of the search result 425 to the query. Also, the emphasis on the sequence of words 435 provides justification as to why the search result 425 is included in the search results 420.;
See also semantic highlighting via para. [0072] The search results 420 are typically listed in a prioritized order based on their relevance to the query. However, the search results may be listed according to any ranking scheme utilized in the data-gathering industry. Further, regions mapped to the matched semantic representations may be emphasized when presented to the user. In one embodiment, the emphasized regions of the identified passages are presented to the user Such that the regions are positioned within actual text of the identified passages as the text appears in the document from which the identified passages are extracted. By way of example, the search result 425 includes a full sentence of content extracted verbatim from a document. A region 430, targeted by mapping a matching semantic representation to the content, includes a sequence of words 435. In this instance, the sequence of words 435 is contiguous; however, in other instances, the sequence of words may be disconnected. Further, the sequence of words 435 included within the highlighted region 430 is emphasized in the form of an answer (“Clinton criticized Obama') that is relevant to, and satisfies, the query, which is in the form of a question (“who criticized obama'). In other instances, emphasized words may be judged as relevant, even though they don't comprise a direct answer to the query.;
See also teachings of 1st/ second type of typographic techniques for different types of highlighting at [0075]. In another embodiment, with continued reference to search result 440, employing the highlighting scheme includes imposing a first type of typographic technique to
emphasize a sequence of words 450 within the region 445. As illustrated, the sequence of words 450 includes “author . . .columnist... criticized obama. Accordingly, the sequence of words 450, encompassed by the region 445, offers a direct and relevant response to the question posed by the query. Also, employing the highlighting scheme may include imposing a second type of typographic technique to emphasize words 455 that provide context for, or modify, the sequence of words 450 within the region 445. As illustrated the words 455 that are included in the search result 440, and possibly adjacent to the sequence of words 450, include Ann Coulter and “heavily, and provide context for, or modify, the sequence of words 450. By way of example, a first type of typographic technique to emphasize a sequence of words 450 may include applying a dark highlighting to the sequence of words 450, while the second type of typographic technique to emphasize words 455 may include applying a light highlighting to the words 455.).
It would have been obvious to one having ordinary skill in the art at the time the time of the effective filing date to apply highlighting search result regions based on semantic/conceptual matching or literal matching as taught by Pell, since it was known in the art that search systems provide for employing a procedure to recognize and highlight words in searched documents corresponding to search terms, or keywords, in a query beyond the exact matches produced by the matching process where in particular, a natural language engine may be implemented to recognize semantic relations between the search terms of the query and content within the searched documents, and to employ techniques for highlighting these recognized words when being presented to a user as search results and accordingly, the accuracy of the search results is increased and the highlighting advantageously directs the user's attention to text in the
searched documents that is most responsive to the query where for instance, the search terms of the query may be written in a format that poses a question, while the highlighted portion of
a search result, which is relevant to the query, may be format ted as an answer that satisfies the question; and when attempting to present search results that include highlighted regions that are relevant to a conceptual meaning of a query, several semantic-related processes are
invoked where a query conditioning pipeline is employed to derive a proposition from a query.
(Pell [0008-0009]).
Berglund/ Martigny/ Peleg / Pell do not disclose:
interleaving, in the user interface, the first set of search results and the
second set of search results, where the first set of search results displayed
in the user interface include an indication of semantic similarity with the search
query and the second set of search results displayed in the user interface
include an indication of lexical similarity with the search query;
However, Costello discloses:
interleaving, in the user interface, the first set of search results and the
second set of search results, where the first set of search results displayed
in the user interface include an indication of semantic similarity with the search
query and the second set of search results displayed in the user interface
include an indication of lexical similarity with the search query.
(Costello teaches groups of search results with a search result "drill down list” of category information including stems, abbreviations, word grouping, spelling variations, semantic relationships, and search result snippets, i.e. causing a user interface to display the first set of content items including a first indication that the first set of content items are lexically/semantically relevant [0047] FIG. 3 illustrates drill down results displayed for an exemplary search query. In one embodiment, the drill down results include a menu of refining search terms that are dynamically derived in response to the processing of a query. In one embodiment, the menu is a multi-level pull-down menu. In the illustrated example, a drill down menu 40 is displayed that lists search term refinements for the search query, "thunderbird" 10. The determination of the categories (i.e., the search term refinements) to be displayed in the drill down menu may be determined based on criteria including, but not limited to, stems, abbreviations, word grouping, spelling variations, semantic relationships, synonyms, acronym expansion, terms that divide a search space into substantially non-overlapping subsets, capitalization, and Markov techniques that consider preceding and subsequent terms for a related query.;
See also [0049] FIG. 4 illustrates tabs, stacks and drill down categories displayed for an exemplary search query. In the illustrated example, search results for an exemplary search query term, "jaguar" 11 are displayed. In an exemplary operation, upon execution of the search term, "jaguar" 11, a user is presented with one or more tabs representative of the different classes of search results associated with the search query. Upon selection of a particular tab, for example, "All Results" 13, the user is presented with one or more "stacks" that organize the
different classes of search results corresponding to the selected tab. As discussed above, the "stacks" may include web documents with similar contexual propositions associated
with the search query term "jaguar" 11. As further illustrated, the user is also presented with a "drill down list" 17 of category information corresponding to the selected tab, "All Results" 13 derived in response to the processing of the query term, "jaguar" 11.;
see also [0070] FIG. 14 illustrates a search results page with displayed snippets. The snippets are also referred to as textual summaries. The two paragraphs 108 and 110 illustrate two
possible snippets from one or more webpages. If the user chooses to display medium length snippets, then the snippet from the second paragraph 110, where the search term "term
A" appears twice (the snippet consists of the 4 highlighted sentences) is displayed.).
It would have been obvious to one having ordinary skill in the art at the time the time of the effective filing date to apply a drill down menu interface listing search term refinements interface as taught by Costello, to the system of Berglund / Martigny / Pell, since it was known in the art that search systems provide for a drill down technique for analyzing the results of a query is disclosed where a Search Result Drill Down Module includes executable instructions to
display a listing of results derived from processing a query where the Search Result Drill Down Module further includes executable instructions to display a menu of refining search terms that is dynamically derived in response to the processing of the query where the analysis includes inferring a set of terms that are good refinements for the query and providing them as guidance to the user for refinement of the current search or for a future search query where these terms may be grouped into meaningful labeled lists and selecting one of the terms from one of these lists executes the more precise query. (Costello [0046]).
As to claim 17, Berglund as modified discloses the media of claim 16, wherein the semantic search generating a query text embedding based on the text;
(Berglund teaches determining similarity between the query embedding and text embeddings
See [0014] From the set of text chunks, a subset of relevant text chunks is identified based at least in part on similarity between the associated text embeddings and a query embedding that
represents the user query;
see [0039] In some implementations, the search engine 114 filters content items based on other relevancy criteria, before or after determining similarity between the query and text embeddings. An example relevancy criterion is based on content metadata that can include, for example, an author of a content item, a time stamp indicating when the content item was created or most recently updated, tags or categorization labels applied to the content item, or a description for the content item. The search engine 114 can filter content items in the content repository to identify a set of content items that match an explicit content metadata item that is
specified in a search query. The filtered set of content items can then be processed to identify semantic matches to the search query (such as content items with text embeddings that are similar to the query embedding).;
see also [0042] The search engine 114 can furthermore filter content items based on a CRM record maintained by the CRM system 130. For example, the search engine can identify content items that are related to an account object within the CRM 130, prior to semantically matching the search query to text embeddings within the related content items.)
generating the set of text embeddings that represent the text summaries of the set of content items by providing as an input the set of content items to the text embedding model;
(Berglund [0014] In some aspects, the techniques described herein include receiving a user query associated with a plurality of content items stored in a content repository maintained by
the content management system. A set of text chunks corresponding to the plurality of content items are retrieved, where each text chunk has an associated text embedding. From the set of text chunks, a subset of relevant text chunks is identified based at least in part on similarity between the associated text embeddings and a query embedding that represents the user query. At least a portion of the subset of relevant text chunks are sent to a large language model (LLM) to cause the LLM to generate an answer description for the user query based on the text chunks.)
and
performing similarity analysis of the query text embedding and the set of text embeddings to determine semantic similarity between the search query and the first subset of content items
(Berglund [0014] A set of text chunks corresponding to the plurality of content items are retrieved, where each text chunk has an associated text embedding. From the set of text chunks, a subset of relevant text chunks is identified based at least in part on similarity between the associated text embeddings and a query embedding that represents the user query.;
See also [0038] The search engine 114 can match content items to the search queries based on text embeddings of the text chunks within a content item (which can be generated as described with respect to FIG. 2A, for example). For example, the search engine 114 generates an embedding similarity score between a query embedding that represents the search query and a respective text embedding associated with a text chunk. A text chunk can be identified as relevant to the query when its embedding similarity score satisfies a similarity threshold.).
As to claim 18, Berglund as modified discloses the media of claim 17, wherein at least one search result includes a result context that indicates at least a portion of the text summary, generated for the content item
(Berglund teaches output can include a response to the question, text associated with the request, or a list of ideas associated with the request, i.e. “a result context that indicates a portion of the text summary that corresponds with the search query” see [0080] As an example, the input text can include meeting notes prepared by a user and the output can include
a high-level summary of the meeting notes. In other examples, the input provided to the transformer includes a question or a request to generate text. The output can include a response to the question, text associated with the request, or a list of ideas associated with the request. For example, the input can include the question "What is the weather like in San Francisco?" and the output can include a description of the weather in San Francisco. As another example, the input can include a request to brainstorm names for a flower shop and the output can include a list of relevant names.;
see also [0021] The answers output by the content management system can include natural language answers which, for example, directly respond to the query (e.g., answering a user's question as a natural language response, rather than only linking to a content item that contains the answer), or summarize one or more content items (e.g., providing a bulleted list of key points from a slide deck).).
As to claim 20, Berglund as modified discloses the media of claim 16, wherein the text
summary of the content item further includes the image caption
(Berglund teaches using description/tags or categorization labels applied to the content item i.e. “wherein the text summary of the content item further includes the image caption” see [0039] In some implementations, the search engine 114 filters content items based on other relevancy criteria, before or after determining similarity between the query and text embeddings. An example relevancy criterion is based on content metadata that can include, for example, an author of a content item, a time stamp indicating when the content item was created or most recently updated, tags or categorization labels applied to the content item, or a description for the content item. The search engine 114 can filter content items in the content repository to identify a set of content items that match an explicit content metadata item that is specified in a search query. The filtered set of content items can then be processed to identify semantic matches to the search query (such as content items with text embeddings that are similar to the query embedding).;
see also [0042] The search engine 114 can furthermore filter content items based on a CRM
record maintained by the CRM system 130. For example, the search engine can identify content
items that are related to an account object within the CRM 130, prior to semantically matching
the search query to text embeddings within the related content items.)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Mishra et al., US Patent No.: US 12,050,658 B2, teaches a first set of text may be received from a client device. A set of content items may be selected from among content items based upon the first set of text and a plurality of sets of content item text associated with the
content items. A set of terms may be determined based upon the first set of text and the set of content items. A similarity profile associated with the set of terms may be generated.
The similarity profile is indicative of similarity scores associated with similarities between terms of the set of terms. Relevance scores associated with the set of terms may be determined based upon the similarity profile. One or more search terms may be selected from among the set of terms based upon the relevance scores. A search may be performed based upon the one or more search terms;
Andreev et al.< US Pub. No. 20150278198 A1, teaches method and system for facilitating a semantic search based on one or more corpuses of natural language texts and presenting clustered results are provided. One or more corpuses of natural language texts are received including indexed linguistic parameters and semantic structures of lexical units. The linguistic parameters and semantic structures are generated during a preliminary syntactico-semantic analysis. Searching for text fragments satisfying a query in the one or more corpuses is performed. Relevance of the search results is estimated according to selected lexical meaning
CONTACT INFORMATION
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EVAN S ASPINWALL whose telephone number is (571)270-7723. The examiner can normally be reached Monday-Friday 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ajay Bhatia can be reached at 571-272-3906. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Evan Aspinwall/Primary Examiner, Art Unit 2156
Prosecution Insights