Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
The Action is responsive to the Application filed on 12/6/2024. Claims 1-20 are pending claims. Claims 1, 5, and 16 are written in independent form.
Priority
Applicant’s claim for benefit of prior-filed provisional application 63/615,743 under 35. U.S.C. 119(e) or under 35 U.S.C. 120, 121, or 365(c) is acknowledged.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-4 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-patentable subject matter. The claimed invention is directed to one or more abstract ideas without significantly more. The judicial exception is not integrated into a practical application. The claims do not include additional elements that are sufficient to amount to significantly more than judicial exception. The eligibility analysis in support of these findings is provided below.
As per Claim 1,
STEP 1:In accordance with Step 1 of the eligibility inquiry (as explained in MPEP 2106), the claimed method (claims 1-4) is directed to one of the eligible categories of subject matter and therefore satisfies Step 1.
STEP 2A Prong One:The independent claim 1 recites the following limitations directed to an abstract idea:
determining, for each image from a database of images, one or more annotations and one or more bounding boxes,
The limitation recites a mental process of observation, evaluation, judgement, and/or opinion capable of being performed by the human mind, or by a human using a pen and paper, by observing and evaluating images in a database of images and for each image, based on the observation and evaluation, making a judgement and/or opinion of one or more annotations and one or more bounding boxes.
wherein each bounding box of the one or more bounding boxes encloses a region of image pixels of a respective image that contains at least a portion of a particular utility asset depicted in the respective image;
The limitation recites a mental process of observation, evaluation, judgement, and/or opinion capable of being performed by the human mind, or by a human using a pen and paper, by observing and evaluating a particular utility asset depicted in the respective image, and based on the observation and evaluation, making a judgement and/or opinion of a bounding box that encloses a region that contains at least a portion of the utility asset.
determining, based on the annotations, a first subset of images from the images that share one or more annotations;
The limitation recites a mental process of observation, evaluation, judgement, and/or opinion capable of being performed by the human mind, or by a human using a pen and paper, by observing and evaluating the annotations and the images, and based on the observation and evaluation, making a judgement and/or opinion of a subset of images that share one or more annotations.
generating, based on the first subset of images and the shared one or more annotations, a textual token representing the first subset of images
The limitation recites a mental process of observation, evaluation, judgement, and/or opinion capable of being performed by the human mind, or by a human using a pen and paper, by observing and evaluating the first subset of images and the shared one or more annotations, and based on the observation and evaluation, making a judgement and/or opinion of a textual token that represents the first subset of images.
determining, based on the image pixels enclosed by the bounding boxes, a second subset of images from the images that share visual features associated with respective utility assets depicted in the respective images; and
The limitation recites a mental process of observation, evaluation, judgement, and/or opinion capable of being performed by the human mind, or by a human using a pen and paper, by observing and evaluating the image pixels enclosed by the bounding boxes, the database of images, and respective utility assets depicted in the respective images, and based on the observation and evaluation, making a judgement and/or opinion of a second subset of images that share visual features associated with the respective utility assets.
generating, based on the second subset of images and the shared visual features, an image token representing the second subset of images
The limitation recites a mental process of observation, evaluation, judgement, and/or opinion capable of being performed by the human mind, or by a human using a pen and paper, by observing and evaluating the second subset of images and the shared visual features, and based on the observation and evaluation, making a judgement and/or opinion of an image token that represents the second subset of images.
STEP 2A Prong Two:Claim 1 recites that the steps are performed using a “database”, which is a high-level recitation of generic computer components and represents mere instructions to apply on a computer as in MPEP 2106.05(f), which does not provide integration into a practical application.
The claim recites the following additional elements:
wherein each annotation is descriptive of a type of utility asset depicted in a respective image and bounded by a particular bounding box,
The limitation recites an insignificant extra-solution activity as selecting a particular type of data being used to represent each annotation as identified in MPEP 2106.05(g) and does not provide integration into a practical application.
storing the textual token in a search index,
The limitation recites a high-level recitation of generic computer components and functions and represents mere instructions to apply on a computer as in MPEP 2106.05(f), which does not provide integration into a practical application.
wherein the textual token comprises a first set of utility asset features representative of the annotations shared by each image in the first subset of images and a corresponding identifier for each image in the first subset of images;
The limitation recites an insignificant extra-solution activity as selecting a particular type of data being used to represent the textual token as identified in MPEP 2106.05(g) and does not provide integration into a practical application.
storing the image token in the search index,
The limitation recites a high-level recitation of generic computer components and functions and represents mere instructions to apply on a computer as in MPEP 2106.05(f), which does not provide integration into a practical application.
wherein the image token comprises an image embedding encoding shared visual features and a corresponding identifier for each image in the second subset of images.
The limitation recites an insignificant extra-solution activity as selecting a particular type of data being used to represent the image token as identified in MPEP 2106.05(g) and does not provide integration into a practical application.
Viewing the additional limitations together and the claim as a whole, nothing provides integration into a practical application.
STEP 2B:
The conclusions for the mere implementation using a computer are carried over and does not provide significantly more.
With respect to “wherein each annotation is descriptive of a type of utility asset depicted in a respective image and bounded by a particular bounding box,” identified as insignificant extra-solution activity above this is also WURC as court-identified see MPEP 2106.05(d)(II)(iv).
With respect to “wherein the textual token comprises a first set of utility asset features representative of the annotations shared by each image in the first subset of images and a corresponding identifier for each image in the first subset of images;” identified as insignificant extra-solution activity above this is also WURC as court-identified see MPEP 2106.05(d)(II)(iv).
With respect to “wherein the image token comprises an image embedding encoding shared visual features and a corresponding identifier for each image in the second subset of images.” identified as insignificant extra-solution activity above this is also WURC as court-identified see MPEP 2106.05(d)(II)(iv).
Looking at the claim as a whole does not change this conclusion and the claim is ineligible.
As per Dependent Claims 2-4,
STEP 1:In accordance with Step 1 of the eligibility inquiry (as explained in MPEP 2106), the claimed method (claims 1-4) is directed to one of the eligible categories of subject matter and therefore satisfies Step 1.
STEP 2A Prong One:The dependent claims 2-4 recite the following limitations directed to an abstract idea:
The limitation of Dependent Claim 3 includes the step(s) of:
wherein the textual token is generated by a machine learning network that comprises a text transformer encoder configured to generate textual embeddings from a clustering of the first set of utility asset features representative of the annotations.
The limitation recites a mathematical concept of executing a mathematical formula/function in the form of a machine learning network comprising a text transformer encoder that outputs the textual token and textual embeddings from a cluster of the first set of utility asset features representative of the annotations.
The limitation of Dependent Claim 4 includes the step(s) of:
wherein the image token is generated by a machine learning network that comprises a vision transformer encoder configured to generate image embeddings from a clustering of the shared visual features.
The limitation recites a mathematical concept of executing a mathematical formula/function in the form of a machine learning network comprising a vision transformer encoder that outputs the image token and image embeddings from a cluster of shared visual features.
STEP 2A Prong Two:The claim(s) recite the following additional elements:
The limitation of Dependent Claim 2 includes the step(s) of:
wherein at least one image is included in both the second subset of images and the first subset of images.
The limitation recites an insignificant extra-solution activity as selecting a particular type of image(s) included/being used to represent the first subset of images and the second subset of images as identified in MPEP 2106.05(g) and does not provide integration into a practical application.
Viewing the additional limitations together and the claim as a whole, nothing provides integration into a practical application.
STEP 2B:
The conclusions for the mere implementation using a computer are carried over and does not provide significantly more.
With respect to Claim 2 reciting “wherein at least one image is included in both the second subset of images and the first subset of images.” identified as insignificant extra-solution activity above this is also WURC when claimed in a merely generic manner as court-identified see MPEP 2106.05(d)(II)(iv).
Looking at the claim as a whole does not change this conclusion and the claim is ineligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4 are rejected under 35 U.S.C. 103 as being unpatentable over Non-Patent Literature Giuseppe et al., "The VISIONE Video Search System: Exploiting Off-the-Shelf Text Search Engines for Large-Scale Video Retrieval", 6 August 2020, Institute of Information Science and Technologies (ISTI), Italian National Research Council (CNR), Via G. Moruzzi 1, 56124 Pisa, Italy (Year: 2020), hereinafter referred to as Giuseppe, and further in view of Jin et al. (U.S. Pre-Grant Publication No. 2023/0206584, hereinafter referred to as Jin).
Regarding Claim 1:
Giuseppe teaches a multi-modal search-based object detection method comprising:
determining, for each image from a database of images, one or more annotations and one or more bounding boxes, wherein each annotation is descriptive of a type of asset depicted in a respective image and bounded by a particular bounding box, and wherein each bounding box of the one or more bounding boxes encloses a region of image pixels of a respective image that contains at least a portion of a particular asset depicted in the respective image;
Giuseppe teaches a database of images by teaching “The subset of the YFCC100M dataset that we used for building the knowledge base was selected by identifying images with relevant textual descriptions and tags. To this scope, we used a metadata cleaning algorithm that leverages on the semantic similarities between images. Its core idea is that if a tag is contained in the metadata of a group of very similar images, then that tag is likely to be relevant for all these images.” (Pages 9-10). Giuseppe further teaches determining annotations and green bounding boxes for an image describing object classes of an asset depicted and bounded and the green box enclosing a region of image pixels that contains the particular asset depicted by teaching “Example of our textual encoding for objects and their spatial locations (second box): the information that the cell a3 contains two cars is encoded as the concatenation a3car a3car. In addition to the position we encode also the number of occurrences of the object in the image (first box): the two person are encoded as person1 person2.” (Page 12 and Figure 7).
determining, based on the annotations, a first subset of images from the images that share one or more annotations;
Giuseppe teaches “if a tag is contained in the metadata of a group of very similar images, then that tag is likely to be relevant for all these images” and “For each image, we have a space-separated concatenation of ENCs, one for all the cells (codloc) in the grid that contains the object (codclass): for example, for the image in Figure 7 the rightmost car is indexed with the sequence fe3car f 3car . . . 5carg where “car” is the codclass of the object car, located in cells e3, f 3, g3, e4, f 4, g4, e5, f 5, g5. This information is stored in the Object&Color BBoxes field of the record associated with the keyframe.” (Page 10 & Figure 7).
generating, based on the first subset of images and the shared one or more annotations, a textual token representing the first subset of images and storing the textual token in a search index,
Giuseppe teaches generating “textual encoding of object classes” and “textual encoding of object bounding boxes” (Figure 7) and “For each image, we have a space-separated concatenation of ENCs, one for all the cells (codloc) in the grid that contains the object (codclass): for example, for the image in Figure 7 the rightmost car is indexed with the sequence fe3car f 3car . . . 5carg where “car” is the codclass of the object car, located in cells e3, f 3, g3, e4, f 4, g4, e5, f 5, g5. This information is stored in the Object&Color BBoxes field of the record associated with the keyframe.” (Page 10 & Figure 7).
wherein the textual token comprises a first set of asset features representative of the annotations shared by each image in the first subset of images and a corresponding identifier for each image in the first subset of images;
Giuseppe teaches “indexing phase and “searching phase” (Figure 5) where as part of the indexing phase “In the full-text search engine, the information extracted from every keyframe is composed of four textual fields, as shown in Figure 5:…Object&Color BBoxes, containing text encoding of colors and objects locations; Object&Color Classes, containing global information on objects and colors in the keyframe;” (Page 9)
determining, based on the image pixels enclosed by the bounding boxes, a second subset of images from the images that share visual features associated with respective assets depicted in the respective images; and
Giuseppe teaches “indexing phase and “searching phase” (Figure 5) where as part of the indexing phase “In the full-text search engine, the information extracted from every keyframe is composed of four textual fields, as shown in Figure 5:…Visual Features, containing text encoding of extracted visual features.” (Page 9).
generating, based on the second subset of images and the shared visual features, an image token representing the second subset of images and storing the image token in the search index, wherein the image token comprises an image embedding encoding shared visual features and a corresponding identifier for each image in the second subset of images.
Giuseppe further teaches “we transform the deep features into a textual encoding suitable for being indexed by a standard full-text search engine. We used the Scalar Quantization-based Surrogate Text representation to transform the deep features into a textual encoding, which was proposed in [56]. The idea behind this approach is to map the real-valued vector components of the R-MAC descriptor into a (sparse) integer vector that acts as the term frequencies vector of a synthetic codebook. Then the integer vector is transformed into a text document by simply concatenating some synthetic codewords so that the term frequency of the i-th codeword is exactly the i-th element of the integer vector. For example, the four-dimensional integer vector [2, 1, 0, 1] is encoded with the text “t1 t1 t2 t4”, where ft1, t2, t3, t4g is a codebook of four synthetic alphanumeric terms.” (Page 13).
Lathia explicitly teaches all of the elements of the claimed invention as recited above except:
an asset as a utility asset;
However, in the related field of endeavor of image processing using a machine learning model, Jin teaches:
an asset as a utility asset;
Jin teaches “pole (or other object) extraction from optical imagery, according to example embodiments. As noted above, mapping and navigation service providers face significant technical challenges with respect to mapping cartographic features across large geographic areas. One area of development for addressing this technical challenge is the use of optical imagery (e.g., optical imagery 101) to facilitate such mapping, particularly for specific object types such as but not limited to poles or pole-like objects. As used herein, poles or pole-like objects generally are made up entirely or at least partially of a long slender section (usually but not necessarily cylindrical). Poles and pole-like objects typically appear frequently in an environment in the form of lamp posts, utility poles, traffic light supports, signposts, columns, and/or the like. Because they are generally installed at fixed positions, they often are mapped as reference points or guides.” (Para. [0029]) where “to enable the keypoint detection deep learning network (e.g., a machine learning model 121 of the machine learning system 105) to learn the different contexts for the bottom point and the top point of the poles, the two semantic keypoints selected for the pole-like object 103 (or any other designated number keypoints) are manually labeled or annotated in a consistent order for every pole-like object 103 in a training dataset comprising optical imagery 101 depicting samples of pole-like objects 103. The size, types of samples, etc. of the training dataset can be selected based on the target levels of generalizability, accuracy, etc. of the trained machine learning model 121. The labeled data is then fed into the keypoint detection deep learning network to train the model 121 to detect the pole-like objects 103 and their two semantic keypoints.” (Para. [0038]).
Thus, it would have been obvious to one of ordinary skill in the art, having the teachings of Jin and Giuseppe at the time that the claimed invention was effectively filed, to have modified the indexing and searching video repository, as taught by Giuseppe, with the geographic database for storing locations/attributes of detected objects, as taught by Jin.
One would have been motivated to make such combination because Jin teaches “a geographic database 125, according to one embodiment. In one embodiment, the geographic database 125 includes geographic data 901 used for (or configured to be compiled to be used for) mapping and/or navigation-related services, such as for providing map embedding analytics according to the embodiments described herein. For example, the map data records stored herein can be used to determine the semantic relationships among the map features, attributes, categories, etc. represented in the geographic data 901. In one embodiment, the geographic database 125 include high definition (HD) mapping data that provide centimeter-level or better accuracy of map features…In one embodiment, the HD mapping data (e.g., HD data records 911) capture and store details such as the slope and curvature of the road, lane markings, roadside objects such as signposts, including what the signage denotes.” (Para. [0093]).
Regarding Claim 2:
Jin and Giuseppe further teach
wherein at least one image is included in both the second subset of images and the first subset of images.
Giuseppe teaches “VISIONE, in fact, integrates several search functionalities and exploits deep learning technologies to mitigate the semantic gap between text and image. Specifically it supports: query by keywords: the user can specify keywords including scenes, places or concepts (e.g., outdoor, building, sport) to search for video scenes; query by object location: the user can draw on a canvas some simple diagrams to specify the objects that appear in a target scene and their spatial locations; query by color location: the user can specify some colors present in a target scene and their spatial locations (similarly to object location above); query by visual example: an image can be used as a query to retrieve video scenes that are visually similar to it.” (Page 5) thereby teaching multiple search functionalities able to produce subsets of images including the at least one image.
Regarding Claim 3:
Jin and Giuseppe further teach:
wherein the textual token is generated by a machine learning network that comprises a text transformer encoder configured to generate textual embeddings from a clustering of the first set of utility asset features representative of the annotations.
Jin further teaches “vision transformers that assemble and progressively combine tokens into image-like representations to represent input images at their full resolutions while maintaining a global receptive field.” (Para. [0075]) and Giuseppe teaches “To take full advantage from these stable search engine technologies, we specifically designed various text encodings for all the features and descriptors extracted from the video keyframes and the user query, and we decided to use the Apache Lucene project. In previous papers, we already exploited the idea of using text encoding, named Surrogate Text Representation [54], to index and search image for deep features [54–57]. In VISIONE, we extend this idea to index also information regarding the position of objects and colors that appear in the images.” (Page 5).
Regarding Claim 4:
Jin and Giuseppe further teach:
wherein the image token is generated by a machine learning network that comprises a vision transformer encoder configured to generate image embeddings from a clustering of the shared visual features.
Jin further teaches “vision transformers that assemble and progressively combine tokens into image-like representations to represent input images at their full resolutions while maintaining a global receptive field.” (Para. [0075]).
Claim(s) 5-19 are rejected under 35 U.S.C. 103 as being unpatentable over Lathia et al. (U.S. Pre-Grant Publication No. 2024/0311421, hereinafter referred to as Lathia) and further in view of Jin et al. (U.S. Pre-Grant Publication No. 2023/0206584, hereinafter referred to as Jin) and Bates et al. (U.S. Patent No. 11,106,706, hereinafter referred to as Bates).
Regarding Claim 5:
Lathia further teaches a multi-modal object search method comprising:
providing, for display on a user device, a user interface configured to receive input representing a search query for one or more images,
Lathia teaches “The image data 12 may be input via a user interface. The user interface can include a search interface of a search application, a marketplace application, a social media application, and/or a viewfinder application. In some implementations, the image data 12 can be associated with a live video feed of a user environment captured via a mobile computing device.” (Para. [0041])
where the user interface permits the input to include at least one of textual data or image data requesting the search query;
Lathia teaches “The image data 12 may be input via a user interface. The user interface can include a search interface of a search application, a marketplace application, a social media application, and/or a viewfinder application. In some implementations, the image data 12 can be associated with a live video feed of a user environment captured via a mobile computing device.” (Para. [0041]).
responsive to receiving a particular search input from the user interface, generating one or more search tokens dependent on whether the particular search input comprises textual data, image data, or both, each of the one or more search tokens encoding features represented by the particular search input;
Lathia teaches “Alternatively and/or additionally, the image can be processed with a tokenizer block to determine and/or generate one or more visual tokens (e.g., tokenized image features). The one or more visual tokens can then be utilized to determine one or more search results.” (Para. [0027])
identifying a set of candidate images responsive to the particular search input from a database of images,
Lathia teaches “the search results may be determined based on both an image embedding and one or more text labels. For example, the one or more text labels may be utilized to determine a plurality of candidate search results, and the image embedding may be utilized to rank the plurality of candidate search results to determine which search results to provide to the user. Alternatively and/or additionally, the image embedding may be utilized to determine a plurality of candidate search results, and the one or more text labels may be utilized to rank the plurality of candidate search results to determine which search results are provided to the user.” (Para. [0026])
where identifying the set of candidate images comprises comparing the search tokens with textual tokens and image tokens stored in a search index, each textual token representing textual descriptions of utility asset features shared by a respective subset of the images within the database, and each image token representing visual features shared by utility assets depicted in a respective subset of the images within the database; and
Lathia teaches “different search techniques may be utilized for searching different datasets. For example, an image embedding without a text label or visual token may be utilized for determining search results for a first dataset (e.g., a general database), and the image embedding with the one or more text labels (and/or with the one or more visual tokens) may be utilized for determining search results for the second dataset (e.g., a specialized database).” (Para. [0035]).
providing, for display on the user device and within the user interface, the candidate images and
Lathia teaches “The one or more first search results and the one or more second search results can then be provided for display in a search results interface.” (Para. [0031]).
Lathia explicitly teaches all of the elements of the claimed invention as recited above except:
images depicting an electric grid asset;
encoding utility asset features; and
positioning at least one candidate image within a respective region of a geographic map of an electric grid that is representative of a geographic location of a particular electrical asset depicted in the at least one candidate image.
However, in the related field of endeavor of image processing using a machine learning model, Jin teaches:
images depicting an electric grid asset;
Jin teaches “pole (or other object) extraction from optical imagery, according to example embodiments. As noted above, mapping and navigation service providers face significant technical challenges with respect to mapping cartographic features across large geographic areas. One area of development for addressing this technical challenge is the use of optical imagery (e.g., optical imagery 101) to facilitate such mapping, particularly for specific object types such as but not limited to poles or pole-like objects. As used herein, poles or pole-like objects generally are made up entirely or at least partially of a long slender section (usually but not necessarily cylindrical). Poles and pole-like objects typically appear frequently in an environment in the form of lamp posts, utility poles, traffic light supports, signposts, columns, and/or the like. Because they are generally installed at fixed positions, they often are mapped as reference points or guides.” (Para. [0029]) where “to enable the keypoint detection deep learning network (e.g., a machine learning model 121 of the machine learning system 105) to learn the different contexts for the bottom point and the top point of the poles, the two semantic keypoints selected for the pole-like object 103 (or any other designated number keypoints) are manually labeled or annotated in a consistent order for every pole-like object 103 in a training dataset comprising optical imagery 101 depicting samples of pole-like objects 103. The size, types of samples, etc. of the training dataset can be selected based on the target levels of generalizability, accuracy, etc. of the trained machine learning model 121. The labeled data is then fed into the keypoint detection deep learning network to train the model 121 to detect the pole-like objects 103 and their two semantic keypoints.” (Para. [0038]).
Features as utility asset features; and
Jin teaches ““to enable the keypoint detection deep learning network (e.g., a machine learning model 121 of the machine learning system 105) to learn the different contexts for the bottom point and the top point of the poles, the two semantic keypoints selected for the pole-like object 103 (or any other designated number keypoints) are manually labeled or annotated in a consistent order for every pole-like object 103 in a training dataset comprising optical imagery 101 depicting samples of pole-like objects 103. The size, types of samples, etc. of the training dataset can be selected based on the target levels of generalizability, accuracy, etc. of the trained machine learning model 121. The labeled data is then fed into the keypoint detection deep learning network to train the model 121 to detect the pole-like objects 103 and their two semantic keypoints.” (Para. [0038]).
Thus, it would have been obvious to one of ordinary skill in the art, having the teachings of Jin and Lathia at the time that the claimed invention was effectively filed, to have modified the systems and methods for a multiple dataset search, as taught by Lathia, with the geographic database for storing locations/attributes of detected objects, as taught by Jin.
One would have been motivated to make such combination because Lathia teaches “ The second dataset may include a specialized dataset associated with specialized search results (e.g., a dataset associated with a specific type of search result (e.g., image search results, location search results, product search results, scholarly search results, and/or verified search results), a dataset associated with a specific object type (e.g., a dataset associated with a particular object), a dataset associated with a specific application (e.g., an image gallery application, a social media application, a shopping application, and/or a music application), and/or a dataset associated with a specific action (e.g., a booking action, a navigation action, a purchase action, and/or an augmented-reality and/or virtual-reality experience action))” (Para. [0032]) without providing further detail about the dataset that provides the location search results, and Jin provides further detail by teaching “the mapping platform 107 can use the object geolocation/attributes 109 to generate digital map data (e.g., as stored in a geographic database 125) indicating locations/attributes of the detected pole-like objects 103.” (Para. [0041]).
Jin and Lathia explicitly teach all of the elements of the claimed invention as recited above except:
positioning at least one candidate image within a respective region of a geographic map of an electric grid that is representative of a geographic location of a particular electrical asset depicted in the at least one candidate image.
However, in the related field of endeavor of geospatial visualization and query tools, Bates teaches:
positioning at least one candidate image within a respective region of a geographic map of an electric grid that is representative of a geographic location of a particular electrical asset depicted in the at least one candidate image.
Bates teaches “providing a portfolio query tool that enables datasets to be linked with geospatial data, and geospatially linked image data, such that generated query results are navigable in multiple views (e.g., a filters view, a map view, a data view, a combination thereof, etc.) by a user through a graphical user interface.” (Col. 4 Lines 1-12) and “query responses can be visualized in a map view that overlays indicia associated with the data onto a geographical map presented in the map view” (Col. 4 Lines 13-20)
Thus, it would have been obvious to one of ordinary skill in the art, having the teachings of Bates, Jin and Lathia at the time that the claimed invention was effectively filed, to have modified the systems and methods for a multiple dataset search, as taught by Lathia, and the geographic database for storing locations/attributes of detected objects, as taught by Jin, with the visualized query responses in a map view that overlays indicia associated with the data onto a geographical map, as taught by Bates.
One would have been motivated to make such combination because Bates teaches “providing a portfolio query tool that enables datasets to be linked with geospatial data, and geospatially linked image data, such that generated query results are navigable in multiple views (e.g., a filters view, a map view, a data view, a combination thereof, etc.) by a user through a graphical user interface. By allowing a user to set custom query parameters, a large dataset (e.g., a managed portfolio of assets) can be viewed and navigated, allowing the user to apply a context-driven, subjective assessment of the suitability of stored data values in geographically defined bounds.” (Col. 4 Lines 1-12) and “the disclosure herein improves the relevant technology of large dataset storage and analysis. Such improvements are brought about by providing a unique set of tools that link data to geospatial information so that queries can be run against the dataset. Moreover, query responses can be visualized in a map view that overlays indicia associated with the data onto a geographical map presented in the map view.” (Col. 4 Lines 13-20).
Regarding Claim 6:
Bates, Jin, and Lathia further teach:
wherein the one or more search tokens are generated by a machine learning network configured to encode the utility asset features from the particular search input,
Lathia teaches “The tokenization block 228 can include one or more machine-learned tokenizers trained to generate and/or determine tokens based on features in an image.” (Para. [0046])
the machine learning network comprising at least one of:
(i) a text transformer encoder configured to generate textual embeddings from a clustering of the textual data; or
Lathia teaches “ the input to the machine-learned model(s) of the present disclosure can be text or natural language data. The machine-learned model(s) can process the text or natural language data to generate an output. As an example, the machine-learned model(s) can process the natural language data to generate a language encoding output. As another example, the machine-learned model(s) can process the text or natural language data to generate a latent text embedding output. As another example, the machine-learned model(s) can process the text or natural language data to generate a translation output. As another example, the machine-learned model(s) can process the text or natural language data to generate a classification output. As another example, the machine-learned model(s) can process the text or natural language data to generate a textual segmentation output. As another example, the machine-learned model(s) can process the text or natural language data to generate a semantic intent output. As another example, the machine-learned model(s) can process the text or natural language data to generate an upscaled text or natural language output (e.g., text or natural language data that is higher quality than the input text or natural language, etc.). As another example, the machine-learned model(s) can process the text or natural language data to generate a prediction output.” (Para. [0110]).
(ii) a vision transformer encoder configured to generate image embeddings from a clustering of the image data.
Lathia teaches “In some implementations, the input to the machine-learned model(s) of the present disclosure can be latent encoding data (e.g., a latent space representation of an input, etc.). The machine-learned model(s) can process the latent encoding data to generate an output. As an example, the machine-learned model(s) can process the latent encoding data to generate a recognition output. As another example, the machine-learned model(s) can process the latent encoding data to generate a reconstruction output. As another example, the machine-learned model(s) can process the latent encoding data to generate a search output. As another example, the machine-learned model(s) can process the latent encoding data to generate a reclustering output. As another example, the machine-learned model(s) can process the latent encoding data to generate a prediction output.” (Para. [0111])Lathia further teaches “In some cases, the input includes visual data and the task is a computer vision task. In some cases, the input includes pixel data for one or more images and the task is an image processing task.”
Jin further teaches a transformer by teaching “vision transformers that assemble and progressively combine tokens into image-like representations to represent input images at their full resolutions while maintaining a global receptive field.” (Para. [0075])
Regarding Claim 7:
Bates, Jin, and Lathia further teach:
responsive to receiving the particular search input and wherein the particular search input comprises textual data:
Lathia teaches “ the input to the machine-learned model(s) of the present disclosure can be text or natural language data. The machine-learned model(s) can process the text or natural language data to generate an output. As an example, the machine-learned model(s) can process the natural language data to generate a language encoding output. As another example, the machine-learned model(s) can process the text or natural language data to generate a latent text embedding output. As another example, the machine-learned model(s) can process the text or natural language data to generate a translation output. As another example, the machine-learned model(s) can process the text or natural language data to generate a classification output. As another example, the machine-learned model(s) can process the text or natural language data to generate a textual segmentation output. As another example, the machine-learned model(s) can process the text or natural language data to generate a semantic intent output. As another example, the machine-learned model(s) can process the text or natural language data to generate an upscaled text or natural language output (e.g., text or natural language data that is higher quality than the input text or natural language, etc.). As another example, the machine-learned model(s) can process the text or natural language data to generate a prediction output.” (Para. [0110]).
obtaining a representative image relevant to the search input; and
Lathia teaches in response to the input, “The one or more first search results and the one or more second search results can then be provided for display in a search results interface.” (Para. [0031]).
providing the representative image for display in the user interface and,
Lathia teaches in response to the input, “The one or more first search results and the one or more second search results can then be provided for display in a search results interface.” (Para. [0031]).
as additional textual data is received from the user interface, identifying, based on the additional textual data, an object within the representative image corresponding to the additional textual data and providing graphical representations of bounding boxes to surround pixels representing the object within the representative image.
Lathia teaches “image data may include one or more bounding boxes generated by an object detection model, one or more classifications generated by one or more classification models, and/or one or more cropped images generated by one or more segmentation models.” (Para. [0052]) and “A user may adjust the search preferences and/or criteria to limit the scope of resources searched, may broaden the scope of the resources search, may select a particular type of search results to obtain, and/or may remove search result type restrictions” (Para. [0028]) thereby teaching providing adjusted/additional input (which can be via text as taught by paragraph [0110]), which adjusts the output with the adjusted search preferences and/or criteria, the output including image data with a bounding box for particular objects in the image.
Regarding Claim 8:
Bates, Jin, and Lathia further teach:
wherein the particular search input comprises image data, and
Lathia teaches “the input to the machine-learned model(s) of the present disclosure can be image data. The machine-learned model(s) can process the image data to generate an output. As an example, the machine-learned model(s) can process the image data to generate an image recognition output (e.g., a recognition of the image data, a latent embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an image segmentation output. As another example, the machine-learned model(s) can process the image data to generate an image classification output. As another example, the machine-learned model(s) can process the image data to generate an image data modification output (e.g., an alteration of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an encoded image data output (e.g., an encoded and/or compressed representation of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an upscaled image data output. As another example, the machine-learned model(s) can process the image data to generate a prediction output.” (Para. [0109])
wherein the representative image relevant to the search input comprises at least one image from the image data.
Lathia teaches “the input to the machine-learned model(s) of the present disclosure can be image data. The machine-learned model(s) can process the image data to generate an output. As an example, the machine-learned model(s) can process the image data to generate an image recognition output (e.g., a recognition of the image data, a latent embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an image segmentation output. As another example, the machine-learned model(s) can process the image data to generate an image classification output. As another example, the machine-learned model(s) can process the image data to generate an image data modification output (e.g., an alteration of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an encoded image data output (e.g., an encoded and/or compressed representation of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an upscaled image data output. As another example, the machine-learned model(s) can process the image data to generate a prediction output.” (Para. [0109]).
Regarding Claim 9:
Bates, Jin, and Lathia further teach:
wherein the search tokens are generated using a first machine learning model,
Lathia teaches producing different tokens using multiple models by teaching “An image can be processed with one or more machine-learned models to determine one or more text labels associated with one or more objects in the image. The one or more text labels can then be utilized as a text query to search for search results associated with the image.” (Para. [0024]) and “Alternatively and/or additionally, the image may be processed with an embedding model (e.g., a machine-learned embedding model) to generate an image embedding that may be descriptive of one or more image features in the image. The use of an image embedding can provide additional detail that may not be captured in a text label-based search” (Para. [0025]) where “In some implementations, the computing system can determine one or more visual tokens associated with the one or more image features. The one or more visual tokens can be determined with a tokenizer block. The tokenizer block may include one or more machine-learned models. In some implementations, the one or more visual tokens can be determined by determining one or more visual tokens associated with a classification for the one or more image features.” (Para. [0055]).
wherein at least one of the textual tokens, the image tokens, or both are generated using a second machine learning model, and
Lathia teaches “An image can be processed with one or more machine-learned models to determine one or more text labels associated with one or more objects in the image. The one or more text labels can then be utilized as a text query to search for search results associated with the image.” (Para. [0024]) and “Alternatively and/or additionally, the image may be processed with an embedding model (e.g., a machine-learned embedding model) to generate an image embedding that may be descriptive of one or more image features in the image. The use of an image embedding can provide additional detail that may not be captured in a text label-based search” (Para. [0025]) where “In some implementations, the computing system can determine one or more visual tokens associated with the one or more image features. The one or more visual tokens can be determined with a tokenizer block. The tokenizer block may include one or more machine-learned models. In some implementations, the one or more visual tokens can be determined by determining one or more visual tokens associated with a classification for the one or more image features.” (Para. [0055]).
wherein the first machine learning model is a lightweight model relative to the second machine learning model.
Lathia teaches “one or more models 120 can be stored and implemented at the user computing device 102 and/or one or more models 140 can be stored and implemented at the server computing system 130.” (Para. [0095]) where “The model trainer 160 can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained” (Para. [0103]) and “the model trainer 160 can train the embedding models 120 and/or 140 based on a set of training data 162. The training data 162 can include, for example, training images, training embeddings, training labels, training image pairs, training image triplets, and/or training classifications.” (Para. [0104]) thereby teaching varying models trained for different environments and on different training data resulting in different weight models.
Regarding Claim 10:
Bates, Jin, and Lathia further teach:
obtaining, by one or more scripts and from a data store, utility asset inventory data; and
Bates teaches “The tool enables a user to query assets in a portfolio (e.g., a large dataset related to geospatially distributed assets, e.g., buildings, property, other fixed location assets, etc.), and navigate through assets in the portfolio geospatially, and via images.” (Col. 11 Lines 15-33)
updating, using the data related to the utility asset, the textual tokens and the image token stored in the search index.
Jin teaches updating the stored asset information by teaching “In one embodiment, the HD mapping data records 911 also include real-time sensor data collected from probe vehicles in the field. The real-time sensor data, for instance, integrates real-time traffic information, weather, and road conditions (e.g., potholes, road friction, road wear, etc.) with highly detailed 3D representations of street and geographic features to provide precise real-time data (e.g., including probe trajectories) also at centimeter-level accuracy. Other sensor data can include vehicle telemetry or operational data such as windshield wiper activation state, braking state, steering angle, accelerator position, and/or the like.” (Para. [0110]) where the tokens stored in the dataset are textual and image tokens taught by Lathia (Para. [0035]) for providing “visually similar search results that may be associated with different resources, different tasks, and/or different actions” (Lathia - Para. [0030]).
Regarding Claim 11:
Bates, Jin, and Lathia further teach:
wherein the utility asset inventory data comprises at least one of a quantity of the particular electrical asset for the geographic location, or a quantity of a defect type for the particular electrical asset.
Bates teaches “when the portfolio database comprises assets such as properties or buildings, such assets have a fixed position. As such, the location of the asset can be visualized geospatially. In the example illustrated, a map of the United States illustrates indicia, e.g., a number of assets located in each state that contains assets.” (Col. 16 Lines 4-19).
Regarding Claim 12:
Bates, Jin, and Lathia further teach:
filtering, based on the textual token and the image token for the input, the images to obtain a filtered subset of images, wherein the filtered subset of images excludes images that do not include at least one token from the textual token and the image token corresponding to the input that match, from the search index, the textual token and the image token;
Lathia teaches “ image data 212 can be obtained, processed to generate an image embedding 216, and utilized to determine one or more search results associated with a plurality of datasets (e.g., a first dataset 218 and a second dataset 222). The plurality of search results (e.g., the set of first search results 220 and the set of second search results 224) may then be ranked and/or filtered based on one or more visual tokens 230.” (Para. [0044]) thereby teaching using tokens to filter images from the datasets.
determining, based on the textual token and the image token, a similarity score for each image in the filtered subset of images, wherein the similarity score indicates a likelihood of a respective image matching the textual token and the image token; and
Lathia teaches “the one or more text labels may be utilized to determine a plurality of candidate search results, and the image embedding may be utilized to rank the plurality of candidate search results to determine which search results to provide to the user. Alternatively and/or additionally, the image embedding may be utilized to determine a plurality of candidate search results, and the one or more text labels may be utilized to rank the plurality of candidate search results to determine which search results are provided to the user.” (Para. [0026]) where the ranking includes relevance scores for the search results indicating relevance/likelihood of a match: “the different sets of search results can be intermingled and provided for display in positions based on relevance scores. For example, at 630, a single search results panel 632 is provided for display in the task bar of the interface. The first search result 626 may be provided for display adjacent to the second search result 628 with the second search result 628 being provided for display in a first position based on a determined relevance score being higher than a determined relevance score for the first search result 626.” (Para. [0071])
identifying the candidate images from the filtered subset of images, the candidate images each having a respective similarity score that exceeds a threshold value.
Lathia teaches results having a score exceeding a threshold value by teaching “search results that meet a relevance threshold based on embedding similarity and/or term based relevance” (Para. [0086]).
Regarding Claim 13:
Bates, Jin, and Lathia further teach:
ranking the candidate images based on the similarity score of the respective candidate image.
Lathia teaches “The first search result 626 may be provided for display adjacent to the second search result 628 with the second search result 628 being provided for display in a first position based on a determined relevance score being higher than a determined relevance score for the first search result 626.” (Para. [0071]).
Regarding Claim 14:
Bates, Jin, and Lathia further teach:
determining that the input comprises the textual data;
Lathia teaches “ the input to the machine-learned model(s) of the present disclosure can be text or natural language data. The machine-learned model(s) can process the text or natural language data to generate an output. As an example, the machine-learned model(s) can process the natural language data to generate a language encoding output. As another example, the machine-learned model(s) can process the text or natural language data to generate a latent text embedding output. As another example, the machine-learned model(s) can process the text or natural language data to generate a translation output. As another example, the machine-learned model(s) can process the text or natural language data to generate a classification output. As another example, the machine-learned model(s) can process the text or natural language data to generate a textual segmentation output. As another example, the machine-learned model(s) can process the text or natural language data to generate a semantic intent output. As another example, the machine-learned model(s) can process the text or natural language data to generate an upscaled text or natural language output (e.g., text or natural language data that is higher quality than the input text or natural language, etc.). As another example, the machine-learned model(s) can process the text or natural language data to generate a prediction output.” (Para. [0110]).
in response to determining that the input comprises the textual data, generating the one or more search tokens based on utility asset features from the textual data; and
Lathia teaches “The tokenization block 228 can include one or more machine-learned tokenizers trained to generate and/or determine tokens” (Para. [0046]) and “the input to the machine-learned model(s) of the present disclosure can be text or natural language data. The machine-learned model(s) can process the text or natural language data to generate an output. As an example, the machine-learned model(s) can process the natural language data to generate a language encoding output. As another example, the machine-learned model(s) can process the text or natural language data to generate a latent text embedding output. As another example, the machine-learned model(s) can process the text or natural language data to generate a translation output. As another example, the machine-learned model(s) can process the text or natural language data to generate a classification output. As another example, the machine-learned model(s) can process the text or natural language data to generate a textual segmentation output. As another example, the machine-learned model(s) can process the text or natural language data to generate a semantic intent output. As another example, the machine-learned model(s) can process the text or natural language data to generate an upscaled text or natural language output (e.g., text or natural language data that is higher quality than the input text or natural language, etc.). As another example, the machine-learned model(s) can process the text or natural language data to generate a prediction output.” (Para. [0110]).Jin teaches the asset features as utility asset features by teaching “pole (or other object) extraction from optical imagery, according to example embodiments. As noted above, mapping and navigation service providers face significant technical challenges with respect to mapping cartographic features across large geographic areas. One area of development for addressing this technical challenge is the use of optical imagery (e.g., optical imagery 101) to facilitate such mapping, particularly for specific object types such as but not limited to poles or pole-like objects. As used herein, poles or pole-like objects generally are made up entirely or at least partially of a long slender section (usually but not necessarily cylindrical). Poles and pole-like objects typically appear frequently in an environment in the form of lamp posts, utility poles, traffic light supports, signposts, columns, and/or the like. Because they are generally installed at fixed positions, they often are mapped as reference points or guides.” (Para. [0029]) where “to enable the keypoint detection deep learning network (e.g., a machine learning model 121 of the machine learning system 105) to learn the different contexts for the bottom point and the top point of the poles, the two semantic keypoints selected for the pole-like object 103 (or any other designated number keypoints) are manually labeled or annotated in a consistent order for every pole-like object 103 in a training dataset comprising optical imagery 101 depicting samples of pole-like objects 103. The size, types of samples, etc. of the training dataset can be selected based on the target levels of generalizability, accuracy, etc. of the trained machine learning model 121. The labeled data is then fed into the keypoint detection deep learning network to train the model 121 to detect the pole-like objects 103 and their two semantic keypoints.” (Para. [0038]).
comparing the one or more search tokens based on the textual data to the textual tokens of the search index.
Lathia teaches comparing search tokens with tokens of a dataset by teaching “The computing system may determine a plurality of general search results by searching the first database based on the image embedding and the one or more visual tokens.” (Para. [0056]). Lathia further teaches comparing text search data with stored text data by teaching “The computing system can determine one or more second search results by searching a second database based on the image embedding and the one or more text labels.” (Para. [0086]).
Regarding Claim 15:
Bates, Jin, and Lathia further teach:
determining that the input comprises the image data;
Lathia teaches “In some implementations, the input to the machine-learned model(s) of the present disclosure can be image data. The machine-learned model(s) can process the image data to generate an output. As an example, the machine-learned model(s) can process the image data to generate an image recognition output (e.g., a recognition of the image data, a latent embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an image segmentation output. As another example, the machine-learned model(s) can process the image data to generate an image classification output. As another example, the machine-learned model(s) can process the image data to generate an image data modification output (e.g., an alteration of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an encoded image data output (e.g., an encoded and/or compressed representation of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an upscaled image data output. As another example, the machine-learned model(s) can process the image data to generate a prediction output.” (Para. [0109])
in response to determining that the input comprises the image data, generating the one or more search tokens based on the utility asset features from the image data; and
Lathia teaches “The tokenization block 228 can include one or more machine-learned tokenizers trained to generate and/or determine tokens” (Para. [0046]) and “In some implementations, the input to the machine-learned model(s) of the present disclosure can be image data. The machine-learned model(s) can process the image data to generate an output. As an example, the machine-learned model(s) can process the image data to generate an image recognition output (e.g., a recognition of the image data, a latent embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an image segmentation output. As another example, the machine-learned model(s) can process the image data to generate an image classification output. As another example, the machine-learned model(s) can process the image data to generate an image data modification output (e.g., an alteration of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an encoded image data output (e.g., an encoded and/or compressed representation of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an upscaled image data output. As another example, the machine-learned model(s) can process the image data to generate a prediction output.” (Para. [0109]).
Jin teaches the asset features as utility asset features by teaching “pole (or other object) extraction from optical imagery, according to example embodiments. As noted above, mapping and navigation service providers face significant technical challenges with respect to mapping cartographic features across large geographic areas. One area of development for addressing this technical challenge is the use of optical imagery (e.g., optical imagery 101) to facilitate such mapping, particularly for specific object types such as but not limited to poles or pole-like objects. As used herein, poles or pole-like objects generally are made up entirely or at least partially of a long slender section (usually but not necessarily cylindrical). Poles and pole-like objects typically appear frequently in an environment in the form of lamp posts, utility poles, traffic light supports, signposts, columns, and/or the like. Because they are generally installed at fixed positions, they often are mapped as reference points or guides.” (Para. [0029]) where “to enable the keypoint detection deep learning network (e.g., a machine learning model 121 of the machine learning system 105) to learn the different contexts for the bottom point and the top point of the poles, the two semantic keypoints selected for the pole-like object 103 (or any other designated number keypoints) are manually labeled or annotated in a consistent order for every pole-like object 103 in a training dataset comprising optical imagery 101 depicting samples of pole-like objects 103. The size, types of samples, etc. of the training dataset can be selected based on the target levels of generalizability, accuracy, etc. of the trained machine learning model 121. The labeled data is then fed into the keypoint detection deep learning network to train the model 121 to detect the pole-like objects 103 and their two semantic keypoints.” (Para. [0038]).
comparing the one or more search tokens based on the image data to the image tokens of the search index.
Lathia teaches comparing search tokens with tokens of a dataset by teaching “The computing system may determine a plurality of general search results by searching the first database based on the image embedding and the one or more visual tokens.” (Para. [0056]).
Regarding Claim 16:
Some of the limitations herein are similar to some or all of the limitations as recited in Claim 5.
Bates, Jin, and Lathia further teach a system for multi-modal electric grid object search, the system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations; (Lathia – Para. [0005])
Regarding Claim 17:
All of the limitations herein are similar to some or all of the limitations as recited in Claim 6.
Regarding Claim 18:
All of the limitations herein are similar to some or all of the limitations as recited in Claim 7.
Regarding Claim 19:
All of the limitations herein are similar to some or all of the limitations as recited in Claim 9.
Claim(s) 20 is rejected under 35 U.S.C. 103 as being unpatentable over Bates, Jin, and Lathia and further in view of Non-Patent Literature Giuseppe et al., "The VISIONE Video Search System: Exploiting Off-the-Shelf Text Search Engines for Large-Scale Video Retrieval", 6 August 2020, Institute of Information Science and Technologies (ISTI), Italian National Research Council (CNR), Via G. Moruzzi 1, 56124 Pisa, Italy (Year: 2020), hereinafter referred to as Giuseppe.
Regarding Claim 20:
Bates, Jin, and Lathia explicitly teach all of the elements of the claimed invention as recited above except:
determining, for each image from a database of images, one or more annotations and one or more bounding boxes, wherein each annotation is descriptive of a type of asset depicted in a respective image and bounded by a particular bounding box, and wherein each bounding box of the one or more bounding boxes encloses a region of image pixels of a respective image that contains at least a portion of a particular asset depicted in the respective image;
determining, based on the annotations, a first subset of images from the images that share one or more annotations;
generating, based on the first subset of images and the shared one or more annotations, a textual token representing the first subset of images and storing the textual token in a search index,
wherein the textual token comprises a first set of asset features representative of the annotations shared by each image in the first subset of images and a corresponding identifier for each image in the first subset of images;
determining, based on the image pixels enclosed by the bounding boxes, a second subset of images from the images that share visual features associated with respective assets depicted in the respective images; and
generating, based on the second subset of images and the shared visual features, an image token representing the second subset of images and storing the image token in the search index, wherein the image token comprises an image embedding encoding shared visual features and a corresponding identifier for each image in the second subset of images.
However, in the related field of a visione video indexing and search system, Giuseppe teaches:
determining, for each image from a database of images, one or more annotations and one or more bounding boxes, wherein each annotation is descriptive of a type of asset depicted in a respective image and bounded by a particular bounding box, and wherein each bounding box of the one or more bounding boxes encloses a region of image pixels of a respective image that contains at least a portion of a particular asset depicted in the respective image;
Giuseppe teaches a database of images by teaching “The subset of the YFCC100M dataset that we used for building the knowledge base was selected by identifying images with relevant textual descriptions and tags. To this scope, we used a metadata cleaning algorithm that leverages on the semantic similarities between images. Its core idea is that if a tag is contained in the metadata of a group of very similar images, then that tag is likely to be relevant for all these images.” (Pages 9-10). Giuseppe further teaches determining annotations and green bounding boxes for an image describing object classes of an asset depicted and bounded and the green box enclosing a region of image pixels that contains the particular asset depicted by teaching “Example of our textual encoding for objects and their spatial locations (second box): the information that the cell a3 contains two cars is encoded as the concatenation a3car a3car. In addition to the position we encode also the number of occurrences of the object in the image (first box): the two person are encoded as person1 person2.” (Page 12 and Figure 7).
determining, based on the annotations, a first subset of images from the images that share one or more annotations;
Giuseppe teaches “if a tag is contained in the metadata of a group of very similar images, then that tag is likely to be relevant for all these images” and “For each image, we have a space-separated concatenation of ENCs, one for all the cells (codloc) in the grid that contains the object (codclass): for example, for the image in Figure 7 the rightmost car is indexed with the sequence fe3car f 3car . . . 5carg where “car” is the codclass of the object car, located in cells e3, f 3, g3, e4, f 4, g4, e5, f 5, g5. This information is stored in the Object&Color BBoxes field of the record associated with the keyframe.” (Page 10 & Figure 7).
generating, based on the first subset of images and the shared one or more annotations, a textual token representing the first subset of images and storing the textual token in a search index,
Giuseppe teaches generating “textual encoding of object classes” and “textual encoding of object bounding boxes” (Figure 7) and “For each image, we have a space-separated concatenation of ENCs, one for all the cells (codloc) in the grid that contains the object (codclass): for example, for the image in Figure 7 the rightmost car is indexed with the sequence fe3car f 3car . . . 5carg where “car” is the codclass of the object car, located in cells e3, f 3, g3, e4, f 4, g4, e5, f 5, g5. This information is stored in the Object&Color BBoxes field of the record associated with the keyframe.” (Page 10 & Figure 7).
wherein the textual token comprises a first set of asset features representative of the annotations shared by each image in the first subset of images and a corresponding identifier for each image in the first subset of images;
Giuseppe teaches “indexing phase and “searching phase” (Figure 5) where as part of the indexing phase “In the full-text search engine, the information extracted from every keyframe is composed of four textual fields, as shown in Figure 5:…Object&Color BBoxes, containing text encoding of colors and objects locations; Object&Color Classes, containing global information on objects and colors in the keyframe;” (Page 9)
determining, based on the image pixels enclosed by the bounding boxes, a second subset of images from the images that share visual features associated with respective assets depicted in the respective images; and
Giuseppe teaches “indexing phase and “searching phase” (Figure 5) where as part of the indexing phase “In the full-text search engine, the information extracted from every keyframe is composed of four textual fields, as shown in Figure 5:…Visual Features, containing text encoding of extracted visual features.” (Page 9).
generating, based on the second subset of images and the shared visual features, an image token representing the second subset of images and storing the image token in the search index, wherein the image token comprises an image embedding encoding shared visual features and a corresponding identifier for each image in the second subset of images.
Giuseppe further teaches “we transform the deep features into a textual encoding suitable for being indexed by a standard full-text search engine. We used the Scalar Quantization-based Surrogate Text representation to transform the deep features into a textual encoding, which was proposed in [56]. The idea behind this approach is to map the real-valued vector components of the R-MAC descriptor into a (sparse) integer vector that acts as the term frequencies vector of a synthetic codebook. Then the integer vector is transformed into a text document by simply concatenating some synthetic codewords so that the term frequency of the i-th codeword is exactly the i-th element of the integer vector. For example, the four-dimensional integer vector [2, 1, 0, 1] is encoded with the text “t1 t1 t2 t4”, where ft1, t2, t3, t4g is a codebook of four synthetic alphanumeric terms.” (Page 13).
Thus, it would have been obvious to one of ordinary skill in the art, having the teachings of Giuseppe, Bates, Jin and Lathia at the time that the claimed invention was effectively filed, to have modified the systems and methods for a multiple dataset search, as taught by Lathia, and the geographic database for storing locations/attributes of detected objects, as taught by Jin, and the visualized query responses in a map view that overlays indicia associated with the data onto a geographical map, as taught by Bates, with the image annotation approach, as taught by Giuseppe.
One would have been motivated to make such combination because Giuseppe teaches “This system is based on an unsupervised image annotation approach that exploits the knowledge implicitly existing in a huge collection of unstructured texts describing images, allowing us to annotate the images without using a specified trained model. The advantage is that the target vocabulary we used for the annotation reflects well the way people actually describe their pictures. Specifically, our system uses the tags and the descriptions contained in the metadata of a large set of media selected from the Yahoo Flickr Creative Commons 100 Million (YFCC100M) dataset [58]. Those tags are validated using WordNet [59], cleaned and then used as the knowledge base for the automatic annotation.” (Page 9)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Bhaisaheb et al. (U.S. Pre-Grant Publication No. 2024/0119046) teaches program synthesis for weakly-supervised multimodal question answering using filtered iterative back-translation (FIBT). Existing approaches for chart question answering mainly address structural, visual, relational, or simple data retrieval queries with fixed-vocabulary answers. The present disclosure implements a two-stage approach where, in first stage, a computer vision pipeline is employed to extract data from chart images and store in a generic schema. In second stage, SQL programs for Natural Language (NL) queries are generated in dataset by using FIBT. To adapt forward and backward models to required NL queries, a Probabilistic Context-Free Grammar is defined, whose probabilities are set to be inversely proportional to SQL programs in training data and sample programs from it. Compositional similarity-based filtration strategy employed on the NL queries generated for these SQL programs enables synthesizing, filtering, and appending NL query-SQL program pairs to training data, iteratively moving towards required NL query distribution.
Gerber (U.S. Pre-Grant Publication No. 2025/0258843) teaches using a machine learning model to generate configuration parameters for a relationship computing model. The configuration parameters can include a time period for a plurality of variables, an alert threshold parameter for the plurality of variables, a lower relationship threshold parameter for each of the plurality of variables, and an upper relationship threshold parameter for each of the plurality of variables. The computing device can configure the relationship computing model using the configuration parameters. The computing device can execute, in some cases using a machine learning language processing model, the configured relationship computing model to generate relationships for pairs of the variables. Based on the execution, the computing device can generate an alert indicating a set of pairs of variables that correspond to a relationship value exceeding an alert threshold.
Ferreira Moreno et al. (U.S. Pre-Grant Publication No. 2022/0198211) teaches automatically identifying, structure and retrieve spatial and/or temporal sequences of digital media content according to semantic specification. Digital media content can be received and information from digital media content can be extracted. Based on the information, a knowledge graph can be constructed or structured to include at least one of spatial and temporal representation of the digital media content. A search query can be received associated with the digital media content. Based on traversing the knowledge graph structure according to at least one of spatial and temporal criterion mapped from the search query, new digital media content can be composed which meets the search query.
Krishnan et al. (U.S. Pre-Grant Publication No. 2023/0306087) teaches retrieving one or more one or more multimodal assets includes receiving a search query for searching for one or more multimodal assets from among a plurality of candidate multimodal assets, encoding the search query into one or more query embedding representations via a trained query representation machine-learning (ML) model, comparing, via a matching unit, the one or more query embedding representations to a plurality of multimodal tensor representations, each of the plurality of multimodal tensor representations being a representation of one of the plurality of candidate multimodal assets, and identifying, based on the comparison, at least one of the plurality of the candidate multimodal assets as a search result for the search query, and providing the at least one of the plurality of the candidate multimodal assets for display as the search result.
Deschaintre et al. (U.S. Pre-Grant Publication No. 2025/0078387) teaches a material search computing system generates a joint feature comparison space by combining joint image-text features of surface material data objects. The joint feature comparison space is a consistent comparison space. The material search computing system extracts a query joint feature set from a query data object that includes text data or image data. In addition, the material search computing system compares the query joint feature set to the joint image-text features included in the joint feature comparison space. Based on the comparison, the material search computing system identifies a result joint feature set and associated result surface material data objects. The material search computing system generates material query result data describing the result surface material data objects, and provides the material query result data to an additional computing system.
Kirmse et al. (U.S. Pre-Grant Publication No. 2016/0005147) teaches geographical image processing of time-dependent imagery. Various assets acquired at different times are stored and processing according to acquisition date in order to generate one or more image tiles for a geographical region of interest. The different image tiles are sorted based on asset acquisition date. Multiple image tiles for the same region of interest may be available. In response to a user request for imagery as of a certain date, one or more image tiles associated with assets from prior to that date are used to generate a time-based geographical image for the user.The reference further teaches “providing an identification of available points in time for which images are available for a geographical location, the identification being provided by a processor of a computer; receiving a request for an image associated with the geographical location for one of the available points in time at the computer; and in response to the request, the computer providing the image associated with the requested geographical location. Portions of the provided image comprise different images of the geographical location captured at different points in time. The different images are selected from a plurality of images comprising images captured before and after the requested point in time. Furthermore, the different images included in the provided image were captured prior to the requested point in time.” (Para. [0009])
Evans et al. (U.S. Pre-Grant Publication No. 2008/0021726) teaches linking one or more disparate community awareness management (CAM) datasets for a community awareness program (CAP) with one or more spatial layers to create linked CAM datasets. One or more data attributes common to a CAM dataset and a spatial layer are identified, and the link is defined between the CAM dataset and the spatial layer. The spatial layer and the linked CAM dataset then may be queried using a single input query. Features from the spatial layer and features from the linked CAM dataset that match the query are generated for display. In one embodiment, a system and method manage CAP assets, transactions, interest areas for the CAP, and buffer areas for the CAP. An audience utility enables entering and maintaining audience data for the CAP. A journal utility enables making journal entries for one or more audience members, CAP assets, transactions, and/or other CAM data. A link document utility enables linking one or more documents to CAM data.The reference further teaches “a CAP asset for an electrical power CAP includes a power transmission line, system, or substation, and a buffer area for the CAP includes an area around the power transmission line, system, or substation.” (Para. [0041]) and “The data management system 606 processes the request, and transmits a response to the image processor 612 with the image data that matches the request. The image processor 612 processes the image data, and generates one or more map images to the program management system 602. In this example, the map images include a map image depicting the geographic area identified by the selected zip code and identifying the point features for the contacts within the zip code, an overview map image identifying the selected zip code and the surrounding zip codes within the selected range, and a map legend identifying the feature data on the map image.” (Para. [0171]).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT F MAY whose telephone number is (571)272-3195. The examiner can normally be reached Monday-Friday 9:30am to 6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Boris Gorney can be reached on 571-270-5626. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROBERT F MAY/Examiner, Art Unit 2154 4/16/2026
/BORIS GORNEY/Supervisory Patent Examiner, Art Unit 2154
Prosecution Insights