Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability should not be negated by the way the invention was made.
Claims 1-4, 6-9, 11, 13, 15-16, and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Barut (US patent number 12,045,288) in view of Ciecko (US 2019/0272336) and Zhang (US 2023/0325996).
As per claims 1, 8, and 18 Barut teaches, a method and method comprising: generating, using one or more multi-modal language models (Barut, fig.1 170, Multi-modal transformer model ), one or more embeddings associated with one or more portions of one or more images (Barut, fig.2 image data 202 to 208 object detector to 212 object embeddings represent one or more embeddings associated with one or more portions of one or more images as it comes in serial form from the image data); determining, based at least on a query, at least an embedding of the one or more embeddings that is associated with a portion of the one or more portions (Barut, fig.2 Natural language 220 represents a query then to word embedding data 222, and the word would represent the portion); determining, based at least on the embedding, that an image of the one or more images is associated with the query (Barut, fig.2 “Object/Entity matching” 218); and sending, to a client device associated with the query, at least one of image data representative of the image or data indicating a location of the portion within the image (Barut, fig.1 Relative location data 121 represents data indicating a location of the portion within the image).
Barut doesn’t clearly teach, determining, based at least on a query that indicates at least an object and positional information for the object, and determining, based at least on the embedding portion, that the object is represented at a location within an image of the one or more images, determining, based at least on the location within the image corresponding to the positional information indicated by the query, that the image is associated with the query.
Ciecko teaches, based at least on a query that indicates at least an object and positional information for the object (Ciecko, fig.5A 500 “query” and 540 “location and object type info”. While the specification of applicant has a more detail view of positional information it is also a location information as several examples are given in the applicant’s specification, and since no details have been given in the claim language positional can be interpreted as location and following 541 “visual image info” could also be interpreted as applicant’s examples in the specification that aren’t in the claim language. In other words, for example, positional in the art could mean, pixel alignment, however according to the specification of applicant it is more of a location of the image object as well as some visual info, example third car from the left. However, no specifics in the claim language), and determining, based at least on the embedding portion, that the object is represented at a location within an image of the one or more images (Ciecko, fig.5A 545 location is being found with 550), determining, based at least on the location within the image corresponding to the positional information indicated by the query, that the image is associated with the query (Ciecko, fig.5A, 555 the files are then returned based on the query).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Barut with those of Ciecko of having the ability to have a query that identifies Object, location and other image information as part of the query and have those images returned.
The motivation would have been to improve system time performance as taught by Ciecko ¶ [0010] “In this manner, only information of particular interest to the device user is displayed for viewing, and the information retrieval process is accelerated.”.
Barut in view of Ciecko doesn’t clearly teach, indicating a desired location within an image at which the object is to be presented and determining location at which the object is represented corresponding to the desired location at which the positional information indicates the object is represented.
Zhang teaches, indicating a desired location within an image at which the object is to be presented (Zhang, fig.7G 728 bounding box indicates the desired location of object 734 ) and determining location at which the object is represented corresponding to the desired location at which the positional information indicates the object is represented (Zhang, fig.7G then 736 shows the determined location and the positional information of that object so it lines up with the image, see ¶[0153] “As shown by FIG. 7G, the object recommendation system 106 further modifies the recommendation in response to yet further user input. As shown, the object recommendation system 106 detects a selection of another foreground object image 734 and generates a composite image 736 using the background image 710 and the foreground object image 734. In particular, the object recommendation system 106 still positions the foreground object image 734 at a location and using a scale indicated by the query bounding box 728 to generate the composite image 736.” Also the system uses multi-modal networks as well ¶ [0204] “or instance, in some cases, the object recommendation system 106 utilizes one of the multi-modal embedding neural networks described therein as the image search engine 1412c.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Zhang with those of Barut in view of Ciecko to be able to indicate a desired location within an image at which the object is to be presented and determining location at which the object is represented corresponding to the desired location at which the positional information indicates the object is represented as seen in fig.7G.
The motivation would have been to improve user experience to create a more dynamic query as seen in fig.7F-G.
As per claims 2 and 9, Barut in view of Ciecko and Zhang teaches, the method of claim 1, wherein: further comprising receiving, from the client device, second data representative of the query, wherein the query includes one or more first words indicating the object and one or more second words indicating the positional information for the object (Ciecko, fig.5A, 542 matching location and image info would represent second words indicating the positional information for the object)
As per claims 3, 11, 13, and 19 Barut in view of Ciecko and Zhang teaches, the method of claim 1, further comprising: receiving, from the client device, a selection associated with the image (Barut, fig.1 172 selected object data); determining, based at least on the selection and using at least one of the embedding or a second embedding associated with the image, a third embedding of the one or more embeddings that is associated with a second portion of the one or more portions (Barut, fig.1 123 word embedding and seeing multiple embeddings); determining, based at least on the second embedding, that a second image of the one or more images is associated with the query; and sending, to the client device, at least one of second image data representative of the second image or second data indicating a second location of the second portion within the second image (Barut, fig.1-4 as the system loops around there will be a second image and a second location of each image. In other words, as different queries get used this would inherently happen, and see fig.1 123 for multiple embeddings. Anything to do with second locations the system would loop and do the tasks creating second locations).
As per claim 4, Barut in view of Ciecko and Zhang teaches, the method of claim 1, wherein the portion of the image is associated with a first object indicated by the query, and wherein the method further comprises: determining, based at least on a second object indicated by the query, at least a second embedding of the one or more embeddings that is associated with a second portion of the one or more portions; and determining, based at least on the second embedding, that the image of the one or more images is again associated with the query (Barut, claim 7 “determining second image data representing at least a second object,…” This represents second embedding in the second object in a second portion).
As per claims 6 and 15, Barut in view of Ciecko and Zhang teaches, the method of claim 1, further comprising: segmenting the one or more images into the one or more portions (Barut, fig.1 object detector 115 would be segmenting ); determining one or more locations associated with the one or more portions within the one or more images (Barut, fig.1 121 relative location data); determining one or more identifiers that associate the one or more portions with the one or more images (Barut, (50) “The speech processing enabled device may also send metadata (e.g., including device identifiers, device type data, contextual data, IP address data, room location data, etc.) to the orchestrator 330.“ This represents determining one or more identifiers ); and storing, in one or more databases, the one or more embeddings, second data representative of the one or more locations, and third data representative of the one or more identifiers (Barut, (30) “Additionally, memory 103 may store data described herein, such as one or more model parameters, query data, result data, training data, etc.” This represents the storing).
As per claims 7 and 16, Barut in view of Ciecko and Zhang teaches, the method of claim 1, further comprising: determining a second embedding based at least on at least one of an image, text, or an inputted embedding from the query, wherein the determining the at least the embedding from the one or more embeddings is based at least on the second embedding (Barut, fig.1 123 Word Embedding Data 123, this represents embedding based at least on at least one of text).
Allowable Subject Matter
Claims 5, 10, 12, 14, 17 and 20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. Further details of these claims place the application in condition of allowance.
Prior art to consider when making further amendments:
Vanchinathan, US patent 12,665,982, Abstract, Abstract, Systems and techniques are provided for processing image data. First image data of a scene can be obtained with a first zoom level and including at least a foreground portion and a background portion. A user input can be received indicative of an adjustment to increase or decrease a zoom level of the foreground relative to the background, corresponding to a second zoom level greater or less than the first zoom level. A second image data of the scene can be obtained, based on the adjustment and using the second zoom level, to include an adjusted foreground portion associated with the second zoom level. The adjusted foreground portion can be segmented from the second image data of the scene. A composite image can be generated based on combining the segmented foreground portion from the second image data with at least a portion of the first image data.
Wu, US Patent 12,411,862 Abstract, “To search for digital twins (avatars) on a computerized platform, a semantic embedding is made of the search query, and this is compared with respective semantic embeddings generated for digital twins on the computerized platform, to determine a first set digital twins whose semantic embeddings are like that of the search query. A trained graph neural network generates respective graph embeddings of feature data of the first set of digital twins, and these graph embeddings are compared with graph embeddings that the trained graph neural network generates for other digital twins on the platform, to determine a second set of digital twins whose graph embeddings are similar to graph embeddings of the first set of digital twins. The search results returned in response to the search query may include at least one digital twin in the second set of digital twins.”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SANTIAGO GARCIA whose telephone number is (571)270-5182. The examiner can normally be reached Monday-Friday 9:30am-5:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SANTIAGO GARCIA/Primary Examiner, Art Unit 2673
/SG/
