DETAILED ACTION
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
2. This action is in response to amendment filed on 2/16/2026, in which claim 29 and 38 was amended, and claims 29 - 48 was presented for examination.
3. Claims 29 – 48 are now pending in the application.
Continued Examination Under 37 CFR 1.114
4. A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 2/16/2026 has been entered.
Response to Arguments
5. Applicant’s arguments with respect to claims 29 - 48 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
6. Claims 29 – 31, 33 – 35, 37 – 40, 43 – 45, and 47 – 48 are rejected under 35 U.S.C. 103 as being unpatentable over Ekambaram et al (US 2019/0005327 A1), in view of Ohazama et al (US 2013/0132396 A1), in view of Patel (WO 2019/212501 A1), and further in view of Yada et al (US 2020/0356592 A1).
As per claim 29, Ekambaram et al (US 2019/0005327 A1) discloses,
A method of obtaining an answer to a query relating to a physical object from a knowledge base (para.[0032]; “the system may store the at least one image
and the environmental context of the object in a remote data storage location” and para.[0034]; “accessing and retrieving the object may include searching for the object using the one or more features identified in the request. ……..the system may rank or weight the object based upon a context of the user and present the object which the system thinks the user is most likely trying to access”).
the method comprising: obtaining an image of the physical object (para.[0019]; “capture one or more images and/or details of an object that is of interest to a user”).
identifying a physical location associated with the physical object (para.[0019]; “identify and capture an environmental context of the object. The environmental context may identify a plurality of features of the environment surrounding the object”).
retrieving an object feature model associated with the physical location (para.[0028]; “environmental context may identify features of the environment surrounding the object”).
the object feature model comprising object features associated with objects present at the physical location (para.[0029]; “Context information may include …….location information, ( e.g., GPS coordinates of where was the object encountered,
home, work, etc.), identification of other objects in the surrounding environment (e.g, people, things, animals, etc.), events corresponding to or associated with encountering the object ( e.g., a birthday party, a meeting, etc.), characteristics of the object (e.g., model, make, type, features, etc.)”).
the object features comprising features associated with the objects present at the physical location (para.[0002]; “identifying and capturing an environmental context of the object, wherein the environmental context (i) identifies a plurality of features of the environment surrounding the object”).
Ekambaram does not specifically disclose retrieving, from the site database based on the identified physical location, an object feature model associated with the physical location, the object feature model comprising a site- specific computer vision model.
However, Ohazama et al (US 2013/0132396 A1) in an analogous art discloses,
retrieving, from the site database based on the identified physical location, an object feature model associated with the physical location, the object feature model comprising a site-specific computer vision model (para.[0012]; “providing
location location-specific content for a particular entity having a plurality of physical locations. The location information of each of the plurality of physical locations, in
correlation with one or more keywords associated with the particular entity, are stored in a database” and para.[0034]; “the data may include images stored in a variety of
formats such as vector-based images or bitmap images”).
Therefore, it would have been obvious to one of ordinary skill in the art before the invention was filed to incorporate location specific content database of the system of Ohazama into identification of environmental information association with observed objects of the system of Ekambaram to provide the user with any relation between the displayed map and the advertised business, such as by showing store locations
on the map.
Neither Ekambaram nor Ohazama specifically disclose the object feature model comprising a site-specific computer vision model trained to recognize objects known to be present at the physical location.
However, Patel (WO 2019/212501 A1) in an analogous art discloses,
the method comprising: obtaining an image of the physical object (NOTE: para.[0012]; “capture digital images .. The digital images 102 may include digital video and/or digital still images of objects 103-1 ... 103-N in a real-world physical environment”).
identifying a physical location associated with the physical object (NOTE: para.[0012]; “objects 103-1 ... 103-N may be people, places, things etc. that are present and/or visible to the digital image capturing device in the physical environment”).
retrieving an object feature model associated with the physical location (NOTE: para.[0009]; “train, from the portion of the unique object localized in the digital images, a new specific model for recognizing the unique object”).
the object feature model comprising a site-specific computer vision model trained to recognize objects known to be present at the physical location (para.[0027]; “The specific model 108 may be trained to specifically identify a unique object 103-2 appearing in the digital images 102 based on its unique characteristics relative to other objects”).
extracting object features from the image using the object feature model (NOTE: para.[0026]; “specific model 108 may be a model that may be utilized to identify a unique object 103-2 based on the unique characteristics of that object relative to other objects of the same type”).
the object features comprising features associated with the objects present at the physical location (NOTE: para.[0026]; “specific model 108 may be a model that may be utilized to identify a unique object 103-2 based on the unique characteristics of that object relative to other objects of the same type”).
Therefore, it would have been obvious to one of ordinary skill in the art before the invention was filed to incorporate specific model train to identify object of the system of Patel into location specific content database of the system of Ohazama to identify object associated with specific environment, thereby improving the quality of information presented to the user in the system of Ekambaram.
Neither Ekambaram nor Ohazama nor Patel specifically disclose extracting object features from the image using the object feature model, receiving a language-based query associated with the physical object, analyzing the language-based query to identify an aspect of the language-based query, combining information from the language-based query and the extracted object features based on the identified aspect of the query to form a unified query for submission to the knowledge base, and submitting the unified query to the knowledge base to obtain an answer to the unified.
However, Yada et al (US 2020/0356592 A1) in an analogous art discloses,
extracting object features from the image using the object feature model (para.[0059]; “image analysis engine 518 will identify the image as a picture of the Roman Coliseum”)
receiving a language-based query associated with the physical object (para.[0055]; “A text analysis engine 516 performs analysis on the input text provided by the input capture system 506, to provide text information”).
analyzing the language-based query to identify an aspect of the language-based query (para.[0057]; “The text information provided by the text analysis engine 516 can include the original text submitted by the user together with analysis results generated by the text analysis engine”)
combining information from the language-based query and the extracted object features based on the identified aspect of the query to form a unified query for submission to the knowledge base (para.[0059]; “the text-based retrieval path, a query expansion component 526 generates a reformulated text query by using the image information generated by the image analysis engine 518 to supplement the text information provided by the text analysis engine” and para.[0060]; “the information produced by the image analysis engine 518 supplements the text information
provided by the text analysis engine”).
and submitting the unified query to the knowledge base to obtain an answer to the unified query (para.[0062]; “the query-processing engine 504 provides query
results to the user in response to the submitted plural-mode query”).
and providing the answer for use in performing operations related to the physical object (para.[0050]; “uses an image-based retrieval engine to provide the query results. In this case, the image results include images that the image-based retrieval engine deems similar to an input image”).
Therefore, it would have been obvious to one of ordinary skill in the art before the invention was filed to incorporate customization of image-based query of the system of Yada into specific model train to identify object of the system of Patel and location specific content database of the system of Ohazama to replace an ambiguous term in the input text with a term obtained from the image analysis engine, thereby enabling improved the quality of search results return to user in the system of Ekambaram.
As per claim 30, the rejection of claim 29 is incorporated and further Ekambaram et al (US 2019/0005327 A1) discloses,
further comprising: analyzing the image to identify the physical location (para.[0002]; “identifying and capturing an environmental context of the object, wherein the environmental context (i) identifies a plurality of features of the environment surrounding the object”).
As per claim 31, the rejection of claim 30 is incorporated and further Ekambaram et al (US 2019/0005327 A1) discloses,
wherein retrieving the object feature model is performed in response to identifying the physical location (para.[0002]; “searching for the at least one of the plurality of features and (ii) retrieving the at least one image of an object corresponding to the at least one of the plurality of features”).
As per claim 33, the rejection of claim 29 is incorporated and further Ekambaram et al (US 2019/0005327 A1) discloses,
wherein the object feature model comprises at least one of: a site-specific computer vision model associated with the physical location and models of objects present at the physical location (para.[0028]; “identify and capture an
environmental context of the object. The environmental context may identify features of the environment surrounding the object when the object is encountered”).
As per claim 34 the rejection of claim 29 is incorporated and further Yada et al (US 2020/0356592 A1) discloses,
further comprising: generating a plurality of unified queries based on information from the language-based query and the extracted object features; determining validity of the plurality of unified queries; and filtering the plurality of unified queries to eliminate invalid queries (para.[0060]; “the information produced by the image analysis engine 518 supplements the text information provided by the text analysis engine ……….model selection component 528 can map information provided by the text analysis engine 516 into an identifier of a classification model to be applied to the user's input image”).
Therefore, it would have been obvious to one of ordinary skill in the art before the invention was filed to incorporate customization of image-based query of the system of Yada into specific model train to identify object of the system of Patel to replace an ambiguous term in the input text with a term obtained from the image analysis engine, thereby enabling improved the quality of search results return to user in the system of Ekambaram.
As per claim 35 the rejection of claim 29 is incorporated and further Yada et al (US 2020/0356592 A1) discloses,
wherein the aspect of the language-based query comprises an intent of the language-based query, and wherein the method comprises analyzing the language-based query with a sequence classification model to determine the intent of the language-based query (para.[0057]; “text analysis results can include domain information, intent information, slot information, part-of-speech information, parse-tree information, one or more text-based semantic vectors, etc” and para.[0079]; “the image analysis engine 518 uses intent information provided by the text analysis component 516 to determine that the user is asking a question about a plant condition”).
Therefore, it would have been obvious to one of ordinary skill in the art before the invention was filed to incorporate customization of image-based query of the system of Yada into specific model train to identify object of the system of Patel to replace an ambiguous term in the input text with a term obtained from the image analysis engine, thereby enabling improved the quality of search results return to user in the system of Ekambaram.
As per claim 37 the rejection of claim 29 is incorporated and further Yada et al (US 2020/0356592 A1) discloses,
wherein the aspect of the language-based query comprises an entity associated with the language-based query, and wherein the method comprises analyzing the language-based query with a token classification model to determine the entity associated with the language-based query (para.[0085]; “The intent determination component determines an intent associated with a user's input query. An intent corresponds to an objective that a user likely wishes to accomplish by submitting an input message”).
Therefore, it would have been obvious to one of ordinary skill in the art before the invention was filed to incorporate customization of image-based query of the system of Yada into specific model train to identify object of the system of Patel to replace an ambiguous term in the input text with a term obtained from the image analysis engine, thereby enabling improved the quality of search results return to user in the system of Ekambaram.
Claims 38 – 40, 43 – 45, and 47 are system claim corresponding to method claims 29 – 31, 33 – 35, and 37 respectively, and rejected under the same reason set forth in connection to the rejection of claims 29 – 31, 33 – 35, and 37 respectively above.
Claim 48 is a knowledge base interface system claim corresponding to method claim 29 29, and rejected under the same reason set forth in connection to the rejection of claim 29 above.
7. Claims 32, 36, 41 - 42, and 46 are rejected under 35 U.S.C. 103 as being unpatentable over Ekambaram et al (US 2019/0005327 A1), in view of Ohazama et al (US 2013/0132396 A1), in view of Patel (WO 2019/212501 A1), in view of Yada et al (US 2020/0356592 A1), and further in view of Panuganty et al (US 2021/0248136 A1).
As per claim 32, the rejection of claim 29 is incorporated, Yada et al (US 2020/0356592 A1) further discloses,
wherein combining information from the language-based query and the extracted object features comprises obtaining the missing information from the extracted object features and combining the missing information obtained from the extracted object features with the information from the language-based query (para.[0102]; “the query expansion component 526 generally operates by supplementing the text information with information obtained from the analysis performed by the image analysis engine”).
Ekambaram et al (US 2019/0005327 A1), Ohazama et al (US 2013/0132396 A1), Patel (WO 2019/212501 A1), and Yada et al (US 2020/0356592 A1) does not specifically disclose analyzing the language-based query to identify missing information that is missing from the language-based query.
However, Panuganty et al (US 2021/0248136 A1) in an analogous art discloses,
further comprising analyzing the language-based query to identify missing information that is missing from the language-based query (para.[0232]; “identification module 1402 analyzes the canonical query to determine whether contextual information is missing from the query”).
Therefore, it would have been obvious to one of ordinary skill in the art before the invention was filed to incorporate identification of contextual information missing from the query of the system of Panuganty into customization of image-based query of the system of Yada and specific model train to identify object of the system of Patel to differentiate the data records into different groups for query output, thereby enabling an accurate characterization of query output (Panuganty: abstract).
As per claim 36, the rejection of claim 35 is incorporated, Ekambaram et al (US 2019/0005327 A1), Ohazama et al (US 2013/0132396 A1), Patel (WO 2019/212501 A1), and Yada et al (US 2020/0356592 A1) does not specifically disclose wherein the missing information is identified based on the determined intent of the language-based query.
However, Panuganty et al (US 2021/0248136 A1) in an analogous art discloses,
wherein the missing information is identified based on the determined intent of the language-based query (para.[0232]; “identification module 1402 analyzes the canonical query to determine whether contextual information is missing from the query”).
Therefore, it would have been obvious to one of ordinary skill in the art before the invention was filed to incorporate identification of contextual information missing from the query of the system of Panuganty into customization of image-based query of the system of Yada and specific model train to identify object of the system of Patel to differentiate the data records into different groups for query output, thereby enabling an accurate characterization of query output (Panuganty: abstract).
Claims 41 - 42 and 46 are system claim corresponding to method claims 32 and 36 respectively, and rejected under the same reason set forth in connection to the rejection of claims 32 and 36 respectively above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AUGUSTINE KUNLE OBISESAN whose telephone number is (571)272-2020. The examiner can normally be reached 9:00am - 5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ajay Bhatia can be reached at (571) 272-3906. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AUGUSTINE K. OBISESAN/
Primary Examiner
Art Unit 2156
4/4/2026
Prosecution Insights