DETAILED ACTION
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
2. This action is in response to amendment filed on 7/7/2006, in which claims 29 – 42 and 44 - 48 was presented for examination.
3. Claims 29 – 42 and 44 - 48 are now pending in the application.
Response to Arguments
4. Applicant’s arguments with respect to claims 29 – 42 and 44 - 48 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Remarks
5. As per claim 1, applicants’ argues in substance in pages 7 – 10 that Ekambaram et al (US 2019/0005327 A1), Ohazama et al (US 2013/0132396 A1)), Patel (WO 2019/212501 A1), and Yada et al (US 2020/0356592 A1) does not disclose retrieving, from the site database based on the identified physical location, an object feature model associated with the physical location, the object feature model comprising a site- specific computer vision model trained to recognize objects known to be present at the physical location
In response to applicants’ argument, Examiner respectfully responds that the combine teaching of Ekambaram et al (US 2019/0005327 A1), Cichon et al (US 2017/0308800 A1), Patel (WO 2019/212501 A1), and Yada et al (US 2020/0356592 A1) discloses each and every features of claim 1 including the features of retrieving, from the site database based on the identified physical location, an object feature model associated with the physical location, the object feature model comprising a site-specific computer vision model trained to recognize objects known to be present at the physical location (Cichon: para.[0074] - para.[0078]).
Cichon discloses integration of machine vision and imagery system (see para.[0002]). The system includes algorithm mixer that allow user input to select one or more vision algorithm (see para.[0006]). A variety source of algorithm provide the algorithms to the algorithm mixer engine data flow. The source of algorithms may include AI trained model database, AI algorithm repository, AI trainer repository, and AI material database (see para.[0062]). The algorithm models are stored in a central database or repository. The computer vision algorithm recorded image or video for search and analysis (see para.[0075]). Once the algorithm models are stored in a repository, they can provide services based on virtual reality. For instances, if a man is at the beach, only the beach machine vision module would be activated or published from the repository to render service to the user based on his location (see para.[0075] – para.[0078]).
As per claim 34, applicant argues in substance in pages 9 – 11 that Ekambaram et al (US 2019/0005327 A1), Ohazama et al (US 2013/0132396 A1)), Patel (WO 2019/212501 A1), and Yada et al (US 2020/0356592 A1) does not disclose generating a plurality of unified queries based on information from the language-based query and the extracted object features, determining validity of the plurality of unified queries, and filtering the plurality of unified queries to eliminate invalid queries.
In response to applicants’ argument, Examiner respectfully respond that the combine teaching of Ekambaram et al (US 2019/0005327 A1), Cichon et al (US 2017/0308800 A1), Patel (WO 2019/212501 A1), Yada et al (US 2020/0356592 A1), and Bhoite et al (US 2020/0401591 A1) discloses generating a plurality of unified queries based on information from the language-based query and the extracted object features, determining validity of the plurality of unified queries, and filtering the plurality of unified queries to eliminate invalid queries (Bhoite: para.[0020]).
Bhoite discloses receiving a uniform query from external module. The uniform query is validated or verify to obtain a modified uniform query. The modified uniform query are transmitted for execution (see para.[0040]).
Thus, the rejection is maintained.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
6. Claims 29 – 31, 33, 35, 37 – 40, 45, and 47 – 48 are rejected under 35 U.S.C. 103 as being unpatentable over Ekambaram et al (US 2019/0005327 A1), in view of Cichon et al (US 2017/0308800 A1), in view of Patel (WO 2019/212501 A1), and further in view of Yada et al (US 2020/0356592 A1).
As per claim 29, Ekambaram et al (US 2019/0005327 A1) discloses,
A method of obtaining an answer to a query relating to a physical object from a knowledge base (para.[0032]; “the system may store the at least one image
and the environmental context of the object in a remote data storage location” and para.[0034]; “accessing and retrieving the object may include searching for the object using the one or more features identified in the request. ……..the system may rank or weight the object based upon a context of the user and present the object which the system thinks the user is most likely trying to access”).
the method comprising: obtaining an image of the physical object (para.[0019]; “capture one or more images and/or details of an object that is of interest to a user”).
identifying a physical location associated with the physical object (para.[0019]; “identify and capture an environmental context of the object. The environmental context may identify a plurality of features of the environment surrounding the object”).
retrieving an object feature model associated with the physical location (para.[0028]; “environmental context may identify features of the environment surrounding the object”).
the object feature model comprising object features associated with objects present at the physical location (para.[0029]; “Context information may include …….location information, ( e.g., GPS coordinates of where was the object encountered,
home, work, etc.), identification of other objects in the surrounding environment (e.g, people, things, animals, etc.), events corresponding to or associated with encountering the object ( e.g., a birthday party, a meeting, etc.), characteristics of the object (e.g., model, make, type, features, etc.)”).
the object features comprising features associated with the objects present at the physical location (para.[0002]; “identifying and capturing an environmental context of the object, wherein the environmental context (i) identifies a plurality of features of the environment surrounding the object”).
Ekambaram does not specifically disclose retrieving, from the site database based on the identified physical location, an object feature model associated with the physical location, the object feature model comprising a site-specific computer vision model.
However, Cichon et al (US 2017/0308800 A1) in an analogous art discloses,
retrieving, from the site database based on the identified physical location, an object feature model associated with the physical location, the object feature model comprising a site-specific computer vision model (para.[0075]; “the computer vision algorithm models to be used on historically recorded images/videos for retrospective video search and analysis”, para.[0076]; “Once the algorithm models are stored in a repository ….. If the man were on the beach, only the beach machine vision modules would be published from the repository”, and para.[0077]; “As a defined location threshold was met, the specific house computer vision models would be published for consumption by the computer vision hardware ……. Custom content or models could be published from the repository by pull or search requests by the user”).
Therefore, it would have been obvious to one of ordinary skill in the art before the invention was filed to incorporate specific location vision model of the system of Cichon
into identification of environmental information association with observed objects of the system of Ekambaram to provide the user with any relation between the displayed map and the advertised business, such as by showing store locations
on the map.
Neither Ekambaram nor Cichon specifically disclose the object feature model comprising a site-specific computer vision model trained to recognize objects known to be present at the physical location.
However, Patel (WO 2019/212501 A1) in an analogous art discloses,
the method comprising: obtaining an image of the physical object (NOTE: para.[0012]; “capture digital images .. The digital images 102 may include digital video and/or digital still images of objects 103-1 ... 103-N in a real-world physical environment”).
identifying a physical location associated with the physical object (NOTE: para.[0012]; “objects 103-1 ... 103-N may be people, places, things etc. that are present and/or visible to the digital image capturing device in the physical environment”).
retrieving an object feature model associated with the physical location (NOTE: para.[0009]; “train, from the portion of the unique object localized in the digital images, a new specific model for recognizing the unique object”).
the object feature model comprising a site-specific computer vision model trained to recognize objects known to be present at the physical location (para.[0027]; “The specific model 108 may be trained to specifically identify a unique object 103-2 appearing in the digital images 102 based on its unique characteristics relative to other objects”).
extracting object features from the image using the object feature model (NOTE: para.[0026]; “specific model 108 may be a model that may be utilized to identify a unique object 103-2 based on the unique characteristics of that object relative to other objects of the same type”).
the object features comprising features associated with the objects present at the physical location (NOTE: para.[0026]; “specific model 108 may be a model that may be utilized to identify a unique object 103-2 based on the unique characteristics of that object relative to other objects of the same type”).
Therefore, it would have been obvious to one of ordinary skill in the art before the invention was filed to incorporate specific model train to identify object of the system of Patel into specific location vison model of the system of Cichon to identify object associated with specific environment, thereby improving the quality of information presented to the user in the system of Ekambaram.
Neither Ekambaram nor Cichon nor Patel specifically disclose extracting object features from the image using the object feature model, receiving a language-based query associated with the physical object, analyzing the language-based query to identify an aspect of the language-based query, combining information from the language-based query and the extracted object features based on the identified aspect of the query to form a unified query for submission to the knowledge base, and submitting the unified query to the knowledge base to obtain an answer to the unified.
However, Yada et al (US 2020/0356592 A1) in an analogous art discloses,
extracting object features from the image using the object feature model (para.[0059]; “image analysis engine 518 will identify the image as a picture of the Roman Coliseum”)
receiving a language-based query associated with the physical object (para.[0055]; “A text analysis engine 516 performs analysis on the input text provided by the input capture system 506, to provide text information”).
analyzing the language-based query to identify an aspect of the language-based query (para.[0057]; “The text information provided by the text analysis engine 516 can include the original text submitted by the user together with analysis results generated by the text analysis engine”)
combining information from the language-based query and the extracted object features based on the identified aspect of the query to form a unified query for submission to the knowledge base (para.[0059]; “the text-based retrieval path, a query expansion component 526 generates a reformulated text query by using the image information generated by the image analysis engine 518 to supplement the text information provided by the text analysis engine” and para.[0060]; “the information produced by the image analysis engine 518 supplements the text information
provided by the text analysis engine”).
and submitting the unified query to the knowledge base to obtain an answer to the unified query (para.[0062]; “the query-processing engine 504 provides query
results to the user in response to the submitted plural-mode query”).
and providing the answer for use in performing operations related to the physical object (para.[0050]; “uses an image-based retrieval engine to provide the query results. In this case, the image results include images that the image-based retrieval engine deems similar to an input image”).
Therefore, it would have been obvious to one of ordinary skill in the art before the invention was filed to incorporate customization of image-based query of the system of Yada into specific model train to identify object of the system of Patel and specific location specific vision model of the system of Cichon to replace an ambiguous term in the input text with a term obtained from the image analysis engine, thereby enabling improved the quality of search results return to user in the system of Ekambaram.
As per claim 30, the rejection of claim 29 is incorporated and further Ekambaram et al (US 2019/0005327 A1) discloses,
further comprising: analyzing the image to identify the physical location (para.[0002]; “identifying and capturing an environmental context of the object, wherein the environmental context (i) identifies a plurality of features of the environment surrounding the object”).
As per claim 31, the rejection of claim 30 is incorporated and further Ekambaram et al (US 2019/0005327 A1) discloses,
wherein retrieving the object feature model is performed in response to identifying the physical location (para.[0002]; “searching for the at least one of the plurality of features and (ii) retrieving the at least one image of an object corresponding to the at least one of the plurality of features”).
As per claim 33, the rejection of claim 29 is incorporated and further Ekambaram et al (US 2019/0005327 A1) discloses,
wherein the object feature model comprises at least one of: a site-specific computer vision model associated with the physical location and models of objects present at the physical location (para.[0028]; “identify and capture an
environmental context of the object. The environmental context may identify features of the environment surrounding the object when the object is encountered”).
As per claim 35 the rejection of claim 29 is incorporated and further Yada et al (US 2020/0356592 A1) discloses,
wherein the aspect of the language-based query comprises an intent of the language-based query, and wherein the method comprises analyzing the language-based query with a sequence classification model to determine the intent of the language-based query (para.[0057]; “text analysis results can include domain information, intent information, slot information, part-of-speech information, parse-tree information, one or more text-based semantic vectors, etc” and para.[0079]; “the image analysis engine 518 uses intent information provided by the text analysis component 516 to determine that the user is asking a question about a plant condition”).
Therefore, it would have been obvious to one of ordinary skill in the art before the invention was filed to incorporate customization of image-based query of the system of Yada into specific model train to identify object of the system of Patel to replace an ambiguous term in the input text with a term obtained from the image analysis engine, thereby enabling improved the quality of search results return to user in the system of Ekambaram.
As per claim 37 the rejection of claim 29 is incorporated and further Yada et al (US 2020/0356592 A1) discloses,
wherein the aspect of the language-based query comprises an entity associated with the language-based query, and wherein the method comprises analyzing the language-based query with a token classification model to determine the entity associated with the language-based query (para.[0085]; “The intent determination component determines an intent associated with a user's input query. An intent corresponds to an objective that a user likely wishes to accomplish by submitting an input message”).
Therefore, it would have been obvious to one of ordinary skill in the art before the invention was filed to incorporate customization of image-based query of the system of Yada into specific model train to identify object of the system of Patel to replace an ambiguous term in the input text with a term obtained from the image analysis engine, thereby enabling improved the quality of search results return to user in the system of Ekambaram.
Claims 38 – 40, 45, and 47 are system claim corresponding to method claims 29 – 31, 35, and 37 respectively, and rejected under the same reason set forth in connection to the rejection of claims 29 – 31, 35, and 37 respectively above.
Claim 48 is a knowledge base interface system claim corresponding to method claim 29, and rejected under the same reason set forth in connection to the rejection of claim 29 above.
7. Claims 32, 36, 41 - 42, and 46 are rejected under 35 U.S.C. 103 as being unpatentable over Ekambaram et al (US 2019/0005327 A1), in view of Cichon et al (US 2017/0308800 A1), in view of Patel (WO 2019/212501 A1), in view of Yada et al (US 2020/0356592 A1), and further in view of Panuganty et al (US 2021/0248136 A1).
As per claim 32, the rejection of claim 29 is incorporated, Yada et al (US 2020/0356592 A1) further discloses,
wherein combining information from the language-based query and the extracted object features comprises obtaining the missing information from the extracted object features and combining the missing information obtained from the extracted object features with the information from the language-based query (para.[0102]; “the query expansion component 526 generally operates by supplementing the text information with information obtained from the analysis performed by the image analysis engine”).
Ekambaram et al (US 2019/0005327 A1), Cichon et al (US 2017/0308800 A1), Patel (WO 2019/212501 A1), and Yada et al (US 2020/0356592 A1) does not specifically disclose analyzing the language-based query to identify missing information that is missing from the language-based query.
However, Panuganty et al (US 2021/0248136 A1) in an analogous art discloses,
further comprising analyzing the language-based query to identify missing information that is missing from the language-based query (para.[0232]; “identification module 1402 analyzes the canonical query to determine whether contextual information is missing from the query”).
Therefore, it would have been obvious to one of ordinary skill in the art before the invention was filed to incorporate identification of contextual information missing from the query of the system of Panuganty into customization of image-based query of the system of Yada and specific model train to identify object of the system of Patel to differentiate the data records into different groups for query output, thereby enabling an accurate characterization of query output (Panuganty: abstract).
As per claim 36, the rejection of claim 35 is incorporated, Ekambaram et al (US 2019/0005327 A1), Cichon et al (US 2017/0308800 A1), Patel (WO 2019/212501 A1), and Yada et al (US 2020/0356592 A1) does not specifically disclose wherein the missing information is identified based on the determined intent of the language-based query.
However, Panuganty et al (US 2021/0248136 A1) in an analogous art discloses,
wherein the missing information is identified based on the determined intent of the language-based query (para.[0232]; “identification module 1402 analyzes the canonical query to determine whether contextual information is missing from the query”).
Therefore, it would have been obvious to one of ordinary skill in the art before the invention was filed to incorporate identification of contextual information missing from the query of the system of Panuganty into customization of image-based query of the system of Yada and specific model train to identify object of the system of Patel to differentiate the data records into different groups for query output, thereby enabling an accurate characterization of query output (Panuganty: abstract).
Claims 41 - 42 and 46 are system claim corresponding to method claims 32 and 36 respectively, and rejected under the same reason set forth in connection to the rejection of claims 32 and 36 respectively above.
8. Claim 34 is rejected under 35 U.S.C. 103 as being unpatentable over Ekambaram et al (US 2019/0005327 A1), in view of Cichon et al (US 2017/0308800 A1), in view of Patel (WO 2019/212501 A1), in view of Yada et al (US 2020/0356592 A1), and further in view of Bhoite et al (US 2020/0401591 A1).
Ekambaram et al (US 2019/0005327 A1), Cichon et al (US 2017/0308800 A1), Patel (WO 2019/212501 A1), and Yada et al (US 2020/0356592 A1) does not specifically disclose generating a plurality of unified queries based on information from the language-based query and the extracted object features, determining validity of the plurality of unified queries, and filtering the plurality of unified queries to eliminate invalid queries.
However, Bhoite et al (US 2020/0401591 A1) in an analogous art discloses,
further comprising: generating a plurality of unified queries based on information from the language-based query and the extracted object features, determining validity of the plurality of unified queries, and filtering the plurality of unified queries to eliminate invalid queries (para.[0040]; “external module 124 may receive a uniform query to validate, verify, and modify the uniform query such that external module 124 transmits a modified uniform query”).
Therefore, it would have been obvious to one of ordinary skill in the art before the invention was filed to incorporate validation of uniform query of the system of Bhoite into customization of image-based query of the system of Yada and specific model train to identify object of the system of Patel to provide avenue where a uniform query can be generated for providing common access to information sources, thereby enabling system to retrieve concise information that improve user experience.
Claim 44 is a system claim corresponding to method claim 34, and rejected under the same reason set forth in connection to the rejection of claim 34 above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
TITLE: Methods circuits assemblies devices systems platforms and functionally associated machine executable code for computer vision assisted construction site inspection, US 2018/0082414 A1 authors: Rozenberg et al.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AUGUSTINE KUNLE OBISESAN whose telephone number is (571)272-2020. The examiner can normally be reached 9:00am - 5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ajay Bhatia can be reached at (571) 272-3906. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AUGUSTINE K. OBISESAN/
Primary Examiner
Art Unit 2156
9/15/2026
Prosecution Insights