Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Notice of Pre-AIA or AIA Status
Claims 1-20 are pending in this application. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 01/09/2025, 06/25/2025, 5/12/2026 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Poddar (US 2021/0118442) and in view of Kharbanda (US 12,266,065).
Re Claim 1, Poddar discloses a method for triggering an intelligent dialogue through an audio-visual reality, executed in a cloud server, the method comprising:
initiating a reality image interface in a user device, wherein a camera is activated to obtain an environment image, and/or a microphone is activated to obtain an environment audio ([0008], user is walking and points a camera at a mural and asks “what is that?”);
receiving location information and a reality image request from the user device, and obtaining an environment object by identifying the environment image, and/or obtaining an environment sound by identifying the environment audio ([0119],-[0121], user points the target object with the camera of the client system and asks “what is this?” The system receives this image and also is able to determine the position/location of the object in question.);
receiving an intelligent dialogue request generated from the user device ([0008], user asks “what is that”. The assistant replies back what “that” is i.e. “the mural you are looking at is Bouquet by Jet Martinez”); and
initiating an intelligent dialogue interface and introducing a chatbot in the intelligent dialogue program, wherein, based on the location information, the environment object, and/or the environment sound, a natural language model is run to generate a dialogue content ([0006], [0008], the assistant system analyzes user input using natural language understanding. It determines what “that” is in the users’ request when asked “Hey Assistant, what is that?” The assistant retrieves additional information from a knowledge base and issues a response. The Assistant performs multi-modal dialog).
Poddar does not disclose, however Kharbanda discloses triggering an intelligent dialogue link point displayed on the reality image interface to activate an intelligent dialogue program (col. 9, lines 23-55, col. 10, lines 42-65, Image augmentation model obtains a query and outputs an augmented-reality experience that includes visual indications of generative model responses. The labels with respect to the objects annotate the image data with the response using generative language models relevant to the model-generated responses. Figs 6a-6c show the labels utilizing language models over the image and starts an input query and response.).
It would have been obvious for one of ordinary skill in the art before the date the current invention was effectively filed to have modified the teachings of Poddar’s visual data assistant with Kharbanda’s generative model responses to include visual indications in the augmented-reality responses. One of ordinary skill in the art would have been motivated to incorporate the teachings with one another in order to generate augmented image data as a response to the user query to help with the user’s task/query at hand
In Re claim 2, Poddar discloses wherein, in the user device, an intelligent model is used to process the environment image and the environment audio that are obtained to identify the environment object and the environment sound surrounding the user device that are then provided to the cloud server and become a basis for the natural language model to generate the dialogue content ([0006]-[0008], uses natural-language process to determine when the user asks “what is that” [0119],-[0121], user points the target object with the camera of the client system and asks “what is this?” The system receives this image and also is able to determine the position/location of the object in question.).
In Re claim 3, Poddar discloses wherein the cloud server further obtains a user preference from user data received from the user device, and obtains real-time environment information through one or more external systems, so that the chatbot further generates the dialogue content based on the user preference and the real-time environment information ([0119]-[0122], user points the target object with the camera of the client system and asks “what is this?” User utilizes a user profile of the user requesting the information to help determine what “that” is in reference with the target object in question.)
In Re claim 4, Poddar discloses wherein a content input by a user is received through the intelligent dialogue interface, and a semantic feature of the content input by the user is obtained, so that the chatbot is configured to run the natural language model to generate the dialogue content based on the semantic feature of the content input by the user, the user preference, and/or the real-time environment information (([0006]-[0008], uses natural-language process to determine when the user asks “what is that” [0119]-[0122], user points the target object with the camera of the client system and asks “what is this?” The assistant retrieves additional information from a knowledge base and issues a response. The Assistant performs multi-modal dialog.).
In Re claim 5, Poddar discloses wherein the dialogue content generated by the natural language model running in the chatbot includes providing multiple recommended options, multiple recommended audio-visual contents, and/or multiple recommended friend links ([0140], the user prediction and proactive user recommendation is able to reply to the question “Hey Assistant, where did I leave my keys?” with “You left them on the kitchen counter at 3pm today” which includes multiple specific entities such as the kitchen counter and the time that it was done).
In Re claim 6, Poddar discloses wherein, in the user device, an intelligent model is used to process the environment image and the environment audio that are obtained to identify the environment object and the environment sound surrounding the user device that are then provided to the cloud server and become a basis for the natural language model to generate the dialogue content ([0077]-[0079], [0116], machine-learning models that are intelligent processes the assistant system that is able to receive an image from a user pointing a camera to the target object and picking audio of the user asking “what is that?” by using natural language understanding [0006]) .
In Re claim 7, one of ordinary level of skill in the art would have been compelled to make the proposed modification to Poddar for the same reasons identified in the rejection of claim 1. In addition, Kharbanda discloses wherein multiple visual ranges corresponding to multiple viewing angles for the location information are calculated based on the location information, a database is queried based on the reality image request and the calculated multiple visual ranges to obtain one or more pieces of location-based data, and the cloud server sends link information of the one or more pieces of location-based data within each of the multiple visual ranges to the user device (col. 9, lines 23-55, col. 10, lines 42-65, Col. 15, lines 45-60, the image data may include a plurality of features associated with individuals, structures, locations, products, and/or other environment features. Machine learned models in the server computing system which stores information consists of natural language data to process the text to generate an output Image augmentation model. It is able to obtain queries and outputs an augmented-reality image with visual indications of generative model responses. The labels with respect to the objects annotate the image data with the response using generative language models relevant to the model-generated responses. Figs 6a-6c show the labels utilizing language models over the image and starts an input query and response.).;
wherein, in the reality image interface, one or more link icons linking the one or more pieces of location-based data are marked in the reality image interface based on respective spatial locations of the one or more pieces of location-based data within each of the multiple visual ranges (col. 9, lines 23-55, col. 10, lines 42-65, Col. 15, lines 45-60, the image data may include a plurality of features associated with individuals, structures, locations, products, and/or other environment features. machine learned models in the server computing system consists of natural language data to process the text to generate an output. Image augmentation model obtains a query and outputs an augmented-reality experience that includes visual indications of generative model responses.).
In Re claim 8, one of ordinary level of skill in the art would have been compelled to make the proposed modification to Poddar for the same reasons identified in the rejection of claim 1. In addition, Kharbanda discloses wherein the reality image interface displays a reality image of each of the multiple visual ranges captured by the user device using the camera, and the reality image is combined with the one or more link icons marked in the one or more spatial locations to form an augmented-reality image (col. 9, lines 23-65, the image augmentation model processes the input from the user and able to detect objects in the image. Boxes are generated with response information and placed in position associated with the target object the label/response is intended for. The image that is queried by the user is of user interest regarding the image of the location environment the user is located.).
In Re claim 9, Poddar discloses wherein, in the user device, an intelligent model is used to process the environment image and the environment audio that are obtained to identify the environment object and the environment sound surrounding the user device that are then provided to the cloud server and become a basis for the natural language model to generate the dialogue content ([0077]-[0079], [0116], machine-learning models that are intelligent processes the assistant system that is able to receive an image from a user pointing a camera to the target object and picking audio of the user asking “what is that?” by using natural language understanding [0006]).
In Re claim 14, Poddar discloses wherein the database includes an audio-visual database that allows the user device to access through a network audio-visual contents stored in the audio-visual database and uploaded and shared by users at each ends of the system ([0009], [0118]-[0121], computer vision (CV) module of the assistant system performs light-weight tagging of entities and contexts of the images and store this information as visual state of the users’ field of view which is then stored in a multimodal dialog state. The assistant system uses the CV module to analyze the visual data to identify one or more objects portrayed in the images to identify as the target object the user is querying about.);
wherein the database includes a user database that stores and updates user data along the progression of time, and records a historical dialogue record of the users used as a dialogue record for learning by a machine learning algorithm in the natural language model ([0122], utilizes previous user’s query about “what is that?” in the past and utilizes the confidence values to these objects to predict the target object the user is querying.).
In Re claim 15, Poddar discloses wherein the database includes a map database to allow each of the users to query location-based data associated with a specific geographic location or spatial coordinates ([0149]-[0151], storing images along with location information for the user in which determines the respective identity of each person. The user may request “Hey Assistant, where are my keys?” to the target object. The plurality of images of the visual data stored in the visual state are continually updated by the CV module with the latest information of the latest target object).
One of ordinary level of skill in the art would have been compelled to make the proposed modification to Poddar for the same reasons identified in the rejection of claim 1. In addition, Kharbanda discloses wherein the database includes a vector database for recording structured information of various texts, pictures, and the audio-visual contents on which vectorization calculations are performed, and the vector database is used to compare various data that matches user personalization (col. 9, lines 23-65, the image augmentation model processes the input from the user and able to detect objects in the image. Boxes are generated with response information and placed in position associated with the target object the label/response is intended for. The image that is queried by the user is of user interest regarding the image of the location environment the user is located. )
In Re claim 19, one of ordinary level of skill in the art would have been compelled to make the proposed modification to Poddar for the same reasons identified in the rejection of claim 1. In addition, Kharbanda discloses wherein the natural language model running in the cloud server uses a transformer model to perform processes of machine translation, document summarization, and document generation to generate the dialogue content Kharbanda (col. 25, lines 28-49, col. 29, lines 32-62, col. 30 line 58 – col. 31 line 15, machine learned models in the server computing system consists of natural language data to process the text to generate an output. Fig. 11a-11b, input from the image is provided via the query input. The natural language data is processed by the machine translation, then the search results are gathered, and then response is generated in a Natural language format).
In Re claim 20, one of ordinary level of skill in the art would have been compelled to make the proposed modification to Poddar for the same reasons identified in the rejection of claim 1. In addition, Kharbanda discloses wherein, in the cloud server, a vector algorithm is executed on a content input by a user, user interests, real-time environment information, and one or more pieces of location-based data within each of multiple visual ranges to mark a text that is obtained, calculate a vector of each of words, and obtain relevant content based on a vector distance between each of the words, and generate the dialogue content matching the user interests and the real-time environment information. Kharbanda (col. 9, lines 23-65, the image augmentation model processes the input from the user and able to detect objects in the image. Boxes are generated with response information and placed in position associated with the target object the label/response is intended for. The image that is queried by the user is of user interest regarding the image of the location environment the user is located.)
Re claims 10-13, and 16-18, they are similar to claims 1, 3-5, and 7-9, and therefore are rejected for the same reasons above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HO T SHIU whose telephone number is (571)270-3810. The examiner can normally be reached Mon-Fri (9:00am - 5:00pm).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Nicholas Taylor can be reached at 571-272-3089. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HO T SHIU/Examiner, Art Unit 2443
HO T. SHIU
Examiner
Art Unit 2443
/CHRISTOPHER B ROBINSON/Primary Examiner, Art Unit 2443
Prosecution Insights