Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
1. This Office Action is in response to the RCE filed on 03/23/2026.
Claims 2, 8 and 15 have been amended.
Claim 1 has been canceled.
Claims 2-21 are pending.
Information Disclosure Statement
2. The information disclosure statement (IDS) filed on 04/30/2026 and 12/22/2025 comply with the provisions of M.P.E.P. 609. The examiner has considered it.
Response to Arguments
3. Applicant's arguments with respect to claims 2-21 have been considered but are moot in view of the new ground(s) of rejection.
Double Patenting
4. The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the "right to exclude" ranted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory obviousness-type double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Omum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); and In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on a nonstatutory double patenting ground provided the conflicting application or patent either is shown to be commonly owned with this application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement.
Effective January 1, 1994, a registered attorney or agent of record may sign a terminal disclaimer. A terminal disclaimer signed by the assignee must fully comply with 37 CFR 3.73(b).
5. Claims 2-21 are rejected on the ground of nonstatutory obviousness-type double patenting as being unpatentable over claims 1-18 of U.S. Patent No. 12,406,316. Although the conflicting claims are not identical, they are not patentably distinct from each other.
Instant Application 18892786
Patent US 12,406,316
Claim 2:
A method comprising, by a client system:
receiving, at the client system via an assistant application, an audio input comprising
speech of a user, wherein the speech comprises:
a co-reference to an entity associated with an attribute; and
a request to perform a task associated with the entity
receiving, at a camera of the client system via the assistant application, a visual input
comprising a real-time view of one or more subjects;
analyzing the visual input to identify-resolve, based at least in part on the attribute, the
entity from which of the one or more subjects in the visual input is the entity;
executing the task associated with the entity; and
presenting, at the client system via the assistant application, an output associated with the task associated with the entity, wherein the output comprises audio information.
Claim 1:
A method comprising, by a client system:
receiving, at the client system, a speech input from a user and a visual input captured by one or more cameras of the client system, wherein the speech input comprises a textual input spoken by the user, and wherein the visual input depicts a real-time view captured by the one or more cameras, wherein the real-time view comprises one or more visual concepts and one or more attributes associated with the one or more visual concepts, and wherein the textual input comprises a co-reference to one or more of the visual concepts;
resolving, based on the one or more attributes and the co-reference, one or more entities corresponding to the one or more visual concepts associated with the co-reference; and
presenting, at the client system, a communication content responsive to the speech input and the visual input, wherein the communication content comprises information associated with executing results of one or more tasks corresponding to the one or more resolved entities.
Claim 8:
A method of operating a client system, the method comprising:
receiving, from a microphone of the client system, an audio input comprising speech of a
user, wherein the speech comprises;
a co-reference to an entity associated with an attribute; and
a request to perform a task associated with the entity;
receiving, from a camera of the client system, a visual input comprising a real-time view
of one or more subjects;
performing a visual analysis of the visual input to identify-resolve, based at least in part
on the co-reference and the one or more attributes, an entity corresponding to a specific subject
which of the one or more subjects, in the visual input is the entity;
in response to a request from the user, executing a task associated with the entity; and
presenting, at the client system, an output associated with the task associated with the
entity, wherein the output comprises audio information.
Claim 17:
One or more computer-readable non-transitory non- volatile storage media embodying software that is operable when executed by a client system to:
receive, at a client system, a speech input from a user and a visual input captured by one or more cameras of the client system, wherein the speech input comprises a textual input spoken by the user, and wherein the visual input depicts a real-time view captured by the one or more cameras, wherein the real-time view comprises one or more visual concepts and one or more attributes associated with the one or more visual concepts, and wherein the textual input comprises a co-reference to one or more of the visual concepts;
resolve, based on the one or more attributes and the co-reference, one or more entities corresponding to the one or more visual concepts associated with the co-reference; and
present, at the client system, a communication content responsive to the speech input and the visual input, wherein the communication content comprises information associated with executing results of one or more tasks corresponding to the one or more resolved entities.
Claim 15:
A method of operating a client system, the method comprising:
receiving, from a microphone of the client system, an audio input comprising speech of a
user, wherein the speech comprises;
a co-reference to an entity associated with an attribute; and
a request to perform a task associated with the entity;
receiving, from a camera of the client system, a visual input comprising a real-time view
of one or more subjects;
processing the visual input to identify-resolve, based at least in part on the attribute, an
entity corresponding to a specific subject which of the one or more subjects, in the visual input is
the entity;
executing the task associated with the entity; and
presenting, at the client system, audio information associated with the task associated
with the entity.
Claim 18:
A client system comprising:
one or more processors; and a non-transitory non-volatile memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
receive, at a client system, a speech input from a user and a visual input captured by one or more cameras of the client system, wherein the speech input comprises a textual input spoken by the user, and wherein the visual input depicts a real-time view captured by the one or more cameras, wherein the real-time view comprises one or more visual concepts and one or more attributes associated with the one or more visual concepts, and wherein the textual input comprises a co-reference to one or more of the visual concepts;
resolve, based on the one or more attributes and the co-reference, one or more entities corresponding to the one or more visual concepts associated with the co-reference; and
present, at the client system, a communication content responsive to the speech input and the visual input, wherein the communication content comprises information associated with executing results of one or more tasks corresponding to the one or more resolved entities.
Examiner’s Note
6. A co-reference to an entity (According to Google): “A co-reference to an entity occurs when two or more different expressions in a text or conversation refer to the exact same real-world person, place, thing, or concept. “
Mohajer et al, US 10,418,032, [Mohajer: Column 2, lines 44-55 (“Alternative ways to approach the analysis and interpretation of natural language input include Parts-Of-Speech, pattern matching and statistical approaches, neural networks, and more techniques. A semantic parser, based on a semantic grammar, is able to reject a syntactically ill-formed input query; reject a meaningless query; recognize the structure of a well-formed, meaningful query;”, i.e., ‘an audio input comprising speech of a user’)] [Mohajer: Column 3, lines 10-20 (“additional steps are often needed after the parsing and interpretation of a query, and before its execution. One such step has been so identified as the co-reference resolution problem, which is generally concerned with finding that (say) a reference to a person (‘Mr. Smith’) points to the same entity as another (‘the man with the felt hat’). A number of approaches to co-reference resolution have been suggested in the literature in computational linguistics and discourse analysis. See Jurafsky and Martin, Speech and Language Processing 2nd Ed, Chapter 21, section 21.7 to 21.9 (2009)”, i.e., ‘a co-reference to an entity ..’)] [Mohajer: Column 4, lines 1-19 (“In the contemplated applications, systems may have tens or hundreds of thousands of users, and ideally will respond in real-time.”)] [Mohajer: Column 12, lines 37-50 (“User input 100 may be received through a user input device. For example, user input 100 may be received as a combination of speech received by a microphone device, text entered from a text entry device such as a physical keyboard, a virtual keyboard, an on-screen keyboard, button pushes, taps, swipes, shakes, any movement of a pointing device such as a mouse, text recognized from a camera, gestures or facial expressions captured by a camera, singing, humming, biometric signals, and any other forms of input that a user can produce. This disclosure focuses on using spoken or written natural language as user input 100, but it is understood that changing or adding modalities does not affect the methods disclosed in a significant way”)].
He et al, US 20190156210, [He: Paragraphs 6 and 44 (“the assistant system may assist a user to obtain information or services. The assistant system may enable the user to interact with it with multi-modal user input (such as voice, text, image, video) in stateful and multi-turn conversations to get assistance” or “send a video including speech to the assistant application”, i.e., ‘an audio input comprising speech of a user’)] [He: Paragraph 61 (“In particular embodiments, the output of the NLU module 220 may be sent to a co-reference module 315 to interpret references of the content objects associated with the user request.”, i.e., ‘a co-reference to an entity’)] [He: Paragraphs 43 and 46 (“The assistant system 140 may additionally assist the user to manage different tasks such as keeping track of events. In particular embodiments, the assistant system 140 may proactively execute pre-authorized tasks that are relevant to user interests and preferences based on the user profile”, i.e., ‘executing a task associated with the entity’)] [He: Paragraph 52 (“The TTS module 275 may convert the communication content to an audio clip. The TTS module 275 may further send the audio clip to the client system 130 via the assistant application 136.”, i.e., ‘wherein the output comprises audio information’)].
Claim Rejections - 35 USC § 102
7. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
8. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
9. Claims 2-5 and 7-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Sahay et al (US 20170078621).
Claim 2:
Sahay suggests a method comprising, by a client system: receiving, at the client system via an assistant application [Sahay: Paragraph 41 (“generation personal assistant system”)], an audio input comprising speech of a user [Sahay: Paragraphs 14 and 36 (“input/output devices (e.g., cameras, microphones, sensors (e.g., voice sensors, context sensors, ambient environment sensors, temperature/weather sensors, etc.),”)] [Sahay: Paragraph 33 (“Capturing/sensing components 261 may further include voice recognition devices, photo recognition devices, facial and other body recognition components, voice-to-text conversion components”)], wherein the speech comprises: a co-reference to an entity associated with an attribute [Sahay: Paragraphs 46 and 65 (“person entities from comments, descriptions, etc., while any pronouns and co-references may be resolved with a co-reference resolution natural language processing as facilitated by co-reference resolution logic (“co-reference logic”)”)]. Sahay suggests a request to perform a task associated with the entity [Sahay: Paragraphs 13-16 (“efficient personal assistant to help users with automatic and dynamic curation of multimedia items (also referenced as “media items”, “media assets” or simply “media”)”)]. Sahay suggests receiving, at a camera of the client system [Sahay: Paragraph 26 (“Personal device 280, like computing device 100, may also include any number and type of input elements 291 (e.g., 2D cameras, 3D cameras, microphones, sensors, detectors, etc.) and output elements”)] [Sahay: Paragraph 102 (“When a gesture or voice command is issued, the system can determine the appropriate screen for the gesture”)] a real-time view of one or more subjects [Sahay: Paragraph 102 (“When a gesture or voice command is issued, the system can determine the appropriate screen for the gesture”)] [Sahay: Paragraph 36 (“Similarly, other forms of media, such as text, pictures, etc., may be captured through one or more input elements 291, such as readers, scanners, cameras”)]. Sahay suggests analyzing the visual input to identify in the visual input is the entity [Sahay: Paragraph 102 (“When a gesture or voice command is issued, the system can determine the appropriate screen for the gesture”)] [Sahay: Paragraph 36 (“Similarly, other forms of media, such as text, pictures, etc., may be captured through one or more input elements 291, such as readers, scanners, cameras”)]. Sahay suggests executing the task associated with the entity [Sahay: Paragraph 39 (“these extracted contexts may then be communicated on to media story mechanism 110 for further processing, such as by emotion understanding engine 209, where these extracted contexts are communicated, in real-time, using communication logic 289 and/or communication/compatibility logic”)] [Sahay: Paragraph 43 (“Now referring to media story mechanism 110, in one embodiment, detection/reception logic 201 may be triggered and used each time user requests, contexts, real-time data, historical data, etc., may be received at detection/reception logic 201. Further, detection/reception logic 201 may be used to perform any number and type of detection and/or reception tasks relating to one or more of capturing/sensing components 261, output components 263, etc., to detect various contents from personal device 280, media source(s) 270, database(s) 265, etc., over communication medium”)]. Sahay suggests presenting, at the client system via the assistant application, an output associated with the task associated with the entity, wherein the output comprises audio information [Sahay: Paragraph 35 (“in one embodiment, output components 263 may include (without limitation) one or more of light sources, display devices and/or screens, audio speakers, tactile components, conductance elements, bone conducting speakers, olfactory or smell visual and/or non/visual presentation devices, haptic or touch visual and/or non-visual presentation devices, animation display devices, biometric display devices, X-ray display devices, high-resolution displays, high-dynamic range displays, multi-view displays, and head-mounted displays (HMDs) for at least one of virtual reality (VR) and augmented reality (AR), etc”)] [Sahay: Paragraph 38 (“Similarly, extraction logic 287 may be used to extract such contexts from any number and type of data/metadata, such as audio chatter (e.g., persons laughing, joking, being serious, crying, etc.), emoticons in text (e.g., smiley face, frown face, angry face, etc.), punctuations (e.g., exclamation points, periods, commas, etc.), visual indications in photos or videos (e.g., person smiling, jumping, resting, drinking, eyes wide open, teeth showing, wet clothes, sand on shoes, etc.), and/or the like”)].
Claim 3:
Sahay suggests wherein the output further comprises text information presented on a display of the client system [Sahay: Paragraphs 13 and 34 (“live chatter between users, voice messages, text messages, comments posted on websites by users, feedback received from users,”)].
Claim 4:
Sahay suggests wherein the task associated with the entity is executed by an assistant system in communication with the assistant application [Sahay: Paragraphs 13 and 41 (“serving as an intelligent and efficient personal assistant to help users with automatic and dynamic curation of multimedia items”)].
Claim 5:
Sahay suggests wherein the task associated with the entity comprises retrieving information about the entity from a service [Sahay: Paragraphs 17 and 22 (“dynamic media curation that takes into account and accordingly extracts any number and type of user-related factors, such as goals, themes, emotions, preferences, beliefs, etc., from any number and type of media sources,”)].
Claim 7:
Sahay suggests wherein the one or more subjects comprise a plurality of persons, the co-reference identifies a particular person of the plurality of persons, and the entity corresponds to the particular person [Sahay: Paragraphs 46 and 65 (“person entities from comments, descriptions, etc., while any pronouns and co-references may be resolved with a co-reference resolution natural language processing as facilitated by co-reference resolution logic (“co-reference logic”)”)].
Claim 8:
Claim 8 is essentially the same as claim 2 and rejected under the same reasons as applied above.
Claim 9:
Claim 9 is essentially the same as claim 3 and rejected under the same reasons as applied above.
Claim 10:
Sahay suggests wherein the speech comprises the request from the user [Sahay: Paragraphs 14 and 36 (“input/output devices (e.g., cameras, microphones, sensors (e.g., voice sensors, context sensors, ambient environment sensors, temperature/weather sensors, etc.),”)].
Claim 11:
Claim 11 is essentially the same as claim 4 and rejected under the same reasons as applied above.
Claim 12:
Claim 12 is essentially the same as claim 5 and rejected under the same reasons as applied above.
Claim 13:
Sahay suggests wherein the entity is resolved based at least in part on an association between specific attributes of the specific subject and the co-reference [Sahay: Paragraphs 46 and 65 (“person entities from comments, descriptions, etc., while any pronouns and co-references may be resolved with a co-reference resolution natural language processing as facilitated by co-reference resolution logic (“co-reference logic”)”)].
Claim 14:
Sahay suggests wherein the one or more subjects comprise a plurality of persons, the co-reference identifies a particular person of the plurality of persons, and the entity corresponds to the particular person [Sahay: Paragraphs 46 and 65 (“person entities from comments, descriptions, etc., while any pronouns and co-references may be resolved with a co-reference resolution natural language processing as facilitated by co-reference resolution logic (“co-reference logic”)”)].
Claim 15:
Claim 15 is essentially the same as claim 2 and rejected under the same reasons as applied above.
Claim 16:
Claim 16 is essentially the same as claim 5 and rejected under the same reasons as applied above.
Claim 17:
Sahay suggests wherein the information is presented via a visual modality [Sahay: Paragraphs 14 and 36 (“input/output devices (e.g., cameras, microphones, sensors (e.g., voice sensors, context sensors, ambient environment sensors, temperature/weather sensors, etc.),”)] [Sahay: Paragraph 33 (“Capturing/sensing components 261 may further include voice recognition devices, photo recognition devices, facial and other body recognition components, voice-to-text conversion components”)].
Claim 18:
Sahay suggests wherein the information is presented via a audio modality [Sahay: Paragraphs 14 and 36 (“input/output devices (e.g., cameras, microphones, sensors (e.g., voice sensors, context sensors, ambient environment sensors, temperature/weather sensors, etc.),”)] [Sahay: Paragraph 33 (“Capturing/sensing components 261 may further include voice recognition devices, photo recognition devices, facial and other body recognition components, voice-to-text conversion components”)].
Claim 19:
Sahay suggests wherein the entity is resolved based at least in part on an attribute associated with the specific subject [Sahay: Paragraphs 14 and 36 (“input/output devices (e.g., cameras, microphones, sensors (e.g., voice sensors, context sensors, ambient environment sensors, temperature/weather sensors, etc.),”)] [Sahay: Paragraph 33 (“Capturing/sensing components 261 may further include voice recognition devices, photo recognition devices, facial and other body recognition components, voice-to-text conversion components”)].
Claim 20:
Sahay suggests wherein the one or more subjects comprise a plurality of persons, the co-reference identifies a particular person of the plurality of persons, and the entity corresponds to the particular person [Sahay: Paragraphs 46 and 65 (“person entities from comments, descriptions, etc., while any pronouns and co-references may be resolved with a co-reference resolution natural language processing as facilitated by co-reference resolution logic (“co-reference logic”)”)].
Claim Rejections - 35 USC § 103
10. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
11. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
12. Claims 6 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Sahay et al (US 20170078621), in view of Michels et al (US 20170147582).
Claim 6:
The combined teachings of Sahay and Michels suggest checking an authorization setting associated with the entity before executing the task associated with the entity [Michels: Paragraph 25 (“The database query engine 202, before parsing, converting, and executing a received database query, may issue an authorization check to the authorization module 310 in order to confirm that the user is authorized to perform, for example, the requested search and ranking tasks. The authorization module 310, upon checking whether a user is authorized to request a particular action, issues authorization messages to other entities, such as the database query engine 20,”)].
Both references (Sahay and Michels) taught features that were directed to analogous art and they were directed to the same field of endeavor, such as data processing. It would have been obvious to one of ordinary skill in the art at the time the invention was made, having the teachings of Sahay and Michels before him/her, to modify the system of Sahay with the teaching of Michels in order to check authorization before executing tasks [Michels: Paragraph 25].
Claim 21:
Claim 21 is essentially the same as claim 6 and rejected under the same reasons as applied above.
13. Any inquiry concerning this communication or earlier communications from the examiner should be directed to [Hung D. Le], whose telephone number is [571-270-1404]. The examiner can normally be communicated on [Monday to Friday: 9:00 A.M. to 5:00 P.M.].
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Apu Mofiz can be reached on [571-272-4080]. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, contact [800-786-9199 (IN USA OR CANADA) or 571-272-1000].
Hung Le
05/19/2026
/HUNG D LE/Primary Examiner, Art Unit 2161
Prosecution Insights