DETAILED ACTION
In response to communications filed 30 September 2025, this is the first Office action on the merits. Claims 1-20 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Claims 1, 10, and 19 recite
determining a reply corresponding to the user query based on the acquired auxiliary visual information and the user query.
These limitations, under their broadest reasonable interpretation, fall within the Mental Processes grouping of abstract ideas. The step of “determining” can be performed as mental observations and/or evaluations, perhaps using pen and paper. Accordingly, the claims recite an abstract idea.
This judicial exception is not integrated into a practical application. Claims 1, 10, and 19 further recite
obtaining, in response to receiving a user query from a user to a digital assistant, auxiliary visual information acquired by a visual acquisition tool associated with the user based on the user query, an acquisition time of the auxiliary visual information being adjacent to an initiation time of the user query.
However, these additional elements do not integrate the judicial exception into a practical application, because they amount to an insignificant extra-solution activity, such as mere data gathering (“obtaining”). See MPEP § 2106.05(g).
Claim 1, 10, and 19 further recite that the step of determining is performed “with a machine learning model.” Claim 10 further recites
An electronic device comprising: at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, wherein the instructions, when executed by the at least one processor, cause the electronic device to perform operations comprising [the steps analyzed above].
Claim 19 further recites
A non-transitory computer-readable storage medium having stored thereon a computer program executable by a processor to perform operations comprising [the steps analyzed above].
However, these additional elements do not integrate the judicial exception into a practical application, because they amount to merely implementing the judicial exception on a generic computer system. See MPEP § 2106.05(f).
Claims 2, 11, and 20 further recite “obtaining . . . the auxiliary visual information acquired by the visual acquisition tool.” Similarly, claims 6 and 15 further recite “obtaining visual information . . . by the visual acquisition tool.” However, these additional elements do not integrate the judicial exception into a practical application, because they amount to an insignificant extra-solution activity, such as mere data gathering (“obtaining”). See MPEP § 2106.05(g).
Claims 4-5 and 13-14 further recite to “acquire first visual information at a first frequency during the reception of the user query” and to “acquire second visual information at a second frequency, the second frequency being less than the first frequency,” respectively. However, these additional elements do not integrate the judicial exception into a practical application, because they amount to an insignificant extra-solution activity, such as mere data gathering (“acquire”). See MPEP § 2106.05(g).
Claims 8 and 17 further recite a “client”; that the “machine learning model is deployed at a server”; “sending . . . to the server . . . via a first communication link”; “sending, via a second communication link . . . to the server to request the server to determine, with the machine learning model, the reply”; and “receiving the reply . . . from the server.” However, these additional elements do not integrate the judicial exception into a practical application, because they amount to merely implementing the judicial exception on a generic computer system. See MPEP § 2106.05(f).
The additional limitations of claims 2-9, 11-18, and 20, not explicitly addressed above, likewise fall within the Mental Processes grouping of abstract ideas. Considering the limitations as an ordered combination adds nothing that is not already present when considering the elements individually. There is no indication that the combination of elements improves the functioning of a computer or improves any other technology.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements amount to no more than insignificant extra-solution activities and/or implementing the judicial exception on a generic computer system. In addition, paragraphs [0003] and [0039] of the specification provides evidence that the step of “obtaining” and to “acquire” were well-understood, routine, and conventional activities of “digital assistant functions” and a “client device.” See also MPEP § 2106.05(d)(II). Therefore, claims 1-20 are not patent eligible.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-3, 8-12, and 17-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Diamant et al. (US 2019/0027147 A1).
Regarding claim 1, Diamant teaches a method for information processing, comprising:
obtaining, in response to receiving a user query from a user to a digital assistant, auxiliary visual information acquired by a visual acquisition tool associated with the user based on the user query, an acquisition time of the auxiliary visual information being adjacent to an initiation time of the user query (see Diamant [0040]-[0042], an “image 136 is captured” responsive to “user 102 provid[ing] a spoken utterance,” where the “camera” is a visual acquisition tool and obtaining the “image” after the “spoken utterance” teaches an acquisition time adjacent to an initiation time of the user query); and
determining, with a machine learning model, a reply corresponding to the user query based on the acquired auxiliary visual information and the user query (see Diamant [0046] and [0032]-[0033], “requested information is provided to the user,” where [0025] teaches that the determining is performed with “machine learning models”).
Regarding claim 10, Diamant teaches an electronic device comprising:
at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, wherein the instructions, when executed by the at least one processor (see Diamant [0051]),
cause the electronic device to perform operations comprising:
obtaining, in response to receiving a user query from a user to a digital assistant, auxiliary visual information acquired by a visual acquisition tool associated with the user based on the user query, an acquisition time of the auxiliary visual information being adjacent to an initiation time of the user query (see Diamant [0040]-[0042], an “image 136 is captured” responsive to “user 102 provid[ing] a spoken utterance,” where the “camera” is a visual acquisition tool and obtaining the “image” after the “spoken utterance” teaches an acquisition time adjacent to an initiation time of the user query); and
determining, with a machine learning model, a reply corresponding to the user query based on the acquired auxiliary visual information and the user query (see Diamant [0046] and [0032]-[0033], “requested information is provided to the user,” where [0025] teaches that the determining is performed with “machine learning models”).
Regarding claim 19, Diamant teaches a non-transitory computer-readable storage medium having stored thereon a computer program executable by a processor (see Diamant [0051] and [0055]) to
perform operations comprising:
obtaining, in response to receiving a user query from a user to a digital assistant, auxiliary visual information acquired by a visual acquisition tool associated with the user based on the user query, an acquisition time of the auxiliary visual information being adjacent to an initiation time of the user query (see Diamant [0040]-[0042], an “image 136 is captured” responsive to “user 102 provid[ing] a spoken utterance,” where the “camera” is a visual acquisition tool and obtaining the “image” after the “spoken utterance” teaches an acquisition time adjacent to an initiation time of the user query); and
determining, with a machine learning model, a reply corresponding to the user query based on the acquired auxiliary visual information and the user query (see Diamant [0046] and [0032]-[0033], “requested information is provided to the user,” where [0025] teaches that the determining is performed with “machine learning models”).
Regarding claims 2, 11, and 20, Diamant teaches wherein obtaining the auxiliary visual information acquired by the visual acquisition tool associated with the user based on the user query comprises:
detecting a requirement for the auxiliary visual information based on the user query (see Diamant [0030] and [0041], “indication of a trigger 134 in the utterance”); and
obtaining, in response to detecting the requirement , the auxiliary visual information acquired by the visual acquisition tool based on the user query (see Diamant [0042], “responsive to receiving the indication of the trigger 134, the camera . . . is activated”).
Regarding claims 3 and 12, Diamant teaches wherein detecting the requirement for the auxiliary visual information based on the user query comprises: determining that the requirement for the auxiliary visual information is detected in response to determining that the user query is related to a target object in an environment where the user is located (see Diamant [0030]-[0032], “identify that a person is holding an object of interest 202 or is pointing to an object of interest, which can be using as a signal to increase confidence that the object of interest 202 is within the camera frame”).
Regarding claims 8 and 17, Diamant teaches wherein the method is applied to a client, and the machine learning model is deployed at a server (see Fig. 1B, “Client Computing Device 104” and “Server Devices(s) 128”), and
wherein determining, with the machine learning model, the reply corresponding to the user query based on the acquired auxiliary visual information and the user query comprises:
sending, in response to detecting the user query, the auxiliary visual information to the server via a first communication link (see Diamant [0042] and [0034], “captured image . . . sent in separate transactions”);
sending, via a second communication link, the received user query to the server to request the server to determine, with the machine learning model, the reply based on the acquired auxiliary visual information and the user query (see [0046] and [0034], “utterance . . . sent in separate transactions”); and
receiving the reply corresponding to the user query from the server (see Diamant [0028], “passes the task or service request to the digital assistant in a response 132”).
Regarding claims 9 and 18, Diamant teaches further comprising: determining a type of the visual acquisition tool based on the user query, the type indicating a camera or a screen recording tool of a client (see Diamant [0042] and [0024], “determination is made as to which camera is relevant for the given interaction”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 4 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Diamant et al. (US 2019/0027147 A1) as applied to claims 1 and 10 above, and further in view of Claire (US 2023/0267299 A1).
Regarding claims 4 and 13, Diamant teaches wherein the visual acquisition tool is maintained in a visual information acquisition state during an interaction between the user and the digital assistant (see Diamant [0042], “camera . . . is activated”).
Diamant does not explicitly teach wherein obtaining the auxiliary visual information acquired by the visual acquisition tool comprises: instructing the visual acquisition tool to acquire first visual information at a first frequency during the reception of the user query.
However, Claire teaches wherein obtaining the auxiliary visual information acquired by the visual acquisition tool comprises: instructing the visual acquisition tool to acquire first visual information at a first frequency during the reception of the user query (see Claire [0095], “environment may be visually sensed by capturing a video recording using a camera,” where the frame rate of the “video recording” is a first frequency).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention acquire first visual information at a first frequency, as taught by Claire, in combination with the techniques taught by Diamant, because the “environment may be visually sensed by capturing a video recording using a camera . . . or sensed in any other way possible based on the capabilities of the available sensors” (see Claire [0090).
Diamant as modified teaches determining the first visual information as at least a portion of the auxiliary visual information (see Diamant [0040]-[0042] and Claire [0095], where at least a portion of the auxiliary visual information, taught by Diamant, is the “video recording” taught by Claire).
Claims 5 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Diamant et al. (US 2019/0027147 A1) in view of Claire (US 2023/0267299 A1) as applied to claims 1, 4, 10, and 13 above, and further in view of Park et al. (US 2024/0419830 A1).
Regarding claims 5 and 14, Diamant as modified teaches obtaining the auxiliary visual information acquired by the visual acquisition tool further comprises: instructing, in response to not receiving the user query, the visual acquisition tool to acquire (see Claire [0062], “camera 110 may always be active”).
Diamant does not explicitly teach the visual acquisition tool to acquire second visual information at a second frequency, the second frequency being less than the first frequency.
However, Park teaches the visual acquisition tool to acquire second visual information at a second frequency, the second frequency being less than the first frequency (see Park [0060], “’always-on’ cameras to monitor the external environment . . . low frame rates”).
Diamant as modified teaches that the “environment may be visually sensed by capturing a video recording using a camera” and that the “camera 110 may always be active”; see Claire [0090] and [0062], respectively. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to acquire second visual information at a second frequency, the second frequency being less than the first frequency, as taught by Park, in combination with the techniques taught by Diamant as modified, because “low frame rates . . . save power” (see Park [0060]).
Diamant as modified teaches determining the second visual information as at least a portion of the auxiliary visual information (see Diamant [0040]-[0042] and Park [0060], where at least a portion of the auxiliary visual information, taught by Diamant, is the second visual information from an “always-on” camera, as taught by Park).
Claims 6-7 and 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over Diamant et al. (US 2019/0027147 A1) as applied to claims 1-2 and 10-11 above, and further in view of Ryu (US 2019/0043120 A1).
Regarding claims 6 and 15, Diamant teaches wherein obtaining the auxiliary visual information acquired by the visual acquisition tool associated with the user comprises: instructing, in response to detecting the requirement, the visual acquisition tool to switch to a visual information acquisition state (see Diamant [0042], “camera . . . is activated”).
Diamant does not explicitly teach obtaining visual information for a predetermined time period acquired by the visual acquisition tool in the visual information acquisition state as the auxiliary visual information.
However, Ryu teaches obtaining visual information for a predetermined time period acquired by the visual acquisition tool in the visual information acquisition state as the auxiliary visual information (see Ryu [0035], “activates the camera of the mobile device for a limited period of time”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to obtain visual information for a predetermined time period, as taught by Ryu, in combination with the techniques taught by Diamant, because “Various embodiments may set a time threshold for video recording, which when exceeded or expired the video recording session is terminated. An embodiment may delete any stored video recording . . . once the respective session is terminated, which is aimed to protect the privacy of the transaction made between the users” (see Ryu [0034]).
Regarding claims 7 and 16, Diamant as modified teaches further comprising: instructing, upon expiration of the predetermined time period, the visual acquisition tool to exit the visual information acquisition state (see Ryu [0035], “once such period of time expires, disables the camera of said mobile device”).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Adya et al. (US 2023/0199297 A1) teaches “determining, based on the semantic analysis, a likelihood that the electronic device requires additional contextual data to satisfy the request, and . . . enabling a camera of the electronic device and determining a response to the request based on data captured by the camera of the electronic device” (see Adya, Abstract).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Kristopher Andersen whose telephone number is (571)270-5743. The examiner can normally be reached 8:30 AM-5:00 PM ET, Monday-Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ann Lo can be reached at (571) 272-9767. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Kristopher Andersen/Primary Examiner, Art Unit 2159
Prosecution Insights