DETAILED ACTION
Response to Amendment
Applicant’s preliminary submission filed on 6/7/2021 has been entered. Claims 1-20 remain pending in this application.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The lengthy specification has not been checked to the extent necessary to determine the presence of all possible minor errors. Applicant’s cooperation is requested in correcting any errors of which applicant may become aware in the specification.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-3, 11-13 of parent U.S. Patent No. 12,126,700. Although the claims at issue are not identical, they are not patentably distinct from each other because the claims of the present application are obvious in view of the claims of the U.S. Patent, because every claim limitation found within the present application is nothing more than a reworded version of a claim limitation found within the U.S. Patent, with some claim limitations being moved around between claims.
For example, claim 1 of the present application is mostly a broader version of claims 1-2 of the U.S Patent, where “a machine-learned model” recited in claim 1 of the U.S. Patent is broadened into “an artificial intelligence assistant system” in the present application, “contextual input data” comprising “information displayed in a user interface” recited in claim 2 of the U.S. Patent is broadened into “visual data” in the present application, and several steps in the U.S. Patent claims are missing in the present application claim because they have been moved to dependent claims within the present application; see claim mapping below.
Present Application 18/896,298
U.S. Patent No. 12,126,700
1. A mobile computing device, comprising:
at least one processor; a plurality of computing applications; an Artificial Intelligence (AI) assistant system; and at least one tangible, non-transitory computer-readable medium that stores instructions that, when executed by the at least one processor, cause the at least one processor to perform operations, the operations comprising: proactively performing, without receipt of a user input requesting performance: receiving an application output comprising visual data from a first computing application of the plurality of computing applications; processing the visual data with the AI assistant system to obtain textual content associated with the application output; and generating an output based on the textual content associated with the application output.
1. A mobile computing device, comprising:
at least one processor; a plurality of computing applications; and at least one tangible, non-transitory computer-readable medium that stores instructions that, when executed by the at least one processor, cause the at least one processor to perform operations, the operations comprising: proactively performing, without receipt of a user input requesting performance:
inputting contextual input data into a machine-learned model; receiving, as an output of the machine-learned model, a model output that describes one or more semantic entities referenced by the contextual input data; selecting a computing application of the plurality of computing applications based at least in part on the model output; providing data descriptive of the one or more semantic entities to the computing application; and receiving an application output from the computing application of the plurality of computing applications in response to providing the data descriptive of the one or more semantic entities to the computing application, wherein the application output received from the computing application describes an available action of the corresponding computing application with respect to the one or more semantic entities; and
performing the available action of the computing application, wherein performing the available action causes display of at least one indicator associated with the available action via a display device associated with the mobile computing device.
2. The mobile computing device of claim 1, wherein the contextual input data comprises at least one of information displayed in a user interface, audio played by the mobile computing device, or ambient audio detected by the mobile computing device.
Similarly, the limitations of claim 2 of the present application map to limitations found in claim 2 of the U.S. Patent, the limitations of claim 3 of the present application map to limitations found in claim 1 of the U.S. Patent, the limitations of claim 4 of the present application map to limitations found in claim 1 of the U.S. Patent, the limitations of claim 5 of the present application map to limitations found in claim 1 of the U.S. Patent, the limitations of claim 6 of the present application map to limitations found in claim 1 of the U.S. Patent, the limitations of claim 7 of the present application map to limitations found in claim 2 of the U.S. Patent, and the limitations of claim 8 of the present application map to limitations found in claim 3 of the U.S. Patent.
Regarding claims 9-16, the instant claims are directed towards a substantially similar invention as recited in claim 1-8, and are thus rejected for substantially similar reasons.
Regarding claims 17-20, the instant claims are directed towards a substantially similar invention as recited in claim 1-4, and are thus rejected for substantially similar reasons.
Similarly, claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over the claims of parent U.S. Patents No. 11,553,063 and 11,831,738; the claims of parent U.S. Patent No. 12,126,700 were previously found to be obvious over the claims of parent U.S. Patents No. 11,553,063 and 11,831,738 and the NSDP rejection was overcome with a Terminal Disclaimer; see prosecution history of U.S. Patent No. 12,126,700 (U.S. Application No. 18/493,509).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Natarajan et al. (US 2019/0327330, hereinafter Natarajan) in view of Aggarwal et al. (US 2016/0350304, hereinafter Aggarwal).
Regarding claim 1, Natarajan discloses a mobile computing device (a client system 130 may include a… cellular telephone – Natarajan ¶0026; computer system 1000 may be… a mobile telephone – Natarajan ¶0104), comprising:
at least one processor (1000 includes a processor 1002 – Natarajan ¶0105);
a plurality of computing applications (130 may include a social-networking application 134 installed on the client system 130 – Natarajan ¶0028; 130 may include an assistant application 136 – Natarajan ¶0029);
an Artificial Intelligence (AI) assistant system (a user at a client system 130 may use the assistant application 136 to interact with the assistant system 140 – Natarajan ¶0029; 140 may use natural-language understanding to analyze the user request – Natarajan ¶0030; “natural-language understanding” is a type of artificial intelligence); and
at least one tangible, non-transitory computer-readable medium that stores instructions that, when executed by the at least one processor, cause the at least one processor to perform operations (1000 includes… memory 1004 – Natarajan ¶0105; 1004 includes main memory for storing instructions for processor 1002 to execute – Natarajan ¶0107), the operations comprising:
proactively performing, without receipt of a user input requesting performance (the assistant xbot 215 may interact with a proactive inference layer 280 without receiving a user input… 280 may further communicate with proactive agents 285 regarding the inference. The proactive agents 285 may execute proactive tasks based on the inference – Natarajan ¶0049):
receiving an application output from a first computing application of the plurality of computing applications (for example, a user’s brother Aaron posted a video on his news feed – Natarajan ¶0070);
processing with the AI assistant system to obtain textual content associated with the application output (FIG. 2 illustrates an example architecture of the assistant system 140 – Natarajan ¶0039; FIG. 2 shows CU composer 270; the personalized machine-learning model, in combination with other data, may be used to optimize the previously built customized user profile, thus resulting in an even better customized user profile… the customized user profile of the user may indicate that the user always chats with his brother John using a nickname (e.g. “Boy”). As a result, the CU composer 270 may modify the generation of the communication content to be more personalized as “Boy posted a video” – Natarajan ¶0070); and
generating an output based on the textual content associated with the application output (140 may according generate the result and send it back to the assistant application 136. The assistance application 136 may further present the result to the user in text – Natarajan ¶0029).
Natarajan does not disclose receiving an application output comprising visual data from a first computing application of the plurality of computing applications, then processing the visual data with the AI assistant system to obtain textual content associated with the application output as claimed. Aggarwal, however, in the same field of endeavor, teaches receiving an application output comprising visual data from a first computing application of the plurality of computing applications, then processing the visual data with the AI assistant system to obtain textual content associated with the application output (a suggested voice-based action query includes an action term and an entity term that are particularized to an entity of content recently viewed… content currently being viewed on the computing device – Aggarwal ¶0006; for example, for content about the movie Blade Runner, the text “Blade Runner” may be identified as the dominant entity based on it appearing in larger font than other text, in a more prominent position than other text, and/or more frequently in other text – Aggarwal ¶0057).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system taught by Natarajan to comprise the receiving and processing visual data limitations as claimed, in order to further execute tasks that are relevant to user interests (Natarajan ¶0006) by controlling whether and/or how to receive content that may be more relevant to the user (Aggarwal ¶0115).
Regarding claim 2, Natarajan and Aggarwal teach wherein generating an output based on the textual content associated with the application output comprises: displaying a graphical indicator comprising the textual content associated with the application output (Render/synchronization module 125 manages the presenting of suggested voice-based action queries to the user, e.g., via a visual display – Aggarwal ¶0047).
Regarding claim 3, Natarajan and Aggarwal teach wherein generating an output based on the textual content associated with the application output comprises: providing the textual content associated with the application output to the first computing application (125 also handles synchronization with other online services, such as when a response or action affects data maintained for the user in another online service – Aggarwal ¶0047).
Regarding claim 4, Natarajan and Aggarwal teach wherein, prior to receiving the application output comprising the visual data, the operations comprise: selecting the first computing application of the plurality of computing applications (an assistant system 140 may assist users to retrieve information from difference sources… 140 may receive a user request for information or services via the assistant application 136 – Natarajan ¶0030; Examiner notes that the claim limitation “prior to receiving the application output…” does not specify that said limitation occur while the claimed invention is “proactively performing, without receipt of a user input requesting performance”, in contrast to the parent U.S. Patents listed above. Examiner is thus broadly interpreting “selecting the first computing application of the plurality of computing applications” as occurring BEFORE “proactively performing” for the purposes of examination).
Regarding claim 5, Natarajan and Aggarwal teach wherein selecting the first computing application of the plurality of computing applications comprises: selecting the first computing application of the plurality of computing applications based on data descriptive of one or more semantic entities (the indication of the content may include text and properties of the text in the content and entity determination module 142 may determine a dominant entity associated with the content based on position, format, frequency, and/or other property of the text in the content – Aggarwal ¶0057).
Regarding claim 6, Natarajan and Aggarwal teach wherein selecting the first computing application of the plurality of computing applications based on the data descriptive of the one or more semantic entities comprises: identifying the one or more semantic entities based on contextual data obtained by the mobile computing device (for example, a screenshot and/or text from FIG. 7A may have been provided as the indication of content and the suggested voice-based action queries 785A and 785B received in response. It is noted that suggested voice-based action query 785A is provided with a suggested time of “7 PM” for making a reservation. The suggested time may be determined based on the context of FIG. 7A (“dinner”), past user reservation history, and/or arbitrarily to provide the user an indication that spoken input can be utilized to make a reservation for “Up and Down Burger Bar” at a desired time – Aggarwal ¶0106).
Regarding claim 7, Natarajan and Aggarwal teach wherein the contextual data comprises at least one of information displayed in a user interface (a screenshot and/or text from FIG. 7A may have been provided as the indication of content – Aggarwal ¶0106; Aggarwal FIG. 7A depicts a user interface; as the claim limitations are recited as a list of alternatives, the prior art only needs to disclose one of the listed alternatives, where at least “information displayed in a user interface” is disclosed), audio played by the mobile computing device, or ambient audio detected by the mobile computing device.
Regarding claim 8, Natarajan and Aggarwal teach wherein the contextual data comprises at least one of calendar data or a location of the mobile computing device (the grammar model(s) and/or other models relied upon by semantic processor module 127 may incorporate various rules to initiate performance of a computer-based action based on text input provided by voice to text module 126... it will be appreciated that some parameters may be directly received as voice input, while some parameters may be determined in other manners, e.g., based upon an indication of content most recently viewed on the computing device, a geographic location of the computing device, etc. – Aggarwal ¶0052; as the claim limitations are recited as a list of alternatives, the prior art only needs to disclose one of the listed alternatives, where at least “a location of the mobile computing device” is disclosed).
Regarding claim 9, the instant claim is directed towards a substantially similar invention as recited in claim 1, and is thus rejected for substantially similar reasons.
Regarding claim 10, the instant claim is directed towards a substantially similar invention as recited in claim 2, and is thus rejected for substantially similar reasons.
Regarding claim 11, the instant claim is directed towards a substantially similar invention as recited in claim 3, and is thus rejected for substantially similar reasons.
Regarding claim 12, the instant claim is directed towards a substantially similar invention as recited in claim 4, and is thus rejected for substantially similar reasons.
Regarding claim 13, the instant claim is directed towards a substantially similar invention as recited in claim 5, and is thus rejected for substantially similar reasons.
Regarding claim 14, the instant claim is directed towards a substantially similar invention as recited in claim 6, and is thus rejected for substantially similar reasons.
Regarding claim 15, the instant claim is directed towards a substantially similar invention as recited in claim 7, and is thus rejected for substantially similar reasons.
Regarding claim 16, the instant claim is directed towards a substantially similar invention as recited in claim 8, and is thus rejected for substantially similar reasons.
Regarding claim 17, the instant claim is directed towards a substantially similar invention as recited in claim 1, and is thus rejected for substantially similar reasons.
Regarding claim 18, the instant claim is directed towards a substantially similar invention as recited in claim 2, and is thus rejected for substantially similar reasons.
Regarding claim 19, the instant claim is directed towards a substantially similar invention as recited in claim 3, and is thus rejected for substantially similar reasons.
Regarding claim 20, the instant claim is directed towards a substantially similar invention as recited in claim 4, and is thus rejected for substantially similar reasons.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LEON Y TSENG whose telephone number is (571)270-3682. The examiner can normally be reached Monday to Friday 8:30 AM to 5:00 PM MST, with every other Friday off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, WING CHAN can be reached at 571-272-7493. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LEON Y TSENG/Examiner, Art Unit 2441
/JOHN A FOLLANSBEE/Supervisory Patent Examiner, Art Unit 2444
Prosecution Insights