DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Priority is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) was filed on 12/23/2024. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Specification
The lengthy specification has not been checked to the extent necessary to determine the presence of all possible minor errors. Applicant’s cooperation is requested in correcting any errors of which applicant may become aware in the specification.
Drawings
The drawings filed 8/2/2024 were accepted.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 USC 101.
Claim 1 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claims do not fall within at least one of the four categories of patent eligible subject matter because they are directed to an abstract idea without significantly more. The claims recite the abstract idea of determining (determining that the automated assistant is not configured to fulfill the first request… determining that the first request includes one or more terms and a first parameter value of a parameter type), identifying (identifying one or more previous routines), and generating a routine or pattern (generating a general routine based on the one or more previous routines, wherein executing the general routine with the first parameter value results in the automated assistant performing the first automated routine; generating a request pattern based on a corresponding user input request of at least one of the previous routines, wherein the request pattern includes one or more of the terms and the parameter type).
Step 2A, Prong 1
The limitations that describe the determining, identifying, and generating a routine or pattern are processes that, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. The claims also include elements of receiving a request from a user and storing (storing the general routine), however nothing in the claims precludes the steps from practically being performed in the mind.
Step 2A, Prong 2
The judicial exception is not integrated into a practical application because the additional elements regarding receiving a request from a user and storing are considered insignificant extra-solution activity. These limitations are not considered improvements to the functioning of a technology or technical field. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the extrasolutionary elements are not considered significantly more than just applying the steps of determining, identifying, and generating a routine or pattern.
Step 2B
In addition to the abstract idea, the claims have the receiving a request from a user and storing, but they represent only well-understood, routine, conventional activity that can be performed on generic computers. The receiving of a request (i.e. user input) is considered mere instructions to apply an exception using generic computer components. Mere instructions to apply an exception using generic computer components cannot provide an inventive concept. The storing of data has been recognized by the courts as being well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity. See Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93 and MPEP 2106.05(d), subsection II. The claims are not patent eligible.
Claim 2 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. As per claim 2, this claim recites an additional element of executing a routine. It also recites similar receiving and determining steps as claim 1, and they are treated similarly to the rejection to claim 1 above.
(Step 2A, prong 2) The judicial exception is not integrated into a practical application because the additional elements regarding executing a routine are considered insignificant extra-solution activity. These limitations are not considered improvements to the functioning of a technology or technical field.
(Step 2B) The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the executing a routine are not considered significantly more than the judicial exception. The executing a routine is considered mere instructions to apply an exception using generic computer components. Mere instructions to apply an exception using generic computer components cannot provide an inventive concept. The additional elements represent only well-understood, routine, conventional activity that can be performed on generic computer systems. The claims are not patent eligible.
Claim 3 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. As per claim 3, this claim recites an additional abstract idea of identifying user action(s), and an additional element of transmitting a notification.
(Step 2A, prong 1) The limitations that describe the identifying user action(s) are processes that, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. The claims also include elements of transmitting a notification, however nothing in the claims precludes the steps from practically being performed in the mind.
(Step 2A, prong 2) The judicial exception is not integrated into a practical application because the additional elements regarding transmitting a notification are considered insignificant extra-solution activity. These limitations are not considered improvements to the functioning of a technology or technical field.
(Step 2B) The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the receiving a configuration parameter are not considered significantly more than the judicial exception. The additional elements represent only well-understood, routine, conventional activity that can be performed on generic computer systems. The transmitting of data (transmitting a notification) has been recognized by the courts as being well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity. See Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network), and MPEP 2106.05(d), subsection II. The claims are not patent eligible.
Claim 4 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. As per claim 4, this claim recites an additional abstract idea of processing the screenshots to identify the actions, and an additional element of receiving screenshots.
(Step 2A, prong 1) The limitations that describe the processing the screenshots to identify the actions are processes that, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. The claims also include elements of receiving screenshots, however nothing in the claims precludes the steps from practically being performed in the mind.
(Step 2A, prong 2) The judicial exception is not integrated into a practical application because the additional elements regarding receiving screenshots are considered insignificant extra-solution activity. These limitations are not considered improvements to the functioning of a technology or technical field.
(Step 2B) The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the receiving screenshots is not considered significantly more than the judicial exception. The additional elements represent only well-understood, routine, conventional activity that can be performed on generic computer systems. The receiving of data has been recognized by the courts as being well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity. See buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) and MPEP 2106.05(d), subsection II. The claims are not patent eligible.
Claim 5 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. As per claim 5, this claim recites an additional abstract idea of processing the indications to determine actions, and additional elements of providing screenshots, a trained machine learning model, and receiving indications.
(Step 2A, prong 1) The limitations that describe the processing the indications to determine actions are processes that, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. The claims also include elements of providing screenshots, a trained machine learning model, and receiving indications, however nothing in the claims precludes the steps from practically being performed in the mind.
(Step 2A, prong 2) The judicial exception is not integrated into a practical application because the additional elements regarding providing screenshots, a trained machine learning model, and receiving indications are considered insignificant extra-solution activity. These limitations are not considered improvements to the functioning of a technology or technical field.
(Step 2B) The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the providing screenshots, a trained machine learning model, and receiving indications is not considered significantly more than the judicial exception. The additional elements represent only well-understood, routine, conventional activity that can be performed on generic computer systems. The sending and receiving of data has been recognized by the courts as being well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity. See buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) and MPEP 2106.05(d), subsection II. Zhao et al (US20190318202A1; filed 4/12/2019) discloses how well-understood, routine, and conventional the machine learning models are: paragraph 3: “Machine Learning (ML) is a technology involving multiple fields, and is constantly applied in actual industry fields.” Mere instructions to apply an exception using generic computer components cannot provide an inventive concept. The claims are not patent eligible.
Claim 6 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. As per claim 6, this claim recites an additional abstract idea of identifying a value, determining, and selecting the parameter type. The identifying a value, determining, and selecting the parameter type are processes that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. There are no other additional elements.
Claim 7 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. As per claim 7, this claim recites an additional element of providing the routine to other assistants.
(Step 2A, prong 2) The judicial exception is not integrated into a practical application because the additional elements regarding providing are considered insignificant extra-solution activity. These limitations are not considered improvements to the functioning of a technology or technical field.
(Step 2B) The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the providing is not considered significantly more than the judicial exception. The additional elements represent only well-understood, routine, conventional activity that can be performed on generic computer systems. The transmitting of data (providing the routine to additional automated assistants) has been recognized by the courts as being well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity. See Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network), and MPEP 2106.05(d), subsection II. The claims are not patent eligible.
As per claim 8, this claim has similar receiving, determining (determining that the automated assistant is not configured to fulfill the first request), and storing steps as claim 1, and similar transmitting, identifying, and determining (determining the routine) steps as claim 3, and is rejected similarly to claims 1 and 3 for each of the respectively similar steps.
Claim 9 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. As per claim 9, this claim recites an additional abstract idea of generating a pattern and associated the pattern with the routine. The generating a pattern and associating is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. There are no other additional elements.
As per claim 10, this claim has similar generating steps as claim 9, and is rejected similarly to claim 9.
As per claim 11, this claim has similar receiving, determining, and causing steps as claim 2, and similar identifying steps as claim 6, and is rejected similarly to claims 2 and 6 for each of the respectively similar steps.
As per claim 12, this claim has similar receiving and causing steps as claim 2, and is rejected similarly to claim 2.
As per claim 13, this claim has similar receiving and processing steps as claim 4, and is rejected similarly to claim 4.
As per claim 14, this claim has similar identifying, determining, and selecting steps as claim 6, and is rejected similarly to claim 6.
Claims 15-20 recite substantially similar limitations to claims 1-6 respectively and are thus rejected along the same rationales.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 8-12 and 14 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Miller et al (US20200310749A1; filed 3/28/2019; hereinafter Miller).
With regards to claim 8, Miller discloses a method implemented by one or more processors (Miller, paragraph 40: “some functions may be carried out by a processor executing instructions stored in memory.”), the method comprising:
receiving a first request from a user for an automated assistant, executing at least in part on a mobile device, to perform an automated routine (Miller, abstract: “The monitoring mode of operation can be initiated when a user command is determined to be a new or unknown command;” paragraph 3: “an interactive computing device, such as a smart speaker or mobile device”);
determining that the automated assistant is not configured to fulfill the first request (Miller, abstract: “The monitoring mode of operation can be initiated when a user command is determined to be a new or unknown command”);
in response to determining that the automated assistant is not configured to fulfill the first request:
transmitting a notification, wherein the notification indicates that the automated assistant is not configured to fulfill the first request and is configured to determine, based on user interactions with an interface of the mobile device, a routine to fulfill the first request (Miller, paragraph 4: “In some instances, user consent may be obtained prior to performing the heightened user-activity monitoring, and in some instances, the computing device may provide an indication to the user that a monitoring mode (or learning mode) of operation is occurring”);
identifying one or more user actions performed by the user, wherein the one or more user actions correspond to one or more interactions of the user with the interface (Miller, paragraph 5: “Upon initiating a monitoring mode of operation, the computing devices associated with a user (i.e., “user devices”) may employ one or more sensors and/or monitoring software services to generate data relevant to a user's activity on a user device(s). Such user activity can be monitored as response activity event(s) (which may be performed by a user or at the direction of the user) associated with one or more user devices. The response activity event(s) may be monitored, tracked, and used for determining a response activity pattern.”);
determining the routine based on the one or more user actions; and storing the routine with an association to the first request (Miller, paragraph 31: “The received user data may be monitored, analyzed (e.g., feature extraction) and information about the response activity event(s) may be stored (e.g., in a user profile, such as user profile 240 of FIG. 2) to facilitate a pattern analysis;” paragraph 54: “The set of known commands and/or response actions may be stored as a list, library, index, and/or in a database, such as response actions 235, which are stored in storage 225”).
With regards to claim 9, which depends on claim 8, Miller discloses generating a request pattern, wherein the request pattern includes one or more terms of the first request and a parameter type of one or more other terms of the first request (Miller, paragraph 5: “The response activity event(s) may be monitored, tracked, and used for determining a response activity pattern. The response activity patterns may include, without limitation, patterns based on time, location, content, or other context;” paragraph 28: “content (e.g., the user command “It's a little chilly in here” relates to the response activity pattern of turning up a smart thermostat by two degrees, as opposed to, the user command “I'm freezing” relates to the response activity pattern of turning up a smart thermostat by four degrees);” the content includes the terms included in the command, and the parameter is interpreted like the “settings or values” described in paragraph 112, which are associated with certain command types); and
associating the request pattern with the routine (Miller, paragraph 29: “the response activity events may be associated with, or used to generate, a response profile that corresponds to the user command, and is associated with the task based on a history of sensed user activity. In this way, when an indication of a user command is received in the future, the response activity event(s) associated with the task can be determined and used to cause an operation to be performed to carry out the task”), wherein the routine includes at least one action that is performed utilizing a value of the parameter type (Miller, paragraph 112: “these prompts may be learned as well, such as wherein a response activity pattern shows a strong pattern for a particular operation feature type or category (e.g., launching a particular app or contacting a particular website, such as the OpenTable app, which is used making restaurant reservations) but the feature's values (i.e., parameter values) of for that feature type are varied (e.g., have a high variance or divergence)… the response activity pattern shows that certain operations are always performed (e.g., launching a restaurant reservations app) but the settings or values associated with those operations vary (e.g., the specific restaurant selected in the app varies)”).
With regards to claim 10, which depends on claim 9, Miller discloses wherein the request pattern is further generated based on previous interactions of the user (Miller, paragraph 81: “features similarity identifier 264 may be used to determine a set of response activity events that have in-common features… Behavior features may comprise behaviors such as user activities that tend to occur with certain locations or activities occurring before or after a given user activity event (or sequence of previous activity events), for example”).
With regards to claim 11, which depends on claim 9, Miller discloses receiving a second request from the user to perform a requested routine (Miller, paragraph 3: “an interactive computing device, such as a smart speaker or mobile device, or a digital assistant program operating thereon, can learn to carry out a specific response activity associated with a particular user command The user command may comprise a request to perform a task, such as turning on the lights.”);
determining that the second request matches the request pattern; identifying a request parameter value from one or more terms of the second request; and causing the routine to be executed with the request parameter value (Miller, paragraph 112: “these prompts may be learned as well, such as wherein a response activity pattern shows a strong pattern for a particular operation feature type or category (e.g., launching a particular app or contacting a particular website, such as the OpenTable app, which is used making restaurant reservations) but the feature's values (i.e., parameter values) of for that feature type are varied (e.g., have a high variance or divergence)… the response activity pattern shows that certain operations are always performed (e.g., launching a restaurant reservations app) but the settings or values associated with those operations vary (e.g., the specific restaurant selected in the app varies)”).
With regards to claim 12, which depends on claim 8, Miller discloses receiving a second request from the user for the automated assistant to perform the automated routine; and causing the routine to be executed (Miller, paragraph 3: “an interactive computing device, such as a smart speaker or mobile device, or a digital assistant program operating thereon, can learn to carry out a specific response activity associated with a particular user command The user command may comprise a request to perform a task, such as turning on the lights.” The activity is learned, so it is executed in response to the command).
With regards to claim 14, which depends on claim 9, Miller discloses wherein generating the request pattern includes: identifying a parameter value included in the request; determining that at least one of the user actions includes the parameter value; and selecting the parameter type of the parameter value to include in the request pattern (Miller, paragraph 112: “these prompts may be learned as well, such as wherein a response activity pattern shows a strong pattern for a particular operation feature type or category (e.g., launching a particular app or contacting a particular website, such as the OpenTable app, which is used making restaurant reservations) but the feature's values (i.e., parameter values) of for that feature type are varied (e.g., have a high variance or divergence)… the response activity pattern shows that certain operations are always performed (e.g., launching a restaurant reservations app) but the settings or values associated with those operations vary (e.g., the specific restaurant selected in the app varies)”).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 6-7, 13, 15-18, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Miller et al (US20200310749A1; filed 3/28/2019; hereinafter Miller) in view of Bradfield (US20220276882A1; filed 3/1/2022).
With regards to claim 1, Miller discloses a method implemented by one or more processors (Miller, paragraph 40: “some functions may be carried out by a processor executing instructions stored in memory.”), the method comprising:
receiving a first request from a user for an automated assistant to perform a first automated routine (Miller, abstract: “The monitoring mode of operation can be initiated when a user command is determined to be a new or unknown command”);
determining that the automated assistant is not configured to fulfill the first request (Miller, abstract: “The monitoring mode of operation can be initiated when a user command is determined to be a new or unknown command”);
in response to determining that the automated assistant is not configured to fulfill the first request (Miller, paragraph 4: “Upon determining that the user command does not correspond to a known response action…”):
determining that the first request includes one or more terms and a first parameter value of a parameter type (Miller, paragraph 5: “The response activity event(s) may be monitored, tracked, and used for determining a response activity pattern. The response activity patterns may include, without limitation, patterns based on time, location, content, or other context;” paragraph 28: “content (e.g., the user command “It's a little chilly in here” relates to the response activity pattern of turning up a smart thermostat by two degrees, as opposed to, the user command “I'm freezing” relates to the response activity pattern of turning up a smart thermostat by four degrees);” the content includes the terms included in the command, and the context includes context parameters such as time, location, etc. The parameter could also be interpreted like the “settings or values” described in paragraph 112, which are associated with certain command types)…
routines includes a plurality of user actions performed by the user and/or other users while interacting with an interface of a mobile device (Miller, paragraph 3: “At a high level, an interactive computing device, such as a smart speaker or mobile device, or a digital assistant program operating thereon, can learn to carry out a specific response activity associated with a particular user command”)…
generating a general routine… wherein executing the general routine with the first parameter value results in the automated assistant performing the first automated routine (Miller, paragraph 29: “the response activity events may be associated with, or used to generate, a response profile that corresponds to the user command, and is associated with the task based on a history of sensed user activity. In this way, when an indication of a user command is received in the future, the response activity event(s) associated with the task can be determined and used to cause an operation to be performed to carry out the task”);
generating a request pattern based on a corresponding user input request of at least one of the previous routines, wherein the request pattern includes one or more of the terms and the parameter type (Miller, paragraph 29: “These response activity event(s) can be associated with the task. From this response activity information, a computer system may learn response activity events that correspond to the user command and are associated with user activity related to the user device(s) that carry out the task (from the user command)”); and
storing the general routine with an association to the request pattern (Miller, paragraph 31: “The received user data may be monitored, analyzed (e.g., feature extraction) and information about the response activity event(s) may be stored (e.g., in a user profile, such as user profile 240 of FIG. 2) to facilitate a pattern analysis;” paragraph 54: “The set of known commands and/or response actions may be stored as a list, library, index, and/or in a database, such as response actions 235, which are stored in storage 225”).
However, Miller does not disclose identifying one or more previous routines, wherein each of the identified previous routines includes a plurality of user actions performed by the user and/or other users… and wherein the previous routines are each associated with a corresponding user input request that includes an input parameter of the parameter type and one or more of the terms; generating… based on the one or more previous routines
Bradfield teaches identifying one or more previous routines, wherein each of the identified previous routines includes a plurality of user actions performed by the user and/or other users… (Bradfield, paragraph 17: “That is, the AI-systems described herein use their previously recorded scripts as guides for deep learning, to be able decipher the steps required to complete user/customer support tasks without the need for referencing any dedicated help base articles or tutorials that are specific to the digital product in question”) and wherein the previous routines are each associated with a corresponding user input request (Bradfield, paragraph 35: “The AI based computer method may further comprise receiving, by the one or more processors, a request from a client device of a user for assistance with a given user task, wherein the client device is a device corresponding to the computing device;” paragraph 214: “The screenshots are time-coded, so that the screenshot can be associated with a specific moment when the various help commands are described in the audio transcript”) that includes an input parameter of the parameter type and one or more of the terms (Bradfield, paragraph 72: “the machine learning model may use, or be trained with, a help base input which may comprise or include text or graphical data or other information which may include numbered steps, capitalization of key action terms (e.g. “Settings”, “Billing”, etc.)”); generating… based on the one or more previous routines (Bradfield, paragraph 17: “That is, the AI-systems described herein use their previously recorded scripts as guides for deep learning, to be able decipher the steps required to complete user/customer support tasks without the need for referencing any dedicated help base articles or tutorials that are specific to the digital product in question;” paragraph 158: “TOOL 4 is able to adapt action scripts that were previously proven/validated on a predecessor digital product to fit a novel digital product”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have combined Miller and Bradfield such that the new routines and patterns are based off of older, similar routines and patterns. This would have enabled the routines to be continuously updated to correct for errors (Bradfield, paragraph 6: “there is also a strong need for automated, AI-based systems and methods that can autonomously and efficiently generate new, superior customer support self-help guides with clarifying graphic components (“help guides”)—which are automatically error-checked and updated to conform in real-time to actual world conditions”).
With regards to claim 2, which depends on claim 1, Miller discloses receiving a second request from the user for the automated assistant to perform a second automated routine, wherein the second request includes one or more of the terms and a second parameter value of the parameter type (Miller, paragraph 3: “an interactive computing device, such as a smart speaker or mobile device, or a digital assistant program operating thereon, can learn to carry out a specific response activity associated with a particular user command The user command may comprise a request to perform a task, such as turning on the lights.” The second automated routine and second parameter value can just be interpreted as the same as the first routine and parameter, so the learned command would just be based on the same command inputs);
determining that the second request matches the request pattern; and in response to determining that the second request matches the request parameter: causing the general routine to be executed with the second parameter value (Miller, paragraph 46: “initiating a response activity triggered by a recognized command;” as cited above, Miller initiates the activity based on the recognize terms (content) and parameters (context) of the command).
With regards to claim 3, which depends on claim 1, Miller discloses in response to determining that the automated assistant is not configured to fulfill the first request (Miller, abstract: “…when a user command is determined to be a new or unknown command…”):
transmitting a notification, wherein the notification indicates that the automated assistant is not configured to fulfill the first request and is configured to determine, based on user interactions with an interface of the mobile device, a routine to fulfill the first request (Miller, paragraph 4: “In some instances, user consent may be obtained prior to performing the heightened user-activity monitoring, and in some instances, the computing device may provide an indication to the user that a monitoring mode (or learning mode) of operation is occurring”); and
identifying one or more user actions performed by the user, wherein the one or more user actions correspond to one or more of the interactions of the user with the interface, wherein determining the general routine is further based on the one or more user actions (Miller, paragraph 5: “Upon initiating a monitoring mode of operation, the computing devices associated with a user (i.e., “user devices”) may employ one or more sensors and/or monitoring software services to generate data relevant to a user's activity on a user device(s). Such user activity can be monitored as response activity event(s) (which may be performed by a user or at the direction of the user) associated with one or more user devices. The response activity event(s) may be monitored, tracked, and used for determining a response activity pattern.”).
With regards to claim 4, which depends on claim 3, Miller does not disclose wherein identifying the one or more user actions includes: receiving a plurality of screenshots of an interface captured while the user is interacting with the interface; and processing the screenshots to identify the one or more user actions.
However, Bradfield teaches wherein identifying the one or more user actions includes: receiving a plurality of screenshots of an interface captured while the user is interacting with the interface (Bradfield, paragraph 120: “While executing those actions, TOOL 2 also simultaneously records a screenshot from the GUI of each step within the action script”); and processing the screenshots to identify the one or more user actions (Bradfield, paragraph 121: “TOOL 2 uses the device interface label information derived by TOOL 1 to assess which interface to use, including the correct version information (e.g. for operating system versions, software application versions/builds, devices with different versions/models, web browser types and versions, etc.) to ensure that the screenshots reflect the actual experience of an end-user”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have combined Miller and Bradfield such that the routines are based off interactions identified using screenshots. This would have provided the invention a means of identifying user actions (Bradfield, paragraph 121: “The screenshots recorded by TOOL 2 are taken from the GUI representation of the actual—or otherwise real or live—interface that an actual customer/user of the website, app, software application or other digital product would be viewing or would otherwise experience… to ensure that the screenshots reflect the actual experience of an end-user”).
With regards to claim 6, which depends on claim 3, Miller discloses wherein generating the request pattern includes: identifying a parameter value included in the request (Miller, paragraph 5: “The response activity event(s) may be monitored, tracked, and used for determining a response activity pattern. The response activity patterns may include, without limitation, patterns based on time, location, content, or other context;” paragraph 28: “content (e.g., the user command “It's a little chilly in here” relates to the response activity pattern of turning up a smart thermostat by two degrees, as opposed to, the user command “I'm freezing” relates to the response activity pattern of turning up a smart thermostat by four degrees);” the content includes the terms included in the command, and the context includes context parameters such as time, location, etc.);
determining that at least one of the user actions includes the parameter value; and
selecting the parameter type of the parameter value to include in the request pattern (Miller, paragraph 5: “The response activity patterns may include, without limitation, patterns based on time, location, content, or other context”).
With regards to claim 7, which depends on claim 1, Miller discloses providing the general routine to one or more additional automated assistants (Miller, paragraph 7: “the response activity pattern (or information derived from the patterns such as operations to perform to carry out a task) may be included in a response profile that is provided on a server or made accessible to other computing devices so that the devices can learn to respond to new or unknown commands. In this way, digital-assistant technology is improved by enabling computing devices (or digital assistants operating thereon) to learn to respond to new or previously unknown commands”).
Claim 13 recites substantially similar limitations to claim 4 and is thus rejected along the same rationale.
Claims 15-18 recite substantially similar limitations to claims 1-4 respectively and are thus rejected along the same rationales.
Claim 20 recites substantially similar limitations to claim 6 and is thus rejected along the same rationale.
Claim(s) 5 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Miller et al in view of Bradfield, and further in view of Jelveh (US20210264219A1; filed 5/12/2021).
With regards to claim 5, which depends on claim 4, Miller and Bradfield do not teach wherein processing the screenshots includes: providing the screenshots as input to a trained machine learning model; receiving, as output, one or more indications of an action that was performed by the user while the screenshots were captured; and processing the indications to determine the one or more user actions.
Jelveh teaches wherein processing the screenshots includes: providing the screenshots as input to a trained machine learning model; receiving, as output, one or more indications of an action that was performed by the user while the screenshots were captured (Jelveh, abstract: “A machine learning model is trained on analysis of graphical image data associated with screen display to determine or infer user intent. An input component receives image data regarding a screen display associated with user interaction with a computing device”); and processing the indications to determine the one or more user actions (Jelveh, abstract: “An analysis component employs the model to determine or infer user intent based on the image data analysis; and an action component provisions services to the user as a function of the determined or inferred user intent;” paragraph 6: “a machine learning model can be trained on analysis of graphical image data associated with screen display to determine or infer user intent and/or action”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have combined Miller, Bradfield, and Jelveh such that the screenshots are interpreted using a machine learning model to identify the user inputs. This would have enabled the invention to accurately identify the inputs (Jelveh, paragraph 8: “The model can employ a recursive learning algorithm or backward propagation of learning across other models or continuous learning algorithm. The model can learn impact of respective images and direct the action component to revise actions as a function of the learned impact. An optimization component can generate inferences, based on the model, regarding potential points of failure, weakness or bottlenecks in connection with taking automated action”).
Claim 19 recites substantially similar limitations to claim 5 and is thus rejected along the same rationale.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Wei et al (US20080114604A1): Teaches recording a sequence of user actions to create a higher order command.
Intharah et al (T. Intharah, D. Turmukhambetov, and G. J. Brostow, “Help, it looks confusing: GUI task automation through demonstration and follow-up questions,” in Proc. 22nd Int. Conf. Intell. User Interfaces, Mar. 2017, pp. 233–243, doi: 10.1145/3025171.3025176.): Teaches recording a sequence of user actions for task automation.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRODERICK C ANDERSON whose telephone number is (313)446-6566. The examiner can normally be reached Monday-Tuesday, Thursday-Saturday 9-5 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen Hong can be reached at 5712724124. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/B.C.A/Examiner, Art Unit 2178
/STEPHEN S HONG/Supervisory Patent Examiner, Art Unit 2178
Prosecution Insights