Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are pending for examination. Claims 1, 8, and 14 are independent.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1
According to the first part of the analysis, in the instant case, claims 1-7 are directed to a method, claims 8-13 are directed to a system, and claims 14-20 is directed to a non-transitory machine-readable medium. Thus, each of the claims falls within one of the four statutory categories (i.e., process, machine, manufacture, or composition of matter).
Regarding Claim 1
2A Prong 1:
generating, (This step for generating a first action is practically performable in the human mind and is understood to be a recitation of a mental process (i.e., judgment/evaluation).)
generating, (This step for generating reflective text is practically performable in the human mind and is understood to be a recitation of a mental process (i.e., judgment/evaluation).)
generating a second prompt based on the first prompt and the reflective text; (This step for generating a second prompt is practically performable in the human mind and is understood to be a recitation of a mental process (i.e., judgment/evaluation).)
generating, (This step for generating a second action is practically performable in the human mind and is understood to be a recitation of a mental process (i.e., judgment/evaluation).) and
updating parameters of the second neural network based language model based on a comparison of a first reward generated based on the first state and a second reward generated based on a resulting second state of the environment after performing the second action. (This step for updating parameters is practically performable in the human mind and is understood to be a recitation of a mental process (i.e., judgment/evaluation) or mathematical calculations.)
2A Prong 2: This judicial exception is not integrated into a practical application.
Additional elements:
A method of training a neural network based agent, the method comprising: (Training a machine learning model is understood as mere instructions to implement an abstract idea on a computer - see MPEP 2106.05(f).))
by a first neural network based language model based on a first prompt describing a target task (This step is adding the words “apply it” (or an equivalent) with the judicial exception, or merely applying a generic neural network as a tool to perform the abstract idea - see MPEP 2106.05(f).)
by a second neural network based language model (This step is adding the words “apply it” (or an equivalent) with the judicial exception, or merely applying a generic neural network as a tool to perform the abstract idea - see MPEP 2106.05(f).)
by the first neural network based language model based on the second prompt (This step is adding the words “apply it” (or an equivalent) with the judicial exception, or merely applying a generic neural network as a tool to perform the abstract idea - see MPEP 2106.05(f).)
The additional elements as disclosed above alone or in combination do not integrate the judicial exception into practical application as they are generic computer functions that are implemented to perform the disclosed abstract idea above.
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
A method of training a neural network based agent, the method comprising: (Training a machine learning model is understood as mere instructions to implement an abstract idea on a computer - see MPEP 2106.05(f).))
by a first neural network based language model based on a first prompt describing a target task (This step is adding the words “apply it” (or an equivalent) with the judicial exception, or merely applying a generic neural network as a tool to perform the abstract idea - see MPEP 2106.05(f).)
by a second neural network based language model (This step is adding the words “apply it” (or an equivalent) with the judicial exception, or merely applying a generic neural network as a tool to perform the abstract idea - see MPEP 2106.05(f).)
by the first neural network based language model based on the second prompt (This step is adding the words “apply it” (or an equivalent) with the judicial exception, or merely applying a generic neural network as a tool to perform the abstract idea - see MPEP 2106.05(f).)
The additional elements as disclosed above in combination of the abstract idea
are not sufficient to amount to significantly more than the judicial exception as they are
generic computer functions that are implemented to perform the disclosed abstract idea above.
Regarding Claim 8: see the rejection of claim 1 above for similar limitations. Same rationale applies.
2A Prong 2:
A system for training a neural network based agent, the system comprising: (The system is understood to be a generic computer element - See MPEP 2106.05(f).)
a memory that stores a first neural network based language model, a second neural network based language model, and a plurality of processor executable instructions; (The memory is understood to be a generic computer element - See MPEP 2106.05(f). The step directed to storing information, is understood to be insignificant extra- solution activity and data gathering. See MPEP 2106.05(g).)
a communication interface that receives a target task; (The communication interface is understood to be a generic computer element - See MPEP 2106.05(f). The step directed to transmitting or receiving information, is understood to be insignificant extra-solution activity and data gathering. See MPEP 2106.05(g).) and
one or more hardware processors that read and execute the plurality of processor-executable instructions from the memory to perform operations comprising: (The processor is understood to be a generic computer element - See MPEP 2106.05(f).)
The additional elements as disclosed above alone or in combination do not integrate the judicial exception into practical application as they are insignificant extra solution activity in combination of generic computer functions that are implemented to perform the disclosed abstract idea above.
2B:
A system for training a neural network based agent, the system comprising: (The system is understood to be a generic computer element - See MPEP 2106.05(f).)
a memory that stores a first neural network based language model, a second neural network based language model, and a plurality of processor executable instructions; (The memory is understood to be a generic computer element - See MPEP 2106.05(f). This step is directed to storing information, which is understood to be insignificant extra-solution activity and is well understood, routine and conventional activity as identified by the court (MPEP 2106.05(d)(ll)(IV)))))
a communication interface that receives a target task; (The communication interface is understood to be a generic computer element - See MPEP 2106.05(f). This step is directed to transmitting or receiving information, which is understood to be insignificant extra-solution activity and is well understood, routine and conventional activity of transmitting and receiving data as identified by the court (MPEP2106.05(d)(ll)(i)))) and
one or more hardware processors that read and execute the plurality of processor-executable instructions from the memory to perform operations comprising: (The processor is understood to be a generic computer element - See MPEP 2106.05(f).)
The additional elements as disclosed above in combination of the abstract idea
are not sufficient to amount to significantly more than the judicial exception as they are
well, understood, routine and conventional activity as disclosed in combination
of generic computer functions that are implemented to perform the disclosed abstract idea above.
Regarding Claim 14: see the rejection of claim 1 above. Same rationale applies.
2A Prong 2 & 2B: The claim recites another additional element “A non-transitory machine-readable medium comprising a plurality of machine-executable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising:” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
Regarding Claims 2, 9, and 15
2A Prong 1:
wherein the generating the reflective text is further based on the first action. (This step is practically performable in the human mind and is understood to be a recitation of a mental process (i.e., judgment/evaluation).)
2A Prong 2 & 2B: The claim does not recite any additional elements.
Regarding Claims 3, 10, and 16
2A Prong 1:
further comprising keeping all parameters of the first neural network based language model frozen while updating parameters of the second neural network based language model. (This step is practically performable in the human mind and is understood to be a recitation of a mental process (i.e., judgment/evaluation) or mathematical calculations.)
2A Prong 2 & 2B: The claim does not recite any additional elements.
Regarding Claims 4, 11, and 17
2A Prong 1: The claim does not recite any Abstract idea.
2A Prong 2 & 2B:
wherein the environment is a website interface. (The specification of data to be stored is understood to be a field of use limitation. The limitation further specifies the environment - See MPEP 2106.05(h).)
Regarding Claims 5, 12, and 18
2A Prong 1: The claim does not recite any Abstract idea.
2A Prong 2 & 2B:
wherein the environment is an e-commerce website, and wherein the first action is associated with making a purchase. (The specification of data to be stored is understood to be a field of use limitation. The limitation further specifies the environment and action - See MPEP 2106.05(h).)
Regarding Claims 6, 13, and 19
2A Prong 1: The claim does not recite any Abstract idea.
2A Prong 2:
wherein determining the first reward comprises receiving a reward indication from a user interface device. (This step is directed to transmitting or receiving information, which is understood to be insignificant extra-solution activity and data gathering. See MPEP 2106.05(g).)
2B:
wherein determining the first reward comprises receiving a reward indication from a user interface device. (This step is directed to transmitting or receiving information, which is understood to be insignificant extra-solution activity and is well understood, routine and conventional activity of transmitting and receiving data as identified by the court (MPEP2106.05(d)(ll)(i))))
Regarding Claims 7 and 20
2A Prong 1: The claim does not recite any Abstract idea.
2A Prong 2 & 2B:
wherein the first prompt is based on a task instruction received via a user interface device. (The specification of data to be stored is understood to be a field of use limitation. The limitation further specifies the environment and action - See MPEP 2106.05(h).)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shinn et al. ("Reflexion: Language Agents with Verbal Reinforcement Learning", hereinafter "Shinn") in view of Kroener et al. (US 20240144023 A1, hereinafter "Kroener").
Regarding Claim 1
Shinn discloses: A method of training a neural network based agent ([Abstract, and Section C Programming]), the method comprising:
generating, by a first neural network based language model based on a first prompt describing a target task, a first action towards completing the target task in an environment; ([Section 3, Page 4 section Actor, and Fig 1-2] describes an actor LLM (i.e. first neural network based language model) prompted to generate actions in an environment to accomplish a task (e.g. see fig 1).)
generating, by a second neural network based language model, a reflective text associated with the first action based on a resulting first state of the environment after performing the first action on the environment, wherein the reflective text is indicative of at least one of a problem with the first action or a suggestion associated with determining actions; ([Section 3, pages 4-5 section Self-reflection, page 5 section The Reflexion process, and Fig 1-2] describes a Self-Reflection model instantiated as an LLM (i.e. second neural network based language model) generates reflective text after observing outcomes of previous actions. Examiner interprets the reflection text as identifying mistakes and proposing improvements.)
generating a second prompt based on the first prompt and the reflective text; ([Section 3, page 5 section memory, and Fig 1-2] describes reflections stored and incorporated into future prompts (i.e. second prompt).)
generating, by the first neural network based language model based on the second prompt, a second action; ([Section 3, Page 4 section Actor, and Fig 1-2] describes the memory component providing additional context to the agent (i.e. second/future prompt) to generate a second action. Examiner interprets the agent retrying the task using the augmented prompt.) and
updating parameters of the second neural network based language model based on a comparison of a first reward generated based on the first state and a second reward generated based on a resulting second state of the environment after performing the second action.
Shinn does not explicitly disclose: updating parameters of the second neural network based language model based on a comparison of a first reward generated based on the first state and a second reward generated based on a resulting second state of the environment after performing the second action.
However, Kroener discloses in the same field of endeavor: updating parameters of the second neural network based language model based on a comparison of a first reward generated based on the first state and a second reward generated based on a resulting second state of the environment after performing the second action. ([Para 0007-0009, 00420-0043, 0051-0055, and Fig 6-7], Kroener describes a policy optimization for updating weights (i.e. parameters) of a critic model (i.e. second neural network) based on a comparison of a first and second reward.)
It would have been obvious to a person of ordinary skill in art before the effective filling date of the invention to implement the function of training a reinforcement model disclosed by Kroener into the method of Reflexion Language Agents disclosed by Shinn to update parameters of a second neural network. The modification would have been obvious because one of the ordinary skills of the art would be motivated to utilize the feature of training a reinforcement model disclosed by Kroener as all the references are in the field of machine learning. A person of ordinary skill of the art would have been motivated to perform the combination for being able to perform optimization of a model parameter to improve performance.
Regarding Claim 8
Shinn in view of Kroener discloses: A system for training a neural network based agent, the system comprising: a memory that stores a first neural network based language model, a second neural network based language model, and a plurality of processor executable instructions; a communication interface that receives a target task; and one or more hardware processors that read and execute the plurality of processor-executable instructions from the memory to perform operations ([Para 0029-0039 and Fig 2] describe a system.) comprising: (Claim 8 is a system claim that corresponds to claim 1 and the rest of the limitations are rejected on the same ground.)
Regarding Claim 14
Shinn in view of Kroener discloses: A non-transitory machine-readable medium comprising a plurality of machine-executable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations ([Para 0029-0039 and Fig 2] describe non-transitory computer readable storage.) comprising: (Claim 14 is a non-transitory machine-readable medium claim that corresponds to claim 1 and the rest of the limitations are rejected on the same ground.)
Regarding Claim 2
Shinn in view of Kroener discloses: The method of claim 1, wherein the generating the reflective text is further based on the first action. ([Section 3, pages 4-5 section Self-reflection, page 5 section The Reflexion process, and Fig 1-2], Shinn describes generating reflective text after observing outcomes of previous actions (i.e. based on first action).)
Regarding Claim 3
Shinn in view of Kroener discloses: The method of claim 1, further comprising keeping all parameters of the first neural network based language model frozen while updating parameters of the second neural network based language model. ([Para 0007-0009, 00420-0043, 0051-0055, and Fig 6-7], Kroener describes a policy optimization for freezing and updating weights (i.e. parameters).)
Regarding Claim 4
Shinn in view of Kroener discloses: The method of claim 1, wherein the environment is a website interface. ([Page 13 and Section B.1 WebShop Limitation], Shinn describes experiment on WebShop (i.e. website environment).)
Regarding Claim 5
Shinn in view of Kroener discloses: The method of claim 4, wherein the environment is an e-commerce website, and wherein the first action is associated with making a purchase. ([Page 13 and Section B.1 WebShop Limitation], Shinn describes WebShop an e-commerce website to locate and purchase products.)
Regarding Claim 6
Shinn in view of Kroener discloses: The method of claim 1, wherein determining the first reward comprises receiving a reward indication from a user interface device. ([Section B.1 WebShop Limitation, Section 4.3, and Section D Reasoning], Shinn describes reflexion on Webshop and ALFworld goals presented through an interface.)
Regarding Claim 7
Shinn in view of Kroener discloses: The method of claim 1, wherein the first prompt is based on a task instruction received via a user interface device. ([Section B.1 WebShop Limitation, Section 4.3, and Section D Reasoning], Shinn describes reflexion on Webshop and ALFworld goals presented through an interface.)
Regarding Claim 9
(Claim 9 recites analogous limitations to claim 2 and therefore is rejected on the same ground as claim 2.)
Regarding Claim 10
(Claim 10 recites analogous limitations to claim 3 and therefore is rejected on the same ground as claim 3.)
Regarding Claim 11
(Claim 11 recites analogous limitations to claim 4 and therefore is rejected on the same ground as claim 4.)
Regarding Claim 12
(Claim 12 recites analogous limitations to claim 5 and therefore is rejected on the same ground as claim 5.)
Regarding Claim 13
(Claim 13 recites analogous limitations to claim 6 and therefore is rejected on the same ground as claim 6.)
Regarding Claim 15
(Claim 15 recites analogous limitations to claim 2 and therefore is rejected on the same ground as claim 2.)
Regarding Claim 16
(Claim 15 recites analogous limitations to claim 3 and therefore is rejected on the same ground as claim 3.)
Regarding Claim 17
(Claim 17 recites analogous limitations to claim 4 and therefore is rejected on the same ground as claim 4.)
Regarding Claim 18
(Claim 18 recites analogous limitations to claim 5 and therefore is rejected on the same ground as claim 5.)
Regarding Claim 19
(Claim 19 recites analogous limitations to claim 6 and therefore is rejected on the same ground as claim 6.)
Regarding Claim 20
(Claim 20 recites analogous limitations to claim 7 and therefore is rejected on the same ground as claim 7.)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Budden et al. (US 20200265305 A1) describes using reinforcement learning system with replay memory. Soyer at al. (US 20210034970 A1) describes reinforcement learning with an actor critic model.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TEWODROS E MENGISTU whose telephone number is (571)270-7714. The examiner can normally be reached Mon-Fri 9:30-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ABDULLAH KAWSAR can be reached at (571)270-3169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TEWODROS E MENGISTU/Examiner, Art Unit 2127
Prosecution Insights