Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. JP2023-049347, filed on March 27, 2023.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
The following is a quotation of pre-AIA 35 U.S.C. 112, fourth paragraph:
Subject to the following paragraph [i.e., the fifth paragraph of pre-AIA 35 U.S.C. 112], a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claims 8,9, 12-15 are rejected under 35 U.S.C. 112(d) or pre-AIA 35 U.S.C. 112, 4th paragraph, as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends.
Claim 8 – Improper Dependency
An independent Claim 8 refers back to another independent Claim 1
Claim 9 – Improper Dependency
An independent Claim 9 refers back to another independent Claim 1
Claim 12 – Improper Dependency
An independent Claim 12 refers back to another independent Claim 1
Claim 13 – Improper Dependency
An independent Claim 13 refers back to another independent Claim 1
Claim 14 – Improper Dependency
An independent Claim 14 refers back to another independent Claim 8
Claim 15 – Improper Dependency
An independent Claim 15 refers back to another independent Claim 9
Applicant may cancel the claim(s), amend the claim(s) to place the claim(s) in proper dependent form, rewrite the claim(s) in independent form, or present a sufficient showing that the dependent claim(s) complies with the statutory requirements.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-15 are rejected under 35 U.S.C. 101 the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 1 is a method claim thus it falls into one of the four categories of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding independent claim 1, following limitations recite a judicial exception:
“increasing a value of a reward by repeating experiencing through the reinforcement learning, the experiencing including arranging and reward determining”
[Mental Process] – increasing a value by repeating experiencing through the reinforcement learning is similar to a human going through multiple steps of educational learning to find and modify a value which involves observations, evaluations, judgments, and opinions that is capable of being performed in the human mind with the assistance of paper and pen
“changing a reward function that defines a relationship between an amount of time taken according to the time schedule and a value of a corresponding reward”
[Mathematical Calculations] – changing a function that defines a relationship between multiple factors requires mathematical computation which recites to an abstract idea
“the arranging includes sequentially changing a state of a timetable by sequentially arranging a plurality of planning factors in the timetable, the timetable defining the time schedule, the plurality of planning factors being given in advance”
[Mental Process] – sequentially changing a state of a timetable associated with multiple timetable factors is simply changing the around the time schedule which involves observations, evaluations, judgments, and opinions that is capable of being performed in the human mind with the assistance of paper and pen
“the reward determining includes determining the value of the corresponding reward based on the reward function and a final state of the timetable in which all the plurality of planning factors are arranged”
[Mathematical Calculations] – determining the value based on the reward function requires mathematical computation which recites to an abstract idea
“the changing includes changing the reward function whose gradient in a partial section is larger than a gradient in the partial section of the reward function before the changing, the partial section being part of a time range from an amount of time taken according to a time schedule corresponding to a reward with a maximum value to an amount of time taken according to a time schedule corresponding to a reward with a minimum value”
[Mental process] – changing the reward function based on a number requires comparing two numbers which involves observations, evaluations, judgments, and opinions that is capable of being performed in the human mind with the assistance of paper and pen
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
Regarding Claim 1, the claim recites additional elements of
“each of the plurality of substrates”
The substrates, wafers used in metrology, are recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
[Even when viewed in combination, the additional elements do no more than {automate the mental processes that a person could perform, using computer components as a tool}, thus the claim as a whole does not integrate into a practical application.]
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional element [1] is considered a mere instruction to apply an exception to the generic computer components. (see MPEP 2106.05(f)) This limitation remains a mere instruction to apply an exception even upon reconsideration. Even when considered in combination, the additional element represents a mere instruction to apply an exception, which cannot provide an inventive concept.
Regarding Claim 2
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 2 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding dependent claim 2, following limitations recite a judicial exception:
“the changing includes changing to a reward function whose gradient in the partial section is larger than a gradient in the partial section of a linear function that is set in all the time range”
[Mental Process] – changing a reward function based on a number requires comparing and decision making to apply the change which involves observations, evaluations, judgments, and opinions that is capable of being performed in the human mind with the assistance of paper and pen
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 2 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 3
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 3 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding dependent claim 3, following limitations recite a judicial exception:
“the changing is repeated more than once, and the changing includes changing to a reward function whose gradient in the partial section is larger than a gradient in the partial section of an active reward function each time the changing is repeated”
[Mental Process] – changing reward function multiple times based on a number requires multiple series comparing and decision making to apply the change which involves observations, evaluations, judgments, and opinions that is capable of being performed in the human mind with the assistance of paper and pen
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 3 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 4
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 4 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding dependent claim 4, following limitations recite a judicial exception:
“the changing further includes shifting the partial section”
[Mental Process] – changing a reward function of multiple sections requires multiple series comparing and decision making to apply the change which involves observations, evaluations, judgments, and opinions that is capable of being performed in the human mind with the assistance of paper and pen
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 4 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 5
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 5 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding dependent claim 5, following limitations recite a judicial exception:
“a first planning factor that is a plan to load a substrate of the plurality of substrates into a substrate processing section of the plurality of substrate processing sections through the conveyance section to process the substrate through the substrate processing section”
[Mental Process] – a plan to load substrates to such sections involves physical movement that a human can which involves observations, evaluations, judgments, and opinions that is capable of being performed in the human mind with the assistance of paper and pen or human bodies
“a second planning factor that is a plan to unload the substrate processed by the substrate processing section from the substrate processing section through the conveyance section”
[Mental Process] – a plan to unload substrates out of such sections involves physical movement that a human can which involves observations, evaluations, judgments, and opinions that is capable of being performed in the human mind with the assistance of paper and pen or human bodies
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
Regarding Claim 5, the claim recites additional elements of
“a plurality of substrate processing sections that process the plurality of substrates, and a conveyance section that conveys the plurality of substrates”
These sections are recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
[Even when viewed in combination, the additional elements do no more than automate the mental processes that a person could perform, using computer components as a tool, thus the claim as a whole does not integrate into a practical application.]
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional element [1] is considered merely computer components that are just to store and execute code-based instructions which are considered a mere instruction to apply an exception and amount to storing and receiving information in memory, which is well-understood, routine, conventional activity (See MPEP 2106.05(d), subsection II). This limitation remains a mere instruction to apply an exception. Even when considered in combination, the additional element represents insignificant extra-solution activity, which cannot provide an inventive concept.
Regarding Claim 6
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 6 is a dependent claim of 5, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 6 does not have any abstract idea by itself, thus uses all the limitations of Claim 5.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 6 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 7
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 7 is a dependent claim of 5, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding dependent claim 7, following limitations recite a judicial exception:
“the arranging includes selecting a planning factor of the plurality of planning factors respectively corresponding to the plurality of substrate processing sections”
[Mental Process] – selecting a planning factor based on the situation of processing sections requires comparing and decision making which involves observations, evaluations, judgments, and opinions that is capable of being performed in the human mind with the assistance of paper and pen or human bodies
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 7 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 8
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 8 is an apparatus claim thus it falls into one of the four categories of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding independent claim 8, it refers back to the generating method of claim 1. Please look at Claim 1 above for the judicial exceptions.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
Regarding Claim 8, the claim recites additional elements of
“storage that stores a creation program that defines the schedule creation program generating method according to claim 1”
The storage is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
“a processor that executes the creation program to generate the schedule creation program”
The processor is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
[Even when viewed in combination, the additional elements do no more than automate the mental processes that a person could perform, using computer components as a tool, thus the claim as a whole does not integrate into a practical application.]
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional elements [1,2] are considered merely computer components that are just to store and execute code-based instructions which are considered a mere instruction to apply an exception and amount to storing and receiving information in memory, which is well-understood, routine, conventional activity (See MPEP 2106.05(d), subsection II). These limitations remain a mere instruction to apply an exception. Even when considered in combination, the additional element represents a mere instruction to apply an exception, which cannot provide an inventive concept.
Regarding Claim 9
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 9 is an apparatus claim thus it falls into one of the four categories of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding independent claim 9, it refers back to the generating method of claim 1. Please look at Claim 1 above for the judicial exceptions.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
Regarding Claim 9, the claim recites additional elements of
“storage that stores a schedule creation program that is created based on the schedule creation program generating method according to claim 1”
The storage is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
“a processor that executes the schedule creation program to create the time schedule”
The processor is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
[Even when viewed in combination, the additional elements do no more than automate the mental processes that a person could perform, using computer components as a tool, thus the claim as a whole does not integrate into a practical application.]
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional elements [1,2] are considered merely computer components that are just to store and execute code-based instructions which are considered a mere instruction to apply an exception and amount to storing and receiving information in memory, which is well-understood, routine, conventional activity (See MPEP 2106.05(d), subsection II). These limitations remain a mere instruction to apply an exception. Even when considered in combination, the additional element represents a mere instruction to apply an exception, which cannot provide an inventive concept.
Regarding Claims 10, 11
Claims 10 and 11 are a non-transitory computer-readable medium claims of Claim 1 which have similar limitations of Claim 1. For the reasons described above with respect to Claim 1, these judicial exceptions are not meaningfully integrated into a practical application, or significantly more than the abstract ideas. The claims do not provide anything more than the abstract ideas of mental processes and mathematical calculations that are practically capable of being performed with the assistance of pen and paper. Therefore, Claims 10 and 11 also recite abstract ideas that do not integrate into a practical application or amount to significantly more than judicial exception, and thus are rejected under U.S.C. 101.
Regarding Claim 12
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 12 is an apparatus claim thus it falls into one of the four categories of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding independent claim 12, it refers back to the generating method of claim 1. Please look at Claim 1 above for the judicial exceptions.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
Regarding Claim 12, the claim recites additional elements of
“a plurality of components that, while a substrate of a plurality of substrates is being processed, are occupied by the substrate”
The components are recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
“storage that stores a creation program that defines the schedule creation program generating method according to claim 1”
The storage is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
“a processor that executes the creation program to generate a schedule creation program for creating the time schedule for the plurality of components, the processor executing the schedule creation program to create the time schedule”
The processor is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
[Even when viewed in combination, the additional elements do no more than automate the mental processes that a person could perform, using computer components as a tool, thus the claim as a whole does not integrate into a practical application.]
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional elements [1,2,3] are considered merely computer components that are just to store and execute code-based instructions which are considered a mere instruction to apply an exception and amount to storing and receiving information in memory, which is well-understood, routine, conventional activity (See MPEP 2106.05(d), subsection II). These limitations remain a mere instruction to apply an exception. Even when considered in combination, the additional element represents a mere instruction to apply an exception, which cannot provide an inventive concept.
Regarding Claim 13
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 13 is an apparatus claim thus it falls into one of the four categories of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding independent claim 13, it refers back to the generating method of claim 1. Please look at Claim 1 above for the judicial exceptions.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
Regarding Claim 13, the claim recites additional elements of
“a plurality of components that, while a substrate of a plurality of substrates is being processed, are occupied by the substrate”
The components are recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
“storage that stores a schedule creation program that is created based on the schedule creation program generating method according to claim 1”
The storage is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
“a processor that executes the schedule creation program to create the time schedule”
The processor is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
[Even when viewed in combination, the additional elements do no more than automate the mental processes that a person could perform, using computer components as a tool, thus the claim as a whole does not integrate into a practical application.]
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional elements [1,2,3] are considered merely computer components that are just to store and execute code-based instructions which are considered a mere instruction to apply an exception and amount to storing and receiving information in memory, which is well-understood, routine, conventional activity (See MPEP 2106.05(d), subsection II). These limitations remain a mere instruction to apply an exception. Even when considered in combination, the additional element represents a mere instruction to apply an exception, which cannot provide an inventive concept.
Regarding Claim 14
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 14 is a system claim thus it falls into one of the four categories of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding independent claim 14, it refers back to the generating apparatus of Claim 8. Please look at Claim 8 above for the judicial exceptions.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
Regarding Claim 12, the claim recites additional elements of
“a substrate processing apparatus that processes a substrate”
The apparatus is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
“a transmitter that transmits the schedule creation program to the substrate processing apparatus”
The transmitter is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
“a plurality of components that, while a substrate of a plurality of substrates is being processed, are occupied by the substrate”
The components are recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
“a receiver that receives the schedule creation program transmitted from the transmitter of the schedule creation program generating apparatus”
The receiver is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
“a processor that executes the schedule creation program to create the time schedule for the plurality of components”
The processor is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
[Even when viewed in combination, the additional elements do no more than automate the mental processes that a person could perform, using computer components as a tool, thus the claim as a whole does not integrate into a practical application.]
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional elements [1,2,3,4,5] are considered merely computer components that are just to store and execute code-based instructions which are considered a mere instruction to apply an exception and amount to storing and receiving information in memory, which is well-understood, routine, conventional activity (See MPEP 2106.05(d), subsection II). These limitations remain a mere instruction to apply an exception. Even when considered in combination, the additional element represents a mere instruction to apply an exception, which cannot provide an inventive concept.
Regarding Claim 15
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 15 is a system claim thus it falls into one of the four categories of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding independent claim 15, it refers back to the generating method of claim 9. Please look at Claim 9 above for the judicial exceptions.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
Regarding Claim 15, the claim recites additional elements of
“a substrate processing apparatus that processes a substrate”
The apparatus is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
“a transmitter that transmits the time schedule to the substrate processing apparatus”
The transmitter is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
“a plurality of components that, while a substrate of a plurality of substrates is being processed, are occupied by the substrate”
The components are recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
“a receiver that receives the time schedule transmitted from the transmitter of the schedule creating apparatus”
The receiver is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
[Even when viewed in combination, the additional elements do no more than automate the mental processes that a person could perform, using computer components as a tool, thus the claim as a whole does not integrate into a practical application.]
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional elements [1,2,3,4] are considered merely computer components that are just to store and execute code-based instructions which are considered a mere instruction to apply an exception and amount to storing and receiving information in memory, which is well-understood, routine, conventional activity (See MPEP 2106.05(d), subsection II). These limitations remain a mere instruction to apply an exception. Even when considered in combination, the additional element represents a mere instruction to apply an exception, which cannot provide an inventive concept.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 2, 5-15 are rejected under 35 U.S.C. 103 as being unpatentable over Chau et al. (Chau), US Patent Application (Listed in IDS filed on March 22, 2024) No. US-2022/0171373-A1, published in June 2022, in view of Viquerat et al. (Viquerat), Non-Patent Literature, “DIRECT SHAPE OPTIMIZATION THROUGH DEEP REINFORCEMENT LEARNING”, published on December 2020, Pages: 16.
As to independent Claim 1,
Chau teaches
a schedule creation program generating method that generates, through reinforcement learning, a schedule creation program for creating a time schedule for a plurality of components included in a substrate processing apparatus, while a substrate of the plurality of substrates is being processed, the plurality of components being occupied by the substrate (Chau,Pg17, Claim1, "A system for processing semiconductor substrates in a tool comprising a plurality of processing chambers configured to process the semiconductor substrates according to a recipe... and schedule operations of the tool based on the optimum scheduling parameters for processing the one of the semiconductor substrates in the plurality of processing chambers according to the recipe"
Pg20, Claim53, "a third neural network coupled to the first and second plurality of neural networks and configured to predict the optimum route for routing the semiconductor substrates between the plurality of processing chambers and to predict the optimum time to schedule the additional semiconductor substrates for processing in the semiconductor processing tool"
Pg18, Claim22, "wherein a state of the tool includes indications of resources of the tool and a processing status of the semiconductor substrated a processing status of the semiconductor substrate"
Pg5, Paragraph85, Lines3-5, "The model is trained using a discrete event simulator and reinforcement learning to self - explore and memorize the best scheduling decisions for a given state of a system", wherein Chau discloses the reinforcement learning to output the optimum times or the best paths to process semiconductor substrates, where the processing chambers are occupied during the processing procedure, thus rendering it functionally equivalent to the claimed invention), wherein the schedule creation program generating method comprises:
increasing a value of a reward by repeating experiencing through the reinforcement learning, the experiencing including arranging and reward determining (Chau, Pg11, Paragraph165, Lines7-9, "Executing an action in a specific state provides the agent with a reward (a numerical score). The goal of the agent is to maximize its total (future) reward. The agent achieves the goal by adding a maximum reward"
Pg11, Paragraph166, "For example, the reinforcement learning method used by the model 1204 can include Q - learning. ... Q - learning maximizes the expected value of the total reward over all successive steps, starting from the current state",
Pg5, Paragraph85, Liines3-5, "The model is trained using a discrete event simulator and reinforcement learning to self - explore and memorize the best scheduling decisions for a given state of a system",
Pg12, Paragraph170, "At 1314, the discrete event simulator 1202 determines whether the final state is reached. The discrete event simulator 1202 repeats steps 1304-1312 until the final state is reached. At 1316, after the final state is reached, the reinforcement training of the model 1204 is complete. At 1318, the model 1204 uses the memorized best next operation for each state when that particular state occurs in the tool during actual wafer processing. This way, using the model 1204, the tool always selects the best path in which to move a wafer through the tool for optimum throughput performance when transitioning from one state to another", wherein Chau discloses the reinforcement learning that maximizes the total reward (the corresponding value of a reward), which the agent is provided with the reward (the corresponding reward determining). Also, the event simulator goes steps to provide the best paths wafers to be processed (the corresponding arrangement), rendering it functionally equivalent to the claimed invention); and
the arranging includes sequentially changing a state of a timetable by sequentially arranging a plurality of planning factors in the timetable, the timetable defining the time schedule, the plurality of planning factors being given in advance to each of the plurality of substrates (Chau, Pg18, Claim22, "for each of the plurality of states, send to the model a current state of the plurality of states and multiple
schedulable operations to progress to a next state of the plurality of states , receive from the model a best operation from the multiple schedulable operations selected by the model based on the current state to progress to the next state , and simulate execution of the best operation to simulate progression to the next state", wherein Chau discloses the schedulable operations (the corresponding planning factors in the timetable) which these operations are executed one by one to progress to a next state is functionally equivalent to the claimed invention);
the reward determining includes determining the value of the corresponding reward based on the reward function and a final state of the timetable in which all the plurality of planning factors are arranged; the time schedule corresponds to the final state of the timetable (Chau, Pg18, Claim22, "train the model to recommend the best operations as the plurality of operations in response to the tool progressing through the plurality of states when processing the semiconductor substrate in the plurality of processing chambers according to the recipe"
Pg11, Paragraph166, "For example, the reinforcement learning method used by the model 1204 can include Q - learning. ... Q - learning maximizes the expected value of the total reward over all successive steps , starting from the current state", wherein Chau discloses the reinforcement learning uses Q-learning that maximizes the expected value of total reward which the model is trained based on the accumulated expected value of the total reward outputted by the learning that completes the final scheduling through the plurality of states (the corresponding final state of the timetable in which the planning factors are arranged as it goes through multiple states and it will eventually reach the final state), rendering it functionally equivalent to the claimed invention.);
Chau teaches about the amount of time taken according to the time schedule and the value of corresponding reward as mentioned above. However, Chau does not teach changing a reward function that defines a relationship between them. From the same field of endeavor, Viquerat teaches the limitation of changing a reward function (Viquerat, Pg8, Section3.2 Reward Shaping, Paragraph1, "It can be observed on figure 7b that the learning process requires a considerable amount of explored shapes to converge toward its final performance level. As could be expected, this number of shapes increases with the number of degrees of freedom involved in the shape generation. In this section, we show that basic reward shaping is enough, in our case, to cut that number by a considerable amount. To do so, the reward is computed following equations (7) and (8), after which it is multiplied by a constant if it is positive, as shown in figure 8a, following:
PNG
media_image1.png
34
365
media_image1.png
Greyscale
", wherein the reward shaping is used for faster convergence and to increase the overall learning speed. Thus, if this reward function uses the relationship mentioned above, it is functionally equivalent to the claimed invention of changing a reward function)
Viquerat further teaches
the changing includes changing the reward function whose gradient in a partial section is larger than a gradient in the partial section of the reward function before the changing, the partial section being part of a time range from an amount of time taken according to a time schedule corresponding to a reward with a maximum value to an amount of time taken according to a time schedule corresponding to a reward with a minimum value (Viquerat, Pg7, Section3.1 Baseline results, Lines18-20, "In all cases, learning happens almost immediately, and continues almost linearly before reaching a plateau, after which the agent keeps exploring the environment, but no learning is specifically visible"
Pg8, Section3.2 Reward Shaping, Paragraph1, "It can be observed on figure 7b that the learning process requires a considerable amount of explored shapes to converge toward its final performance level. As could be expected, this number of shapes increases with the number of degrees of freedom involved in the shape generation. In this section, we show that basic reward shaping is enough, in our case, to cut that number by a considerable amount. To do so, the reward is computed following equations (7) and (8), after which it is multiplied by a constant if it is positive, as shown in figure 8a, following:
PNG
media_image1.png
34
365
media_image1.png
Greyscale
",
PNG
media_image2.png
256
276
media_image2.png
Greyscale
, wherein the baseline, r_t, in figure 8a applies the larger slope to a particular point where r_t > 0 (the corresponding partial section) by multiplying a constant 2, which makes the gradient becomes 2 instead of 1 for faster convergence as an agent or the model finds the shaped section significantly rewarding or the agent will learn accordingly to earn higher reward over the original lower reward (the corresponding minimum value and maximum value) so it can reduce or avoid the stagnation time. This is mathematically and functionally equivalent to the claimed invention of applying larger gradient to the particular section so that the model can learn accordingly to earn higher or maximum reward to avoid the plateau.)
Chau and Viquerat are analogous to the claimed invention as they are from the same field of endeavor of deep reinforcement learning based optimization methods for complex engineering systems. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the reinforcement learning-based scheduling and tool state-transition simulation method of Chau with the reward shaping technique of Viquerat that alters the gradient of the reward function in a partial section by scaling it with a constant factor. The motivation is as recited by Viquerat (Viquerat, Pg8, Section3.2.1, Lines4-5, "In this section, we show that basic reward shaping is enough, in our case, to cut that number by a considerable amount" and Pg9, Figure8, "Using a shaped reward increases the overall learning speed", Pg1, Abstract, Lines9-12, "the optimization process itself is agnostic to details of the use case, and thus our work paves the way to new generic shape optimization strategies both in fluid mechanics, and more generally in any domain where a relevant reward function can be defined") such that by sharpening the gradient of the reward function in a specific section, it would provide a much more distinct and pronounced learning signal to the scheduling agent. This mathematical adjustment directly prevents the reinforcement learning model from stagnating in uninformative plateaus where no learning is specifically visible, thereby maximizing the eploration efficiency and significantly cutting down the number of simulated episodes required to establish the optimum scheduling parameters for the substrate processing components.
As to dependent Claim 2,
The combination of Chau and Viquerat teaches, as mentioned above, all the limitations of Claim 1. The combination teaches about the overall technique being implemented to avoid the stagnation time of scheduling the operations of substrates. It implements the dynamic reward function such that it changes its gradient sharper and larger as it iterates of the reinforcement learning. It changes the gradients not only in one partial section but it shifts around different sections of window.
Chau, however, does not teach the following limitations but from the same field of endeavor, Viquerat further teaches
the schedule creation program generating method according to claim 1, wherein
the changing includes changing to a reward function whose gradient in the partial section is larger than a gradient in the partial section of a linear function that is set in all the time range (Viquerat, Pg7, Section3.1 Baseline results, Lines18-20, "In all cases, learning happens almost immediately, and continues almost linearly before reaching a plateau, after which the agent keeps exploring the environment, but no learning is specifically visible"
Pg8, Section3.2 Reward Shaping, Paragraph1, "It can be observed on figure 7b that the learning process requires a considerable amount of explored shapes to converge toward its final performance level. As could be expected, this number of shapes increases with the number of degrees of freedom involved in the shape generation. In this section, we show that basic reward shaping is enough, in our case, to cut that number by a considerable amount. To do so, the reward is computed following equations (7) and (8), after which it is multiplied by a constant if it is positive, as shown in figure 8a, following:
PNG
media_image1.png
34
365
media_image1.png
Greyscale
",
PNG
media_image2.png
256
276
media_image2.png
Greyscale
, wherein the baseline in the figure 8a is linear function from -5 which has slope of 1 (the corresponding linear function) which the reward shaping is applied from 0 by applying multiplication of 2 to the slope, which the gradient is larger, which is mathematically and functionally equivalent to the claimed invention.)
As to dependent Claim 5,
The combination of Chau and Viquerat teaches, as mentioned above, all the limitations of Claim 1. The combination teaches about the overall technique being implemented to avoid the stagnation time of scheduling the operations of substrates. It implements the dynamic reward function such that it changes its gradient sharper and larger as it iterates of the reinforcement learning. It changes the gradients not only in one partial section but it shifts around different sections of window.
Chau further teaches
the schedule creation program generating method according to claim 1, wherein the plurality of components include
a plurality of substrate processing sections that process the plurality of substrates (Chau, Pg6, Paragraph91, Lines2-4, "The substrate processing tool 100 includes a plurality of processing chambers 104-1 , 104-2 , . . . , and 104 - M"), and
a conveyance section that conveys the plurality of substrates (Chau, Pg6, Paragraph92, Lines7-11, “For example, a transfer robot 112 is arranged to transfer substrates from loading stations 116 to airlocks , or load locks , 120 , and a vacuum transfer robot 124 of a vacuum transfer module 128 is arranged to transfer substrates from the load locks 120 to the various processing chambers”,
Pg6, Paragraph93, Lines3-5, "The transport controller 134 control robots 112 and 124 , actuators and sensors related to the transportation of substrates to and from the substrate processing tool 100", wherein the conveyance section is simply a transportation section which the robot arms are designed to pick substrates on the section which Chau also discloses the similar architecture, rendering it functionally equivalent to the claimed invention),
wherein the plurality of planning factors include
a first planning factor that is a plan to load a substrate of the plurality of substrates into a substrate processing section of the plurality of substrate processing sections through the conveyance section to process the substrate through the substrate processing section (Chau, Pg20, Claim53, "a first plurality of neural networks configured to predict processing times for processing the semiconductor substrates in the plurality of processing chambers, respectively"
Pg6, Paragraph93, Lines3-5, "The transport controller 134 control robots 112 and 124, actuators and sensors related to the transportation of substrates to and from the substrate processing tool 100", wherein the first networks predicting such times for processing the substrates in the chambers will inherently go through the conveyance section as it these substrates must be loaded in to the chambers such that the this prediction is functionally equivalent to the first planning factor to loading substrates to the processing section for the process of substrates), and
a second planning factor that is a plan to unload the substrate processed by the substrate processing section from the substrate processing section through the conveyance section (Chau, Pg20, Claim53, "a second plurality of neural networks configured to predict transfer times for first and second robots, respectively, wherein the first and second robots are respectively configured to transfer the semiconductor substrates into the semiconductor processing tool and between the plurality of processing chambers; and a third neural network coupled to the first and second plurality of neural networks and configured to predict the optimum route for routing the semiconductor substrates between the plurality of processing chambers and to predict the optimum time to schedule the additional semiconductor substrates for processing in the semiconductor processing tool", wherein these second and third networks are used to predict times to remove the substrates that have been processed out of the chambers. This prediction inherently includes times needed to transfer the processed substrates so unprocessed substrates can be loaded, rendering it functionally equivalent to the claimed invention.)
As to dependent Claim 6,
The combination of Chau and Viquerat teaches, as mentioned above, all the limitations of Claim 5. It teaches about the physical limitations of the architecture where the processing sections are occupied when the substrates are being processed and the architecture including the conveyance section. Also, it teaches about the two planning factors that play important roles in scheduling the operations which are the times of loading the substrates to process them and unloading the substrates that have been processed through the conveyance section.
Chau further teaches
the schedule creation program generating method according to claim 5, wherein an amount of time taken according to the first planning factor is larger than an amount of time taken according to the second planning factor (Chau, Pg12, Paragraph175, Lines9-15, “For a two - layer processing recipe , the wafers entering the tool will first transfer to a preprocessing module , then to a first processing module , then to a second processing module , and then to a post - processing module , and then the wafers exit the tool ; and so on . As can be appreciated, recipes with more processing layers can have longer processing times”,
Pg13, Paragraph188, Lines8-14, “Program execution time or processing time is an amount of time a processing module takes to complete processing wafers. Transfer time for a robot is an amount of time a robot takes to move wafers from point A to point B (e.g., from one processing module to another or from an airlock to a processing module, and from a loading station of the tool to an airlock)”
Pg20, Claim53, "a first plurality of neural networks configured to predict processing times for processing the semiconductor substrates in the plurality of processing chambers, respectively; a second plurality of neural networks configured to predict transfer times for first and second robots, respectively, wherein the first and second robots are respectively configured to transfer the semiconductor substrates into the semiconductor processing tool and between the plurality of processing chambers", wherein Chau discloses both processing times and transfer times (the corresponding first and second factors) such that it is inherent physical property that the processing of substrates takes longer than transferring the substrates from point A to B using robot arms, rendering it functionally equivalent to the claimed invention)
As to dependent Claim 7,
The combination of Chau and Viquerat teaches, as mentioned above, all the limitations of Claim 5. It teaches about the physical limitations of the architecture where the processing sections are occupied when the substrates are being processed and the architecture including the conveyance section. Also, it teaches about the two planning factors that play important roles in scheduling the operations which are the times of loading the substrates to process them and unloading the substrates that have been processed through the conveyance section.
Chau further teaches
the schedule creation program generating method according to claim 5, wherein
the plurality of planning factors includes respective planning factors corresponding to the plurality of substrate processing sections, and the arranging includes selecting a planning factor of the plurality of planning factors respectively corresponding to the plurality of substrate processing sections (Chau, Pg20, Claim53, "a first plurality of neural networks configured to predict processing times for processing the semiconductor substrates in the plurality of processing chambers, respectively; a second plurality of neural networks configured to predict transfer times for first and second robots, respectively, wherein the first and second robots are respectively configured to transfer the semiconductor substrates into the semiconductor processing tool and between the plurality of processing chambers; and a third neural network coupled to the first and second plurality of neural networks and configured to predict the optimum route for routing the semiconductor substrates between the plurality of processing chambers and to predict the optimum time to schedule the additional semiconductor substrates for processing in the semiconductor processing tool", wherein the first networks are configured to predict times for processing substrates in the multiple processing chambers which corresponds to the planning factors for each processing section. The third networks are configured to predict optimum routes for processing substrates which inherently arranges the processing chambers to be occupied in the optimal way, rendering it functionally equivalent to the claimed invention of arranging substrates according the planning factors.)
As to independent Claim 8,
it is an apparatus claim that contains similar limitations of Claim 1 and thus rejected under the same rationale with the following additional limitation:
storage that stores a creation program that defines the schedule creation program generating method according to claim 1; and a processor that executes the creation program to generate the schedule creation program (Chau, Pg1, Paragraph6, Lines1-5, “A system for processing semiconductor substrates in a tool comprising a plurality of processing chambers configured to process the semiconductor substrates according to a recipe, comprises a processor and memory storing instructions for execution by the processor.”)
As to independent Claim 9,
it is an apparatus claim that contains similar limitations of Claim 1 and thus rejected under the same rationale with the following additional limitation:
storage that stores a schedule creation program that is created based on the schedule creation program generating method according to claim 1; and a processor that executes the schedule creation program to create the time schedule (Chau, Pg1, Paragraph6, Lines1-5, “A system for processing semiconductor substrates in a tool comprising a plurality of processing chambers configured to process the semiconductor substrates according to a recipe, comprises a processor and memory storing instructions for execution by the processor.”)
As to independent Claim 10,
it is a non-transitory computer-readable medium claim that contains similar limitations of Claim 1 and thus rejected under the same rationale.
As to independent Claim 11,
it is a non-transitory computer-readable medium claim that contains similar limitations of Claim 1 and thus rejected under the same rationale.
As to independent Claim 12,
it is an apparatus claim that contains similar limitations of Claim 1 and thus rejected under the same rationale with the following additional limitation:
storage that stores a creation program that defines the schedule creation program generating method according to claim 1, and a processor that executes the creation program to generate a schedule creation program for creating the time schedule for the plurality of components, the processor executing the schedule creation program to create the time schedule (Chau, Pg1, Paragraph6, Lines1-5, “A system for processing semiconductor substrates in a tool comprising a plurality of processing chambers configured to process the semiconductor substrates according to a recipe, comprises a processor and memory storing instructions for execution by the processor.”)
As to independent Claim 13,
it is an apparatus claim that contains similar limitations of Claim 1 and thus rejected under the same rationale with the following additional limitation:
storage that stores a schedule creation program that is generated through the schedule creation program generating method according to claim 1, and a processor that executes the schedule creation program to create the time schedule for the plurality of components (Chau, Pg1, Paragraph6, Lines1-5, “A system for processing semiconductor substrates in a tool comprising a plurality of processing chambers configured to process the semiconductor substrates according to a recipe, comprises a processor and memory storing instructions for execution by the processor.”)
As to independent Claim 14,
it is a system claim that contains similar limitations of Claim 8 and thus rejected under the same rationale with the following additional limitations:
a substrate processing apparatus that processes a substrate (Chau, Pg(PDF)2, Figure 1),
a transmitter that transmits the schedule creation program to the substrate processing apparatus; a receiver that receives the schedule creation program transmitted from the transmitter of the schedule creation program generating apparatus (Chau, Pg9, Paragraph130, “The model can be implemented in many ways. For example, the model can be integrated into the tool's system software. Alternatively, the model can be implemented independently and remotely from the tool's system software, and the prediction results generated by the model can be supplied to the tool's system software. For example, for ease of model maintenance, the model may be run outside of the system software of a tool. The model can receive input parameters from the system software based on a wafer – flow selected by the operator. The model can then compute and predict the best scheduling parameter values and send them back to the system software. For example, the model can be deployed in a cloud as a software-as-a-service”), and
a processor that executes the schedule creation program to create the time schedule for the plurality of components (Chau, Pg1, Paragraph6, Lines1-5, “A system for processing semiconductor substrates in a tool comprising a plurality of processing chambers configured to process the semiconductor substrates according to a recipe, comprises a processor and memory storing instructions for execution by the processor.”)
As to independent Claim 15,
it is a system claim that contains similar limitations of Claim 8 and thus rejected under the same rationale with the following additional limitations:
a substrate processing apparatus that processes a substrate (Chau, Pg(PDF)2, Figure 1),
a transmitter that transmits the time schedule to the substrate processing apparatus; a receiver that receives the time schedule transmitted from the transmitter of the schedule creating apparatus (Chau, Pg9, Paragraph130, “The model can be implemented in many ways. For example, the model can be integrated into the tool's system software. Alternatively, the model can be implemented independently and remotely from the tool's system software, and the prediction results generated by the model can be supplied to the tool's system software. For example, for ease of model maintenance, the model may be run outside of the system software of a tool. The model can receive input parameters from the system software based on a wafer – flow selected by the operator. The model can then compute and predict the best scheduling parameter values and send them back to the system software. For example, the model can be deployed in a cloud as a software-as-a-service”), and
a processor that executes the schedule creation program to create the time schedule for the plurality of components (Chau, Pg1, Paragraph6, Lines1-5, “A system for processing semiconductor substrates in a tool comprising a plurality of processing chambers configured to process the semiconductor substrates according to a recipe, comprises a processor and memory storing instructions for execution by the processor.”)
Claims 3, 4 are rejected under 35 U.S.C. 103 as being unpatentable over Chau and Viquerat as mentioned in Claim 1, in further view of Luo et al. (Luo), Non-Patent Literature, “Accelerating Reinforcement Learning for Reaching Using Continuous Curriculum Learning”, published in July 2020, Pages: 8.
As to dependent Claim 3,
The combination of Chau and Viquerat teaches, as mentioned above, all the limitations of Claim 1. The combination teaches about the overall technique being implemented to avoid the stagnation time of scheduling the operations of substrates. It implements the dynamic reward function such that it changes its gradient sharper and larger as it iterates of the reinforcement learning. It changes the gradients not only in one partial section but it shifts around different sections of window.
From the combination, Viquerat teaches about the larger slope or the gradient in the partial section can lead to faster convergence and an efficient learning. However, both Chau and Viquerat do not teach the following limitations but from the same field of endeavor, Luo teaches
the schedule creation program generating method according to claim 1, wherein
the changing is repeated more than once (Luo, Pg5, Left Column, Paragraph2, Lines3-5, "changing the curriculum up to every episode. In this study, we refer to the parameter as the epoch t number for updating the policy"), and
the changing includes changing to a reward function whose gradient in the partial section is larger than a gradient in the partial section of an active reward function each time the changing is repeated (Luo, Pg5, Left Column, Paragraph3, Lines26-30, "If alpha is smaller than 1, the initial decaying process is slower with a smaller slope alpha. The decay function is a linear function when alpha is equal to 1. If alpha is larger than 1, then the larger of the slope, the faster the decay of the initial part of the precision behaves."
Pg5, Equation 11,
PNG
media_image3.png
41
299
media_image3.png
Greyscale
, wherein Luo discloses a power-based decay function to dictate the rate of change for the reward function across these repeated epochs. Specifically, when the control parameter alpha is set greater than 1, the larger of the slope, the faster the decay of the initial part of the precision behaves, which means a larger mathematical slope/gradient is dynamically applied to contract the reward boundaries as the training process is repeated, which is functionally equivalent to the claimed invention of changing the gradient of the reward function larger each time it is repeated)
Chau, Viquerat, and Luo are analogous to the claimed invention as they are from the same field of endeavor of deep reinforcement learning-based optimization and control frameworks for continuous physical/engineering environments. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the reinforcement learning-based scheduling and state-transition simulation method of Chau, the gradient-steepening reward shaping technique of Viquerat with the continuous curriculum learning framework of Luo that repeatedly shifts the reward function parameters across multiple training epochs. The motivation is as recited by Luo (Luo, Pg1, Abstract, Lines15-17, "a static training schedule is suboptimal, and using an appropriate decay function for curriculum learning provides superior results in a faster way") such that by replacing static reward mechanism with Luo's iterative curriculum framework, it can dynamically shift the target reward boundaries across repeated epochs rather than fixing them in a suboptimal static schedule. When this shifting framework is combined with Viquerat’s technique of applying a larger gradient/slope to the reward function at each progressive stage, the combined approach smooths the overall training process. This allows the substrate scheduling model to continuously adapt to sequentially higher difficulty levels, providing a distinct learning signal that prevents optimization stagnation and enables the agent to converge to optimum scheduling parameters in a significantly faster and superior manner.
As to dependent Claim 4,
The combination of Chau and Viquerat teaches, as mentioned above, all the limitations of Claim 1. The combination teaches about the overall technique being implemented to avoid the stagnation time of scheduling the operations of substrates. It implements the dynamic reward function such that it changes its gradient sharper and larger as it iterates of the reinforcement learning. It changes the gradients not only in one partial section but it shifts around different sections of window.
However, both Chau and Viquerat do not teach the following limitations but from the same field of endeavor, Luo teaches
the schedule creation program generating method according to claim 1, wherein the changing further includes shifting the partial section (Luo, Pg2, Right Column, Paragraph3, Lines2-4, "By changing the precision requirement ε up to every epoch, we add a new task to the curriculum, which can smooth the training process and is easy to implement"
Pg5, Left Column, Paragraph3, Lines17-26, "It contains four parameters: start and end precision (e_0 and e_m respectively), the number of epochs that a decay function should experience (s), and the monotonic reduction slope of the decay (alpha). The decay function is formalized as a power function (11): _eq11_ where k in [0,s] represents the current epoch number, s is the total number of epochs for the entire precision reduction process, and alpha in (0, infinity) is the slope of the decay function. The precision (ε) of the training process decreases gradually following the equation (11)
PNG
media_image3.png
41
299
media_image3.png
Greyscale
", wherein Luo teaches a continuous curriculum learning strategy where the boundary parameter ε of the reward function is gradually decreased and shifted up to every epoch (starting from precision e_0 to e_m) according to a power-based decay function (Equation11) to smooth the training process, which is functionally equivalent to the claimed invention of the changing reward function continuously occurs throughout multiple partial sections.)
Chau, Viquerat, and Luo are analogous to the claimed invention as they are from the same field of endeavor of deep reinforcement learning-based optimization and control frameworks for continuous physical/engineering environments. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the reinforcement learning-based scheduling and tool state-transition simulation method of Chau, the reward shaping technique of Viquerat that alters the gradient of the reward function in a partial section by scaling it with a constant factor with the continuous curriculum learning framework of Luo which repeatedly updates environment parameters up to every epoch rather than using a static schedule to prevent learning inefficiencies. The motivation is as recited by Luo (Luo, Pg1, Abstract, Lines15-17, "a static training schedule is suboptimal, and using an appropriate decay function for curriculum learning provides superior results in a faster way") such that shifting this partial section according to the agent's training epochs allows the reward function adaptively guide the scheduler through sequential difficulty levels, preventing optimization stagnation at each stage.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DONG YOON JUNG whose telephone number is (571)270-0198. The examiner can normally be reached 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached at (571) 272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DONG YOON JUNG/ Examiner, Art Unit 2145
/CESAR B PAULA/ Supervisory Patent Examiner, Art Unit 2145
Prosecution Insights