Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This action is in response to the amendments filed 02 February 2026. Claims 1-5 are amended. Claims 7-12 are withdrawn. Claims 13 and 14 are newly added. Claims 1-14 are pending and have been examined.
Response to Arguments
Applicant' s arguments, see page 9, filed 04 February 2026, with respect to the objection to Claim 2 for minor informalities have been fully considered and are persuasive. The the objection to Claim 2 for minor informalities has been withdrawn.
APPLICANT'S ARGUMENT: Applicant argues (page 9, paragraph 6) that "The amendments to claim 2 are believed to overcome the informalities, and the objection is believed to be overcome."
EXAMINER'S RESPONSE: The the objection to Claim 2 for minor informalities has been withdrawn.
Applicant' s arguments, see page 10, filed 04 February 2026, with respect to the rejection to Claims 1-5 under 35 U.S.C. 112(b) have been fully considered and are persuasive. The rejections to Claims 1-5 under 35 U.S.C. 112(b) have been withdrawn.
APPLICANT'S ARGUMENT: Applicant argues (page 10, paragraph 1) that "Claims 1-5 are amended to correct antecedent basis, and the rejection is believed to be overcome."
EXAMINER'S RESPONSE: The rejections to Claims 1-5 under 35 U.S.C. 112(b) have been withdrawn.
Applicant' s arguments, see pages 11-14, filed 04 February 2026, with respect to the rejections of Claims 1-4 and 6 under 35 U.S.C. 102(a) and Claim 5 35 U.S.C. 103 have been fully considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
APPLICANT'S ARGUMENT: Applicant argues (page 12, paragraph 3) that "Singh is directed to a different system from the proxy interpreter system of the present invention. Particularly, there is no teaching or suggestion in Singh that the reinforcement learning module uses a Duelling Double Deep Q Network implemented through two streams.... Furthermore, in comparison, the implementation of the reinforcement module of the proxy interpreter system of the present invention does not require training with simulations on training data as Singh does."
EXAMINER'S RESPONSE: Examiner notes that Applicant's arguments are moot. Amended Claims 1-4 and 6, and newly recited Claim 13, are rejected as obvious in view of Singh in view of Wu. Amended Claim 5 is rejected in view of Singh in view of Wu in view of Givon. Newly recited Claim 14 is rejected in view of Singh in view of Wang in view of Cronje in view of Cho in view of Wu.
Claim Objections
The objection of Claim 2 for minor informalities of the previous Office Action is withdrawn in light of arguments and/or amendments.
Claim Rejections - 35 USC § 112
The rejections of Claims 1-5 under 35 U.S.C. 112(b) of the previous Office Action have been withdrawn in light of arguments and/or amendments.
Claims 13 and 14 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 13 recites the limitation "machine state" in "to aid in quality improvement and fine tuning a quality value for a given machine state." Claim 13 does not introduce a machine prior to referring to the cited state, nor does independent Claim 1 introduce a machine. There is insufficient antecedent basis for this limitation in the claim. For the purposes of examination, Claim 13 has been interpreted to read: "to aid in quality improvement and fine tuning a quality value for a given system state" (emphasis added). Appropriate correction or clarification is required.
Claim 14 recites the limitation "the machine" in "any irrelevant state of the machine." Claim 14 does not introduce a machine prior to referring to the cited machine, nor does independent Claim 1 introduce a machine. There is insufficient antecedent basis for this limitation in the claim. For the purposes of examination, Claim 14 has been interpreted to read: "any irrelevant state of a system" (emphasis added). Appropriate correction or clarification is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-4, 6, and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Singh, et al. (US 2021/0110300 A1, hereinafter "Singh") in view of Wang, et al., "Dueling Network Architectures for Deep Reinforcement Learning" (hereinafter "Wang").
Regarding Claim 1, Singh teaches:
A proxy interpreter system controlling a legacy system (Singh, Fig. 1, depicting system 100 with robots 132 and 134 controlling the items of block AUTOMATE corresponding to the instant legacy system, as in [0027]: "FIG. 1 is an architectural diagram illustrating an RPA [robotic process automation] system 100, according to an embodiment of the present invention" and [0032]: "Both attended and unattended robots may automate various systems and applications including, but not limited to, mainframes, web applications, VMs, enterprise applications ..., and computing system applications (e.g., desktop and laptop applications, mobile device applications, wearable computer applications, etc.)"), using artificial intelligence (Singh, Fig. 1, 660, "Call Deployed ML Model at Runtime by RPA Robots") connected to a PC control system (Singh, Fig. 5, 505, "Bus" and [0054]: "FIG. 5 is an architectural diagram illustrating a computing system 500 configured to implement reinforcement learning in RPA.... Computing system 500 includes a bus 505 or other communication mechanism for communicating information, and processor(s) 510 coupled to bus 505 for processing information"), the proxy interpreter system comprising:
a server, communicatively coupled to the PC control system through multiple channels including inputs and outputs, which operates the legacy system (Singh, [0040]: "FIG. 2 is an architectural diagram illustrating a deployed RPA system 200, according to an embodiment of the present invention. In some embodiments, RPA system 200 may be, or may be a part of, RPA system 100 of FIG. 1. ... On the client side, a robot application 210 includes executors 212, an agent 214, and a designer 216. ... All messages in this embodiment are logged into conductor 230, which processes them further via database server 240, indexer server 250, or both. As discussed above with respect to FIG. 1, executors 212 may be robot components"), receives and sends operating commands through hardware interfaces (Singh, [0042]: "agent 214 may open a WebSocket channel that is later used by conductor 230 to send commands to the robot (e.g., start, stop, etc.)"), teaching sequences and respective responses with reference to an image displayed on a display terminal (Singh, [0049]: "Workflows may include user-defined activities 320 and UI automation activities 330. Some embodiments are able to identify non-textual visual components in an image, which is called computer vision (CV) herein. Some CV activities pertaining to such components may include, but are not limited to, click, type, get text, hover, element exists, refresh scope, highlight, etc."); and
a reinforcement learning module (Singh, Fig. 5, 545, "Reinforcement Learning Module") arranged to learn and store the operating commands and the sequences ... and to create an operating flow for the operating commands (Singh, [0017]: "Some embodiments pertain to reinforcement learning in RPA. In reinforcement learning, there is an exploration phase. Process mining may provide the various states of a system (e.g., by deploying listeners on computing systems and collecting information pertaining to actions taken by users, robots, or both, on those computing systems, such as clicking buttons, opening/closing applications, entering text, etc.)" and [0019]: "Some embodiments employ a policy network, which tweaks and defines the reward function. In some embodiments, this may be accomplished by the system watching actions taken by a human. ... This helps the robot get closer to the reward function goal. In other words, the policy network informs the robot whether it is getting closer to the winning state") ... as part of a process triggered by an operator when setting up and configuring the legacy system (Singh, [0033]: "Conductor 120 may have various capabilities including, but not limited to, provisioning, deployment, versioning, configuration, queueing, monitoring, logging, and/or providing interconnectivity. Provisioning may include creating and maintenance of connections between robots 130 and conductor 120 (e.g., a web application). Deployment may include assuring the correct delivery of package versions to assigned robots 130 for execution. ... Configuration may include maintenance and delivery of robot environments and process configurations. ... Conductor 120 may provide interconnectivity by acting as the centralized point of communication for third-party solutions and/or applications"), wherein the reinforcement learning module uses a ... Network (Singh, [0009]: "a computer-implemented method for training an ML model for RPA using reinforcement learning includes training the ML model using a policy network. The ML model has a reward function") implemented through two streams, and the two streams comprise a ... stream for learning common quality value of each operating state of the legacy system (Singh, [0020]: "To get closer to the winning state, the robot could analyze the underlying distribution. If results are falling in the same range ( e.g., within half a standard deviation, one standard deviation, etc.), this provides information regarding the performance of the robot. The analysis could be a lookup function, a statistical distribution, etc.") and an ... stream for learning the corresponding action for a given state of the legacy system (Singh, [0019]: "Some embodiments employ a policy network, which tweaks and defines the reward function. In some embodiments, this may be accomplished by the system watching actions taken by a human")
Singh teaches a reinforcement learning module arranged to learn and store the operating commands and the sequences as part of a process triggered by an operator when setting up and configuring the legacy system, wherein the reinforcement learning module uses a Network implemented through two streams.
Singh does not explicitly teach wherein the reinforcement learning module uses a Duelling Double Deep Q Network implemented through two streams, and the two streams comprise a value stream for learning common quality value of each operating state ... and an advantage stream for learning the corresponding action for a given state.
However, Wang teaches:
wherein the reinforcement learning module uses a Duelling Double Deep Q Network implemented through two streams (Wang, p. 1, 1. Introduction: "we take an alternative but complementary approach of focusing primarily on innovating a neural network architecture that is better suited for model-free RL [reinforcement learning]. ... ¶ The proposed network architecture, which we name the dueling architecture, explicitly separates the representation of state values and (state-dependent) action advantages. The dueling architecture consists of two streams that represent the value and advantage functions, while sharing a common convolutional feature learning module. The two streams are combined via a special aggregating layer to produce an estimate of the state-action value function Q" and 2.2. Double Deep Q-networks: "we use the improved Double DQN (DDQN) [Double Deep Q-networks] learning algorithm of van Hasselt et al. (2015)"), and the two streams comprise a value stream for learning common quality value of each operating state ... and an advantage stream for learning the corresponding action for a given state ... (Wang, p. 4, 3. The Dueling Network Architecture: "To bring this insight to fruition, we design a single Q-network architecture, as illustrated in Figure 1, which we refer to as the dueling network. ... [I]nstead of following the convolutional layers with a single sequence of fully connected layers, we instead use two sequences (or streams) of fully connected layers. The streams are constructed such that they have they have the capability of providing separate estimates of the value and advantage functions. Finally, the two streams are combined to produce a single output Q function" and "we need to keep in mind that
Q
s
,
a
;
θ
,
α
,
β
is only a parameterized estimate of the true Q-function. Moreover, it would be wrong to conclude that
V
s
;
θ
,
β
is a good estimator of the state-value function, or likewise that
A
s
,
a
;
θ
,
α
provides a reasonable estimate of the advantage function. ¶ ... we let the last module of the network implement the forward mapping
Q
s
,
a
;
θ
,
α
,
β
=
V
s
;
θ
,
β
+
A
s
,
a
;
θ
,
α
-
max
a
'
∈
A
A
s
,
a
'
;
θ
,
α
(8)
... Hence, the stream
V
s
;
θ
,
β
provides an estimate of the value function, while the other stream produces an estimate of the advantage function").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Singh regarding a reinforcement learning module arranged to learn and store the operating commands and the sequences as part of a process triggered by an operator when setting up and configuring the legacy system, wherein the reinforcement learning module uses a Network implemented through two streams with those of Wang regarding wherein the reinforcement learning module uses a Duelling Double Deep Q Network implemented through two streams, and the two streams comprise a value stream for learning common quality value of each operating state and an advantage stream for learning the corresponding action for a given state.
The motivation to do so would be to facilitate improvement of existing reinforcement learning algorithms for model-free learning where redundant or similar actions are added to the learning problem (Wang, p. 1, 1. Introduction: "we take an alternative but complementary approach of focusing primarily on innovating a neural network architecture that is better suited for model-free RL. This approach has the benefit that the new network can be easily combined with existing and future algorithms for RL. That is, this paper advances a new network (Figure 1), but uses already published algorithms" and p. 2, 1. Introduction: "we demonstrate that the dueling architecture can more quickly identify the correct action during policy evaluation as redundant or similar actions are added to the learning problem").
Regarding Claim 2, the rejection of Claim 1 is incorporated. The Singh/Wang combination teaches:
wherein the hardware interfaces to operate the legacy system through the PC control system including input/output ports, USB ports, ethernet port, VGA port, mouse, keyboard and display interface (Singh, Fig. 5, depicting display 525, keyboard 530, and cursor control device 535, and [0058]: "A keyboard 530 and a cursor control device 535, such as a computer mouse, a touchpad, etc., are further coupled to bus 505 to enable a user to interface with computing system 500" and [0056]: "computing system 500 includes a communication device 520, such as a transceiver, to provide access to a communications network via a wireless and/or wired connection" and [0057]: "Processor(s) 510 are further coupled via bus 505 to a display 525, such as a plasma display, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display ... or any other suitable display for displaying information to a user," where Singh's system interfaces and peripherals reasonably suggest USB, VGA, and ethernet ports) are utilised to learn and create domain knowledge required to operate the legacy system (Singh, [0026]: "In some embodiments, robots may utilize reinforcement learning to automatically build policy networks themselves. These policy networks may be built based on observation of human interactions with the computing system in a given scenario (e.g., which APis are called)" and [0022]: "the policy network may be used to first determine whether sending an email is even possible or desired. Once this determination is made, the policy network may then determine whether the email addresses from the spreadsheet and the email body text can be copied into the email before sending").
Regarding Claim 3, the rejection of Claim 2 is incorporated. The Singh/Wang combination teaches:
wherein the proxy interpreter system resides as a software module within the PC control system that is controlling the legacy system and utilise the hardware interfaces to learn and create the domain knowledge required to operate the legacy system (Singh, Figs. 4 and 5, and [0059]: "Memory 515 stores software modules that provide functionality when executed by processor(s) 510. The modules include an operating system 540 for computing system 500. The modules further include a reinforcement learning module 545 that is configured to perform all or part of the processes described herein or derivatives thereof").
Regarding Claim 4, the rejection of Claim 2 is incorporated. The Singh/Wang combination teaches:
wherein the proxy interpreter system accumulates the domain knowledge from interactions with the legacy system through commands and responses monitored through the hardware interfaces for all operating states of the legacy system (Singh, [0017]: "Some embodiments pertain to reinforcement learning in RPA. In reinforcement learning, there is an exploration phase. Process mining may provide the various states of a system (e.g., by deploying listeners on computing systems and collecting information pertaining to actions taken by users, robots, or both, on those computing systems, such as clicking buttons, opening/closing applications, entering text, etc.)").
Regarding Claim 6, the rejection of Claim 4 is incorporated. The Singh/Wang combination teaches:
wherein the domain knowledge created through the application of deep learning techniques and continuous reinforced learning, is subsequently utilised to operate the legacy system without the intervention of a human operator (Singh, [0058]: "A keyboard 530 and a cursor control device 535, such as a computer mouse, a touchpad, etc., are further coupled to bus 505 to enable a user to interface with computing system 500. ... [T]he user may interact with computing system 500 remotely via another computing system in communication therewith, or computing system 500 may operate autonomously").
Regarding Claim 13, the rejection of Claim 1 is incorporated.
Wang further teaches:
further comprising an experience buffer (Wang, p. 11, A. Double DQN Algorithm, line 9, "add transition tuple
s
,
a
,
r
,
s
'
to
D
," where
D
is a replay buffer, and where the tuple is added each step for each episode, and p. 4.2. General Atari Game-Playing: "We train the dueling network with the DDQN algorithm as presented in Appendix A") from which the advantage stream is regularly updated (Wang, p. 11, A. Double DQN Algorithm, line 16, "Replace target parameters
θ
-
←
θ
every
N
-
steps," where
θ
are parameters for the advantage stream
A
, as in p. 4, 3. The Dueling Network Architecture: "Let us consider the dueling network shown in Figure 1, where we make one stream of fully-connected layers output a scalar
V
s
;
θ
,
β
, and the other stream output an
A
-dimensional vector
A
s
,
a
;
θ
,
α
") to aid in quality improvement (Wang, p. 4, 2.3. Prioritized Replay: "To strengthen the claim that our dueling architecture is complementary to algorithmic innovations, we show that it improves performance for both the uniform and the prioritized replay baselines (for which we picked the easier to implement rank-based variant), with the resulting prioritized dueling variant holding the new state-of-the-art") and fine tuning a quality value (Wang, p. 7, 4.2. General Atari Game-Playing, Combining with Prioritized Experience Replay: "So in our final experiment, we investigate the integration of the dueling architecture with prioritized experience replay. We use the prioritized variant of DDQN (Prior. Single) as the new baseline algorithm, which replaces with the uniform sampling of the experience tuples by rank-based prioritized sampling") for a given machine state (Wang, p. 11, A. Double DQN Algorithm, line 15, "Do a gradient descent step with loss," where the gradient update for model parameters occurs for each step for each episode).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Singh/Wang combination regarding the reinforcement learning module using a Duelling Double Deep O Network implemented through two streams comprising an advantage stream with the further teachings of Wang regarding an experience buffer from which the advantage stream is regularly updated to aid in quality improvement and fine tuning a quality value for a given machine state.
The motivation to do so would be to facilitate improved data efficiency during model update steps of model training (Wang, p. 3, 2.1. Deep Q-networks: "Another key ingredient behind the success of DQN is experience replay.... Experience replay increases data efficiency through re-use of experience samples in multiple updates and, importantly, it reduces variance as uniform sampling from the replay buffer reduces the correlation among the samples used in the update").
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Singh, et al. (US 2021/0110300 A1, hereinafter "Singh") in view of Wang, et al., "Dueling Network Architectures for Deep Reinforcement Learning" (hereinafter "Wang") in view of Givon (US 2019/0138635 A1, hereinafter "Givon").
Regarding Claim 5, the rejection of Claim 4 is incorporated. The Singh/Wang combination teaches:
wherein commands and responses stored in a recipe ... (Singh, [0047]: "The persistence layer includes a pair of servers in this embodiment -- database server 240 ... and indexer server 250. Database server 240 in this embodiment stores the configurations of the robots, robot groups, associated processes, users, roles, schedules, etc.," where Singh's stored configuration corresponds to the instant stored recipe, and where Singh's configurations pertain to business process and environment, as in [0033]: "Configuration may include maintenance and delivery of robot environments and process configurations" where a process pertains to [0028]: "The automation project enables automation of rule-based processes by giving the developer control of the execution order and the relationship between a custom set of steps developed in a workflow") for a specific device type (Singh, [0032]: "Both attended and unattended robots may automate various systems and applications including, but not limited to, mainframes, web applications, VMs, enterprise applications ..., and computing system applications (e.g., desktop and laptop applications, mobile device applications, wearable computer applications, etc.)") are acquired from keyboard commands and mouse movements made by the operator with reference to the image on the display terminal (Singh, [0049]: "a developer uses designer 310 to develop workflows that are executed by robots. Workflows may include user-defined activities 320 and UI automation activities 330. Some embodiments are able to identify non-textual visual components in an image, which is called computer vision (CV) herein. Some CV activities pertaining to such components may include, but are not limited to, click, type, get text, hover, element exists, refresh scope, highlight, etc."), combined with relevant responses received on the ethernet port and the input/output ports from the legacy system to the proxy interpreter system (Singh, [0056]: "computing system 500 includes a communication device 520, such as a transceiver, to provide access to a communications network via a wireless and/or wired connection," where Singh's wired network communications corresponds to the instant ethernet) to create the domain knowledge for the legacy system (Singh, [0019]: "Some embodiments employ a policy network, which tweaks and defines the reward function. In some embodiments, this may be accomplished by the system watching actions taken by a human. If the human corrects the operation by taking an action on a computing system, whether knowing that he or she is interacting with a robot or not, the policy network may learn from this action and incorporate it into the reward function. This helps the robot get closer to the reward function goal," where Singh's operations, actions, and goals correspond to the instant domain knowledge, as in [0003]: "random actions guided by current knowledge are employed in an attempt to get closer to an objective function (i.e., a 'winning' state)").
The Singh/Wang combination teaches a proxy interpreter system wherein commands and responses are stored in a persisted recipe.
Singh does not explicitly teach a recipe file.
However, Givon teaches:
a recipe file (Givon, [0058]: "some agent applications may be deployed upon system initiation, substantially automatically. The system includes an initialization module to, upon initialization of said system, cause said deployment controller to deploy one or more agent applications by default. The initialization module may include basic sensor feed type identification functionality, and may cause deployment of agent applications whose functions include sensor feed access and monitoring to support automatic system self-configuration. According to embodiments where the systems has been configured and is being initiated as a part of a system restart or reboot, the initiation module may cause the deployment module to deploy agent applications in accordance to configuration instructions stored in a system configuration file or con data structure").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Singh regarding a proxy interpreter system wherein commands and responses are stored in a persisted recipe with those of Givon regarding a recipe file.
The motivation to do so would be to ensure proper initialization of a deployed agent for its task in a given environment and device configuration (Givon, [0022]: "Initial agent application deployment according to embodiments of the present invention may include: ... following deployment instructions associated with each selected template, as constrained by the system's physical configurations (i.e. processing platform resources, network addresses, process ID's, etc.), to properly install and interconnect each selected template's corresponding agent application").
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Singh, et al. (US 2021/0110300 A1, hereinafter "Singh") in view of Wang, et al., "Dueling Network Architectures for Deep Reinforcement Learning" (hereinafter "Wang") in view of Cronje (US 2021/0248400 A1, hereinafter "Cronje") in view of Cho, et al. (US 2021/0318252 A1, hereinafter "Cho") in view of Wu, et al. (US 2020/0364485 A1, hereinafter "Wu").
Regarding Claim 14, the rejection of Claim 1 is incorporated.
Wang further teaches:
a deep learning module to enhance quality ... (Wang, p. 1, 1. Introduction: "Over the past years, deep learning has contributed to dramatic advances in scalability and performance of machine learning (LeCun et al., 2015). One exciting application is the sequential decision-making setting of reinforcement learning (RL) and control. Notable examples include deep Q-learning ... ¶ ... Here, we take an alternative but complementary approach of focusing primarily on innovating a neural network architecture that is better suited for model-free RL" and p. 4, 2.3. Prioritized Replay: "To strengthen the claim that our dueling architecture is complementary to algorithmic innovations, we show that it improves performance for both the uniform and the prioritized replay baselines"), the deep learning module comprising:
a ... model ... to ensure no action is taken by an action classifier for any irrelevant state of the machine (Wang, p. 11, A. Double DQN Algorithm, line 9, "Set
s
'
←
x
, and add transition tuple
s
,
a
,
r
,
s
'
to
D
, replacing the oldest tuple if
D
≥
N
r
," where Wang's oldest tuple corresponds to the instant irrelevant state, and line 13, "Define
a
m
a
x
s
'
;
θ
=
arg
max
a
'
Q
s
'
,
a
'
;
θ
," where Wang's argmax reasonably suggests an action classifier, and where the sampled mini-batch of possible actions cannot have the irrelevant state) ... frozen model generated (Wang, p. 11, A. Double DQN Algorithm, where Wang's "
θ
– initial network parameters" corresponds to the instant frozen model) and constantly updated ... by assigning a confidence vector (Wang, p. 11, A. Double DQN Algorithm, line 9, "add transition tuple
s
,
a
,
r
,
s
'
to
D
," where Wang's algorithm variant using a prioritized-replay buffer corresponds to the instant confidence vector, as in 2.3. Prioritized Replay: "Their key idea was to increase the replay probability of experience tuples that have a high expected learning progress ... ¶ ... [W]e show that it improves performance for both the uniform and the prioritized replay baselines (for which we picked the easier to implement rank-based variant), with the resulting prioritized dueling variant holding the new state-of-the-art," where Wang's high expected learning progress corresponds to the instant confidence).
a set of feature maps ... for image (Wang, p. 8, 4.2. General Atari Game-Playing, Saliency maps: "to visualize the salient part of the image as seen by the value stream, we compute the absolute value of the Jacobian of
V
^
with respect to the input frames.... Similarly, to visualize the salient part of the image as seen by the advantage stream.... Both quantities are of the same dimensionality as the input frames and therefore can be visualized easily alongside the input frames. ¶ ... As observed in the introduction, the value stream pays attention to the horizon where the appearance of a car could affect future performance. The value stream also pays attention to the score. The advantage stream, on the other hand, cares more about cars that are on an immediate collision course," where Wang's game screen saliency maps reasonably suggest image feature maps, given p. 4, 3. The Dueling Network Architecture: "The lower layers of the dueling network are convolutional as in the original DQNs").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Singh/Wang combination regarding a reinforcement learning module implemented through two streams comprising a value stream and an advantage stream with the further teachings of Wang regarding a deep learning module to enhance quality, a frozen model generated and constantly updated to ensure action is taken by an action classifier, and a set of feature maps for image.
The motivation to do so would be to facilitate development of models that learn to perform tasks involving control with immediate and distant visual ranges, and to do so with faster learning convergence (Wang, p. 5, 4.1. Policy evaluation: "In the dueling network, the stream
V
s
;
θ
,
β
learns a general value that is shared across many similar actions at s, hence leading to faster convergence. This is a very promising result be cause many control tasks with large action spaces have this property, and consequently we should expect that the dueling network will often lead to much faster convergence than a traditional single stream network" and p. 8, 4.2. General Atari Game-Playing, Saliency maps: "the value stream pays attention to the horizon where the appearance of a car could affect future performance. The value stream also pays attention to the score. The advantage stream, on the other hand, cares more about cars that are on an immediate collision course").
The Singh/Wang combination teaches a deep learning module to enhance quality, a frozen model generated and constantly updated to ensure action is taken by an action classifier, and a set of feature maps for image.
The Singh/Wang combination does not explicitly teach comprising a deep learning module to enhance quality of ... inspection ..., the deep learning module comprising: a frozen model generated and constantly updated through using an object detection model ...; a custom built LSTM (Long short term memory) model to distinguish between similar images in different ... states; and ... using a modified YOLO (You Only Look Once) ... for image.
However, Cronje teaches:
comprising a deep learning module (Cronje, [0076]: "The processor (not shown) is configured to run algorithms (450) to implement a set of convolutional neural networks (CNNs) including" and [0092]: "State of the art, deep convolutional neural networks are used") to enhance quality of ... inspection ... (Cronje, [0094]: "Feature-based, single-shot and multi-shot classifications are ensembled ( combined) to create a more accurate model for behavior classification," where Cronje's classification is performed for driver inspection, as in [0079]: "a classifier group ( 456) which assess the facial features received from the facial feature extraction group in combination with objects detected by the object detection group to classify predefined operator behaviors"), the deep learning module comprising:
a frozen model generated and constantly updated through using an object detection model ... (Cronje, [0070] The CNN model to be trained is initialized in different ways. ... A pre-trained model (322) can also be used for initialization, this method is known as transferred-learning" and [0026]: "The classifier group may include an ensemble function to ensemble the outputs of the classifiers together with the output of the single image CNN of the operator together with the combination of the single image CNN and the LSTM recurrent network by a weighted sum of the three classifiers where the weights are determined by optimizing the weights on the training dataset. The ensembled output from the classifiers is used to determine the operator behavior," where Cronje's pre-trained model corresponds to the instant frozen model);
a custom built LSTM (Long short term memory) model (Cronje, [0068]: "For LSTM networks, a stream of multiple images is created for training") to distinguish between similar images in different ... states (Cronje, [0026]: "The classifier group may include an ensemble function to ensemble the outputs of the classifiers together with ... the LSTM recurrent network.... The ensembled output from the classifiers is used to determine the operator behavior" and [0067]: "The individual results are combined by a weighted sum where the weights are determined by optimizing the weights on the training dataset, to arrive at a final operator state (289)"); and
... using a modified YOLO (You Only Look Once) ... for image... (Cronje, [0059]: "A detection CNN takes an image as input and outputs the bounding region in 2-dimensional image coordinates for each class detected. ... Standard object detector CNN architectures exist such as You Only Look Once (YOLO)" and [0070]: "The CNN model to be trained is initialized in different ways. ... A pre-trained model (322) can also be used for initialization, this method is known as transferred-learning. The pre-trained model (322) is a model previously trained on a totally different subject matter and the training performed will fine-tune the weights for the specific task," which reasonably suggests use of a fine-tuned YOLO model).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Singh/Wang combination regarding a deep learning module to enhance quality, a frozen model generated and constantly updated to ensure action is taken by an action classifier, and a set of feature maps for image with those of Cronje regarding a deep learning module to enhance quality of inspection, the deep learning module comprising: a frozen model generated and constantly updated through using an object detection model; a custom built LSTM (Long short term memory) model to distinguish between similar images in different states; and using a modified YOLO (You Only Look Once) for image.
The motivation to do so would be to facilitate training of object detection in scenarios with temporal state resulting in improved classification accuracy (Cronje, [0067]: "Each of the classifiers (274, 276 [i.e., CNN and LSTM] and 278) mentioned before in paragraph 3 can be used as a mobile device usage classifier on its own. The accuracy of the classification is further improved by combining the classification results (282, 284 and 286) of all the classifiers. ... The individual results are combined by a weighted sum where the weights are determined by optimizing the weights on the training dataset, to arrive at a final operator state (289)"
The Singh/Wang/Cronje combination teaches a deep learning module to enhance quality, a frozen model generated and constantly updated to ensure action is taken by an action classifier, and a set of feature maps for image, the deep learning module comprising: a frozen model generated and constantly updated through using an object detection model; a custom built LSTM (Long short term memory) model to distinguish between similar images in different states; and using a modified YOLO (You Only Look Once) for image.
The Singh/Wang/Cronje combination does not explicitly teach enhance quality of defect inspection on object surfaces.
However, Cho teaches:
enhance quality of defect inspection on object surfaces (Cho, [0047]: "the defect inspection device 100 may include other components for inspecting whether a surface of the inspection object is defective" and [0048]: "The processor 110 may perform calculations for learning the neural network, which include processing of input data for learning in deep learning (DL), extracting a feature in the input data, calculating an error, updating a weight of the neural network using backpropagation, and the like" and [0148]: "The neural network may be learned in a direction to minimize errors of an output").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Singh/Wang/Cronje combination regarding a deep learning module to enhance quality with those of Cho regarding enhance quality of defect inspection on object surfaces.
The motivation to do so would be to facilitate improved defect detection rates using a deep learning model regardless of image complexity (Cho, [0136]: "the processor 110 inputs the image data of the inspection object including the edge field into the deep learning based model and obtains outputs having various types of forms, and as a result, in the present disclosure, the visibility of the defect of the inspection object may be secured regardless of complexity of the image, thereby increasing a defect recognition rate").
The Singh/Wang/Cronje/Cho combination teaches using a modified YOLO (You Only Look Once) for image.
The Singh/Wang/Cronje/Cho combination does not explicitly teach a set of feature maps using ... a modified CTPN (Connectionist Text Proposal Network) for ... text.
However, Wu teaches:
a set of feature maps using ... a modified CTPN (Connectionist Text Proposal Network) for ... text (Wu, [0057]: "image segmentation can use a CTPN model to segment printed, handwritten, and signature text out the front of check images to generate extracted text box images. ... A CTPN model can use VGG 16 to generate a feature map" and [0045]:"the image segmentation 120 configures/implements a CTPN and a Textboxes++ network. This involves developing, training and tuning the networks based on real check images to achieve the parameters and weights. The networks are customized based on real check images to achieve acceptable performance," where Wu's trained and tuned CTPN network reasonably suggests a modified network).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Singh/Wang/Cronje/Cho combination regarding using a modified YOLO (You Only Look Once) for image with those of Wu regarding a set of feature maps using a modified CTPN (Connectionist Text Proposal Network) for text.
The motivation to do so would be to facilitate model training and tuning for scenarios where recognized text may occur in multiple sizes or configurations (Wu, [0058]: "FIG. 3A shows an example network structure for some embodiments of the CTPN model. A CTPN model can use ten (or any other predefined number of) different heights varying from 11 to 273 pixels" and [0059]: "The system server can adopt eight different heights from 16 to 194 pixels, since text on check images might be limited in height and size. The system server can reduce the horizontal gap between text boxes to 25 pixels to better fit check image data. ... The system server can use this pre-trained model to continue training on real check images to tune the weights to better fit check images").
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT N DAY whose telephone number is (703)756-1519. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/R.N.D./Examiner, Art Unit 2122
/KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122
Prosecution Insights