DETAILED ACTION
Claims 1-4, 6-7, 11-14 and 16-17 are pending in this application. Claims 1 and 11 are amended, claims 5, 8-10, 15, and 18-20 are canceled.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy was received on 12/6/2022.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 04/15/2026 has been entered.
Response to Arguments
U.S.C. 101
Applicant’s arguments regarding the amendments made to claims 1 and 11 (see Remarks filed 07/17/2026) have been fully considered by the examiner are not persuasive. The examiner disagrees with the applicant’s assertion that claims 1-4, 6-7, 11-14 and 16-17 are drawn to significantly more than an abstract idea.
Taking claim 1 as the example, the limitation of “an area defining unit, configured to obtain a recording area of a screen” is a step that could be practically performed as step of mere data gathering under step 2A prong 1, wherein a human could determine a location of interest to record a video of. Under step 2A prong 2, the claim recites the additional element of an area defining unit, however the additional elements fail to translate the claim into practical application or amounts to significantly more. Similarly, under Step 2B, the claim fails to recite any limitations which amount to significantly more an abstract idea or meaningfully translate the claim into practical application.
Claim 1 further recites the following limitations;
an area defining unit, configured to obtain a recording area of a screen, wherein the recording area is a scope of a remote control window, the remote control window displays an interface of a semiconductor machine loaded at a remote end, the recording area is a partial scope of the screen of the electronic device, and the recording area is the entire scope of the interface of the semiconductor machine; (Step of mere data gathering where a user could record a screen which displays anything, not limited to a semiconductor device)
a recording unit, configured to record a video according to the recording area; (Step of mere data gathering, in which a human could record a video of a screen)
an analysis unit, configured to analyze a plurality of actions, including text input, text deletion, cursor movement, menu popping out and menu closing, according to the video, (Mental process, where a human could reasonably look at a video and make note of the actions in the video, further the actions include actions which could reasonably performed manually by a person using a computer for normal functions such as typing an email or opening an application)
including: a filter, configured to filter off some frames in the video whose changes belonging to a cursor; (Mental process where a human could remove or delete image frames with a cursor changes/movement)
a creation unit, configured to build a plurality of steps of a script according to the actions; (Mental process in which a human could look at a video of an action, and write a script of the actions)
an execution unit, configured to execute the actions of the script on the remote control window to automatically complete operations of the semiconductor machine; (Abstract idea in which the user may manually execute a script)
wherein the analysis unit is configured to identify a window and a reference pattern thereon, where in the reference pattern is shown on the interface of the semiconductor machine located at the remote end; (Mental process of determining a window and a cursor movement pattern)
analyze a relative location of the cursor on the window relative to the reference pattern; (Mental process in which a human could look at a cursor movement and determine its location relative to a shape or pattern on the image)
define an execution position of a click action on the window according to the relative location of the cursor on the window relative to the reference pattern, wherein the reference pattern is different from the cursor; (Mental process of determining a mouse position for execution and the cursor’s relative location from a pattern or shape or icon on the screen)
and record the click action with the execution position on the window as one of the actions. (Mere data gathering where a person could record a mouse click in the script, they are generating based on watching a video)”
The steps above are drawn to either a mental process or step of mere data gathering under step 2A prong 1 as noted by the examiner because a human could reasonably look at a screen recording of a user completing a task, and make notes of the icon/reference pattern being click on, and the position of the mouse. Under step 2A prong 2, the claim recites the additional elements of “a recording unit”, “an analysis unit”, “a filter” and “a creation unit”, however the additional elements fail to translate the claim into practical application or amounts to significantly more. Similarly, under Step 2B, the claim fails to recite any limitations which amount to significantly more an abstract idea or meaningfully translate the claim into practical application. Therefore, at this time, Examiner maintains all rejections made under 35 U.S.C. 101. The applicant is encouraged to amend the above claims to add in additional elements which meaningfully translate the claims into practical application.
35 U.S.C. 103
Applicant’s arguments (see Remarks filed 07/17/2026) regarding the rejections made to claims 1 and 11 under 35 U.S.C. 103 over Jin in view of Klementiev have been fully considered by the examiner and are persuasive. However, given the amendments made to claims 1 and 11, a new grounds of rejection is presented in view of Jin, Klementiev, Sano and Fang, as fully discussed below.
Claim Interpretation
1. The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are:
Area defining unit of claim 1 (See [0026] of applicant’s specification).
recording unit of claim 1 (See [0026] of applicant’s specification).
analysis unit of claim 1 (See [0026] of applicant’s specification).
creation unit of claim 1 (See [0026] of applicant’s specification).
Action analyzer of claim 2, 3, 4, 5, 6, and 7 (See [0026] of applicant’s specification).
because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 101
2. 35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-4, 6-7, 11-14 and 16-17 are rejected under U.S.C. 101 as being directed to an abstract idea without significantly more.
Regarding claims 1 and 11, claims 1 and 11 are directed to the abstract ideas, mental processes or mere data gathering.
The claims recite the following limitations;
an area defining unit, configured to obtain a recording area of a screen, wherein the recording area is a scope of a remote control window, the remote control window displays an interface of a semiconductor machine loaded at a remote end, the recording area is a partial scope of the screen of the electronic device, and the recording area is the entire scope of the interface of the semiconductor machine; (Step of mere data gathering where a user could record a screen which displays anything, not limited to a semiconductor device)
a recording unit, configured to record a video according to the recording area; (Step of mere data gathering, in which a human could record a video of a screen)
an analysis unit, configured to analyze a plurality of actions, including text input, text deletion, cursor movement, menu popping out and menu closing, according to the video, (Mental process, where a human could reasonably look at a video and make note of the actions in the video, further the actions include actions which could reasonably performed manually by a person using a computer for normal functions such as typing an email or opening an application)
including: a filter, configured to filter off some frames in the video whose changes belonging to a cursor; (Mental process where a human could remove or delete image frames with a cursor changes/movement)
a creation unit, configured to build a plurality of steps of a script according to the actions; (Mental process in which a human could look at a video of an action, and write a script of the actions)
an execution unit, configured to execute the actions of the script on the remote control window to automatically complete operations of the semiconductor machine; (Abstract idea in which the user may manually execute a script)
wherein the analysis unit is configured to identify a window and a reference pattern thereon, where in the reference pattern is shown on the interface of the semiconductor machine located at the remote end; (Mental process of determining a window and a cursor movement pattern)
analyze a relative location of the cursor on the window relative to the reference pattern; (Mental process in which a human could look at a cursor movement and determine its location relative to a shape or pattern on the image)
define an execution position of a click action on the window according to the relative location of the cursor on the window relative to the reference pattern, wherein the reference pattern is different from the cursor; (Mental process of determining a mouse position for execution and the cursor’s relative location from a pattern or shape or icon on the screen)
and record the click action with the execution position on the window as one of the actions. (Mere data gathering where a person could record a mouse click in the script, they are generating based on watching a video)”
The steps above are drawn to either a mental process or step of mere data gathering under step 2A prong 1 as noted by the examiner because a human could reasonably look at a screen recording of a user completing a task and make notes of the icon/reference pattern being click on, and the position of the mouse. Under step 2A prong 2, the claim recites the additional elements of “a recording unit”, “an analysis unit”, “a filter” and “a creation unit”, however the additional elements fail to translate the claim into practical application or amounts to significantly more. Similarly, under Step 2B, the claim fails to recite any limitations which amount to significantly more an abstract idea or meaningfully translate the claim into practical application.
Dependent claims 2-4, 6-7, 11-14 and 16-17 do not add limitations that meaningfully translate the abstract idea into a practical application or add significantly more.
Regarding claims 2 and 12, the claims add the limitations.
“Configured to extract a plurality of frames from the video”
“configured to analyze a plurality of changes in the frames”
“configured to define a plurality of segmentation nodes from the frames, wherein the changes in each of the segmentation nodes are greater than a predetermined degree”
“configured to obtain the actions between the segmentation nodes, which are adjacent”
The limitation recited by claims 2 and 12 are drawn to either mental processes or mere data gathering without significantly more.
Regarding claims 3 and 13, the claims add the limitations;
“records a newly added text in the changes between the segmentation nodes, which are adjacent, to obtain a text input action”
Which is drawn to mere data gathering without significantly more.
Regarding claims 4 and 14, the claims add the limitation;
“record newly added text by using optical character recognition (OCR) technology”
Which is drawn to a mental process. The usage of OCR technology is not enough to translate the claims into practical application or amount to significantly more.
Regarding claims 6 and 16, the claims add the limitation;
“records a newly added rectangular frame in each of the changes between the segmentation nodes, which are adjacent, to obtain a circle action” which is drawn to mere data gathering without significantly more.
Regarding claims 7 and 17, the claims add the limitation;
“records a newly added highlighted area in each of the changes between the segmentation nodes, which are adjacent, to obtain a text highlight action.” Which is drawn to both mere data gathering and the mental process of examining changes in data without significantly more.
These claims add limitations that fail to tie the abstract ideas into practical application or add significantly more, including decision making, mental processes and data gathering.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
3. Claim 1 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Jin (KR 102307471 B1) in view of Klementiev (US 20050278728 A1) and in further view of Sano (US 20090150794 A1) and Fang (US 20190333205A1).
Regarding claim 1 Jin discloses;
An electronic device, comprising:
an area defining unit (Jin [0015] a template tool module, automates a specific area, Applicant defined the units as being “realized by a circuit, a chip, a circuit board, a code, a computer program product or a storage device for storing code, therefore the template tool module would be analogous to this), configured to obtain a recording area of a screen (Jin [0015] The template tool module is responsible for producing a template to automate a task for the area within the template);
wherein the recording area is a scope of a remote control window (Jin, figures 2 and 3 show a remote-control window and other programs being used by the user and recorded),
[ the remote control window displays an interface of a semiconductor machine loaded at a remote end,
the recording area is a partial scope of the screen of the electronic device ,
and the recording area is the entire scope of the interface of the semiconductor machine; ]
a recording unit (Jin [0015] Modeling tool module, Applicant defined the units as being “realized by a circuit, a chip, a circuit board, a code, a computer program product or a storage device for storing code, therefore the Modeling tool module would be analogous to this), configured to record a video according to the recording area (Jin [0015] records task processes to be automated);
[an analysis unit, configured to analyze a plurality of actions, including text input, text deletion, cursor movement, menu popping out and menu closing, according to the video, including: a filter, configured to filter off changes belonging to a cursor;]
a creation unit (Jin, [0027] script writing unit, Applicant defined the units as being “realized by a circuit, a chip, a circuit board, a code, a computer program product or a storage device for storing code, therefore the script recorder would be analogous to this), configured to build a plurality of steps of a script according to the actions (Jin [0027] the script writing unit writes a custom script of the task a user wishes to have automated);
Jin does not teach;
the remote control window displays an interface of a semiconductor machine loaded at a remote end,
the recording area is a partial scope of the screen of the electronic device ,
and the recording area is the entire scope of the interface of the semiconductor machine;
an analysis unit, configured to analyze a plurality of actions, including text input, text deletion, cursor movement, menu popping out and menu closing, according to the video,
including: a filter, configured to filter off changes belonging to a cursor;
an execution unit, configured to execute the actions of the script on the remote control window to automatically complete operations of the semiconductor machine;
wherein the analysis unit is configured to identify a window and a reference pattern thereon, where in the reference pattern is shown on the interface of the semiconductor machine located at the remote end;
analyze a relative location of the cursor on the window relative to the reference pattern;
define an execution position of a click action on the window according to the relative location of the cursor on the window relative to the reference pattern, wherein the reference pattern is different from the cursor;
and record the click action with the execution position on the window as one of the actions.
However, in the same field of endeavor of script creation, Klementiev teaches;
an analysis unit, configured to analyze a plurality of actions, including text input, text deletion, cursor movement, menu popping out and menu closing, according to the video (Klementiev, [0016] the system uses User Interface Recording software which capture the inputs as a video [0051] input data is derived from user actions such as mouse movements and clicks, [0078] and [0081] the recorder records events on the screen which are then aggregated, such even include mouse motion/cursor motion, as well as keystrokes, including the user hitting the “delete” button (text deletion) [0065] the collector (analysis unit) records expand events such as a menu expansion (menu popping out), or collapse of a menu (closing a menu), [0063] keyboard inputs from the user are captured, such as pressed keys or combinations of pressed keys (text input), [0071] the user may add text strings into the edit box and this may be captured),
including: a filter, configured to filter off changes belonging to a cursor (Klementiev, [0016] the system uses User Interface Recording software which capture the inputs as a video [0051] input data is derived from user actions such as mouse movements and clicks, [0073] the collected user input data can be filtered, [0076] the user can filter out certain patterns of the input data such as clicking or expanding a certain menu. Since the user or the system can filter the data collected such as mouse movements and clicks based on user specification or based on the script creation goals, this would be analogous to filtering the mouse clicks off.);
The combination of Jin and Klementiev would have been obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. Jin teaches a method of generating a script to execute actions being performed by a user and recorded, but it does not adequately teach filtering off mouse clicks using a filter. Klementiev teaches this deficiency, the motivation for the addition of this feature of Klementiev lies in that the ability to filter the mouse actions that are not relevant to what is being input into the script. (Klementiev, [0018]- [0019], [0025] and [0081]- [0084])
Neither Jin nor Klementiev teaches; the remote control window displays an interface of a semiconductor machine loaded at a remote end,
the recording area is a partial scope of the screen of the electronic device ,
and the recording area is the entire scope of the interface of the semiconductor machine;
an execution unit, configured to execute the actions of the script on the remote control window to automatically complete operations of the semiconductor machine;
wherein the analysis unit is configured to identify a window and a reference pattern thereon, where in the reference pattern is shown on the interface of the semiconductor machine located at the remote end;
analyze a relative location of the cursor on the window relative to the reference pattern;
define an execution position of a click action on the window according to the relative location of the cursor on the window relative to the reference pattern, wherein the reference pattern is different from the cursor;
and record the click action with the execution position on the window as one of the actions.
However, in the same field of endeavor, Sano teaches; the remote control window displays an interface of a semiconductor machine loaded at a remote end (Sano, Figure 10, shows a block diagram of an interface showing the semiconductor exposure apparatus, which shows information about the semiconductor and its status, as well as showing parts of the semiconductor, [0004] details the use of remote semiconductor modeling during manufacturing, [0009] semiconductor exposure apparatus is used to monitor control parameters and visualize the semiconductor components during manufacture (semiconductor interface)),
the recording area is a partial scope of the screen of the electronic device ((Sano Figure 5, shows the screen with the semiconductor interface displayed, while the use of screen recording is not explicitly disclosed, it would be obvious to one of ordinary skill in the art that continuous monitoring of a device could also be recorded),
and the recording area is the entire scope of the interface of the semiconductor machine (Sano figure 10 shows both the semiconductor exposure apparatus and the remote-control terminal);
an execution unit, configured to execute the actions of the script on the remote control window to automatically complete operations of the semiconductor machine (Sano, [0076]- [0081] the user interface executes programs which automatically adjust a semiconductor’s exposure parameters and run these semiconductor operations remotely);
The combination of Jin, Klementiev, and Sano would have been obvious to one of ordinary skill in the art of robotic automation of semiconductor interface manufacturing and testing prior to the effectively filed date of the presently claimed invention. The Script generation method of Jin, and Klementiev would have been improved by the manufacturing and monitoring system of Sano because one of ordinary skill in the art wanting to automate the process of semiconductor testing would have use the semiconductor monitoring and testing method of Sano and combined it with the recording and script generation methods of Jin and Klementiev to create a system with functionally equivalent capabilities to those disclosed above. Additionally, while Sano does not explicitly disclose the use of screen recording in monitoring the semiconductor interface during manufacture and testing, it does teach remote monitoring and the use of user control and testing using a terminal, therefore one of ordinary skill in the art could have reasonably added the step of screen recording given the disclosures of Jin and Klementiev. (Sano [0001]- [0009])
Neither, Jin, Klementiev nor Sano teach;
wherein the analysis unit is configured to identify a window and a reference pattern thereon, where in the reference pattern is shown on the interface of the semiconductor machine located at the remote end;
analyze a relative location of the cursor on the window relative to the reference pattern;
define an execution position of a click action on the window according to the relative location of the cursor on the window relative to the reference pattern, wherein the reference pattern is different from the cursor;
and record the click action with the execution position on the window as one of the actions.
However, in the same field of endeavor, Fang teaches
wherein the analysis unit is configured to identify a window and a reference pattern thereon (Fang, [0040] defects on a semiconductor wafer (reference pattern) are displayed to a user interface (window) as well as reference images of that defect, where [0041]-[0043] the system identifies defect patterns on the device and shows reference images on the screen window), where in the reference pattern is shown on the interface of the semiconductor machine located at the remote end (Fang, [0040] defects on a semiconductor wafer (reference pattern) are displayed to a user interface as well as reference images of that defect);
analyze a relative location of the cursor on the window relative to the reference pattern (Fang, [0039] the display is connected to an input device, where the input device is a cursor which allows the user to click and control the display, the user clicks are recorded positionally in (x and y), [0055]-[0045] figure 5 shows an example of the semiconductor image analysis by the user, where the pattern on the semiconductor can be aligned with a selection “polygon” positionally, [0048] and [0062] this can be defined by the user with the input device (cursor) therefore the cursor position relative to a defect pattern (reference pattern) on the semiconductor would be analyzed to know if it was selected.);
define an execution position of a click action on the window according to the relative location of the cursor on the window relative to the reference pattern, wherein the reference pattern is different from the cursor (Fang, [0039] the display is connected to an input device, where the input device is a cursor which allows the user to click and control the display, the user clicks are recorded positionally in x and y (execution position relative to the pattern), [0055]-[0045] figure 5 shows an example of the semiconductor image analysis by the user, where the pattern on the semiconductor can be aligned with a selection “polygon” positionally, [0048] and [0062] this can be defined by the user with the input device (cursor) therefore the cursor position relative to a defect pattern (reference pattern) on the semiconductor would be analyzed to know if it was selected);
and record the click action with the execution position on the window as one of the actions (Fang, [0039] the display is connected to an input device, where the input device is a cursor which allows the user to click and control the display, the user clicks are recorded positionally in x and y (execution position relative to the pattern, [0030] the operators’ feedback is collected/recorded through the input device).
The combination of Jin, Klementiev, Sano and Fang would have been obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. The script automation methods of Jin, Klementiev and Sano would be reasonably improved by the semiconductor defect assessment of Fang. This addition allows the user to select and determine defects and other manufacturing adjustments for a semiconductor efficiently and remotely, thereby making the process more accurate and faster. (Fang, [0021]- [0025] and [0030])
Regarding claim 11, the combination of Jin, Klementiev, Sano and Fang teaches; A script creation method for robotic process automation (RPA), comprising:
obtaining a recording area of a screen (Jin [0015] The template tool module is responsible for producing a template to automate a task for the area within the template);
wherein the recording area is a scope of a remote control window (Jin, figures 2 and 3 show a remote-control window and other programs being used by the user and recorded),
the remote control window displays an interface of a semiconductor machine loaded at a remote end (Sano, Figure 10, shows a block diagram of an interface showing the semiconductor exposure apparatus, which shows information about the semiconductor and its status, as well as showing parts of the semiconductor, [0004] details the use of remote semiconductor modeling during manufacturing, [0009] semiconductor exposure apparatus is used to monitor control parameters and visualize the semiconductor components during manufacture (semiconductor interface)).
the recording area is a partial scope of the screen of the electronic device (Sano Figure 5, shows the screen with the semiconductor interface displayed, while the use of screen recording is not explicitly disclosed, it would be obvious to one of ordinary skill in the art that continuous monitoring of a device could also be recorded),
and the recording area is the entire scope of the interface of the semiconductor machine (Sano figure 10 shows both the semiconductor exposure apparatus and the remote-control terminal);
recording a video according to the recording area (Jin [0015] records task processes to be automated);
analyzing a plurality of actions, including text input, text deletion, cursor movement, menu popping out and menu closing, according to the video (Klementiev, [0016] the system uses User Interface Recording software which capture the inputs as a video [0051] input data is derived from user actions such as mouse movements and clicks [0078] and [0081] the recorder/collector (analysis unit) records events on the screen which are then aggregated, such even include mouse motion/cursor motion, as well as keystrokes, including the user hitting the “delete” button (text deletion) [0065] the collector records expand events such as a menu expansion (menu popping out), or collapse of a menu (closing a menu), [0063] keyboard inputs from the user are captured, such as pressed keys or combinations of pressed keys (text input), [0071] the user may add text strings into the edit box and this may be captured),
wherein in the step of analyzing the plurality of actions, some frames in the video whose changes belonging to the cursor are filtered off (Klementiev, [0016] the system uses User Interface Recording software which capture the inputs as a video [0051] input data is derived from user actions such as mouse movements and clicks, [0073] the collected user input data can be filtered, [0076] the user can filter out certain patterns of the input data such as clicking or expanding a certain menu. Since the user or the system can filter the data collected such as mouse movements and clicks based on user specification or based on the script creation goals, this would be analogous to filtering the mouse clicks off.);
building a plurality of steps of a script according to the actions (Jin [0027] the script writing unit writes a custom script of the task a user wishes to have automated);
executing the actions of the script on the remote control window to automatically complete operations of the semiconductor machine (Sano, [0076]- [0081] the user interface executes programs which automatically adjust a semiconductor’s exposure parameters and run these semiconductor operations remotely);
wherein the step of analyzing the actions includes; identifying a window and a reference pattern thereon (Fang, [0040] defects on a semiconductor wafer (reference pattern) are displayed to a user interface (window) as well as reference images of that defect, where [0041]-[0043] the system identifies defect patterns on the device and shows reference images on the screen window), where in the reference pattern is shown on the interface of the semiconductor machine located at the remote end (Fang, [0040] defects on a semiconductor wafer (reference pattern) are displayed to a user interface as well as reference images of that defect);
analyzing a relative location of the cursor on the window relative to the reference pattern (Fang, [0039] the display is connected to an input device, where the input device is a cursor which allows the user to click and control the display, the user clicks are recorded positionally in (x and y), [0055]-[0045] figure 5 shows an example of the semiconductor image analysis by the user, where the pattern on the semiconductor can be aligned with a selection “polygon” positionally, [0048] and [0062] this can be defined by the user with the input device (cursor) therefore the cursor position relative to a defect pattern (reference pattern) on the semiconductor would be analyzed to know if it was selected.);
defining an execution position of a click action on the window according to the relative location of the cursor on the window relative to the reference pattern, wherein the reference pattern is different from the cursor (Fang, [0039] the display is connected to an input device, where the input device is a cursor which allows the user to click and control the display, the user clicks are recorded positionally in x and y (execution position relative to the pattern), [0055]-[0045] figure 5 shows an example of the semiconductor image analysis by the user, where the pattern on the semiconductor can be aligned with a selection “polygon” positionally, [0048] and [0062] this can be defined by the user with the input device (cursor) therefore the cursor position relative to a defect pattern (reference pattern) on the semiconductor would be analyzed to know if it was selected);
and recording the click action with the execution position on the window as one of the actions (Fang, [0039] the display is connected to an input device, where the input device is a cursor which allows the user to click and control the display, the user clicks are recorded positionally in x and y (execution position relative to the pattern, [0030] the operators’ feedback is collected/recorded through the input device).
The combination of Jin, Klementiev, Sano, and Fang would have been obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. Jin teaches a method of generating a script to execute actions being performed by a user and recorded, but it does not adequately teach filtering off mouse clicks using a filter. Klementiev teaches this deficiency, the motivation for the addition of this feature of Klementiev lies in that the ability to filter the mouse actions that are not relevant to what is being input into the script (Klementiev, [0018]- [0019], [0025] and [0081]- [0084]). The script generation method of Jin and Klementiev would have been improved by the manufacturing and monitoring system of Sano. One of ordinary skill in the art wanting to automate the process of semiconductor testing would have use the semiconductor monitoring and testing method of Sano and combined it with the recording and script generation methods of Jin and Klementiev to create a system with functionally equivalent capabilities to those disclosed above (Sano [0001]- [0009]). Further, the methods of Jin, Klementiev and Sano would be reasonably improved by the semiconductor defect assessment of Fang. This addition allows the user to select and determine defects and other manufacturing adjustments for a semiconductor efficiently and remotely, thereby making the process more accurate and faster (Fang, [0021]- [0025] and [0030]).
4. Claims 2-4, and 12-13 are rejected under 35 U.S.C. 103 as being unpatentable Jin (KR 102307471 B1) in view of Klementiev (US 20050278728 A1) and in further view of Sano (US 20090150794 A1) a4d Fang (US 20190333205A1) and in further view of Otterness (20210224950 A1).
Regarding claim 2 the combination of Jin, Klementiev, Sano and Fang teaches; The electronic device according to claim 1, wherein the analysis unit comprises:
and an action analyzer (Jin, [0025], the processing unit), configured to obtain the actions between the segmentation nodes, which are adjacent (Jin [0025] processing unit analyzes actions inputs across elements (segmentation nodes)).
The combination does not teach;
an extractor, configured to extract a plurality of frames from the video;
a comparator, configured to analyze a plurality of changes in the frames;
a divider, configured to define a plurality of segmentation nodes from the frames, wherein the changes in each of the segmentation nodes are greater than a predetermined degree;
However, in the same field of endeavor Otterness teaches;
an extractor (Otterness, [0033], event recognition module is performing an equivalent function to an extractor), configured to extract a plurality of frames from the video (Otterness, the module may receive all frames and analyze a sampling of the frames);
a comparator (Otterness, [0033] the event recognition module, performs equivalent function to a comparator), configured to analyze a plurality of changes in the frames (Otterness, [0033] one or more regions of the frame be analyzed for additional information, icons can be detected between frames);
a divider (Otterness, [0033] the event recognition module is performing a functionally equivalent task to a divider), configured to define a plurality of segmentation nodes from the frames, wherein the changes in each of the segmentation nodes are greater than a predetermined degree (Otterness, [0033] one or more regions of the frame be analyzed for additional information (Regions of the frame are being interpreted as functionally equivalent to a segmentation node since they are breaking the frame into portions as shown in applicant’s figures 7-10), icons can be detected between frames [0038] the frames can be processed to detect changes in select regions of the game, for example, the appearance of a certain number of icons can indicate an event);
The combination of Jin, Klementiev, Sano, Fang, and Otterness would have been obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. The motivation for the combination lies in that the frame analysis of Otterness would allow regions of video frames to be analyzed for action inputs as described in Jin. (Otterness [0033], [0035], and [0038])
Regarding claim 3 the combination of Jin, Klementiev, Sano, Fang, and Otterness, The electronic device according to claim 2, wherein the action analyzer records a newly added text in the changes between the segmentation nodes, which are adjacent, to obtain a text input action (Jin, [0025] Processing unit extra elements of a sequence of events (segmentation node) [0049] user’s work is recorded repeatedly [0050] the modeling tool module (Action analyzer) can apply OCR to images captured from user’s actions [0051] OCR is used to extract text and perform pattern recognition from the user’s action).
Regarding claim 4 the Jin, Klementiev, Sano, Fang, and Otterness; The electronic device according to claim 3, wherein the action analyzer obtains the newly added text by using an optical character recognition (OCR) technology (Jin, [0049] user’s work is recorded repeatedly [0050] the modeling tool module (Action analyzer) can apply OCR to images captured from user’s actions [0051] OCR is used to extract text and perform pattern recognition from the user’s action).
Regarding claim 12 the combination of Jin, Klementiev and Otterness teaches; The script creation method for robotic process automation according to claim 11, wherein the step of analyzing the actions according to the video comprises:
extracting a plurality of frames from the video (Otterness, the module may receive all frames and analyze a sampling of the frames);
analyzing a plurality of changes in the frames (Jin, [0051] Use of OCR extraction to extract information from videos or images, [0127] data scraping methods (extraction) can be used on binary data from images or multimedia data, examiner is interpreting the use of multiple extraction methods as the system having one or more extractors, capable of extracting data/frames from a video);
defining a plurality of segmentation nodes from the frames, wherein the changes in each of the segmentation nodes are greater than a predetermined degree (Otterness, [0033] one or more regions of the frame be analyzed for additional information (Regions of the frame are being interpreted as functionally equivalent to a segmentation node since they are breaking the frame into portions as shown in applicant’s figures 7-10), icons can be detected between frames [0038] the frames can be processed to detect changes in select regions of the game, for example, the appearance of a certain number of icons can indicate an event);
and obtaining the actions between the segmentation nodes, which are adjacent (Jin [0025] processing unit analyzes actions inputs across elements (segmentation nodes)).
The combination of Jin, Klementiev, Sano, Fang, and Otterness would have been obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. The motivation for the combination lies in that the frame analysis of Otterness would allow regions of video frames to be analyzed for action inputs as described in Jin. (Otterness [0033], [0035], and [0038])
Regarding claim 13 the combination of Jin, Klementiev, Sano, Fang, and Otterness teaches; The script creation method for robotic process automation according to claim 12, wherein a newly added text in the changes between the segmentation nodes, which are adjacent, is recorded to obtain a text input action. (Jin, [0025] Processing unit extra elements of a sequence of events (segmentation node) [0049] user’s work is recorded repeatedly [0050] the modeling tool module (Action analyzer) can apply OCR to images captured from user’s actions [0051] OCR is used to extract text and perform pattern recognition from the user’s action).
Regarding claim 14 the Jin, Klementiev, Sano, Fang, and Otterness teaches; The script creation method for robotic process automation according to claim 13, wherein the newly added text is obtained by using an optical character recognition (OCR) technology (Jin, [0049] user’s work is recorded repeatedly [0050] the modeling tool module (Action analyzer) can apply OCR to images captured from user’s actions [0051] OCR is used to extract text and perform pattern recognition from the user’s action).
5. Claims 6-7 and 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Jin (KR 102307471 B1) in view of Klementiev (US 20050278728 A1) and in further view of Sano (US 20090150794 A1) and Fang (US 20190333205A1) and in further view of Otterness (20210224950 A1) and Butin (US 2010205529 A1).
Regarding claim 6 the combination of Jin, Klementiev, Sano, Fang, and Otterness fails to teach; The electronic device according to claim 2, wherein the action analyzer records a newly added rectangular frame in each of the changes between the segmentation nodes, which are adjacent, to obtain a circle action.
However, in the same field of endeavor, Butin teaches; The electronic device according to claim 2, wherein the action analyzer records a newly added rectangular frame in each of the changes between the segmentation nodes, which are adjacent, to obtain a circle action (Butin, [0156] The system defines an rectangle area around the mouse pointer to obtain the mouse action, which is functionally equivalent to a circle action since this could obtain any pattern the mouse clicks make, including but not limited to a circle.)
The combination of Jin, Klementiev, Sano, Fang, Otterness and Butin would have been obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. The motivation for the combination lies in that the use of a predefined analysis as described in Butin would allow the mouse action or pattern of mouse clicks to be determined more accurately. (Butin, [0154]- [0156])
Regarding claim 7 The combination of Jin, Klementiev, Sano, Fang, Otterness and Butin teaches; The electronic device according to claim 2, wherein the action analyzer records a newly added highlighted area in each of the changes between the segmentation nodes, which are adjacent, to obtain a text highlight action (Butin, [0093] the user is able to input new text queries to be analyzed at different points in using the system, this is functionally equivalent to highlighting because new text is being selected/input to be recognized,[0100] text can be searched for and matched based upon the user’s query [0101] resultant text may be colored to show the user the result (text highlight action)).
The combination of Jin, Klementiev, Sano, Fang, Otterness and Butin would have been obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. Motivation for the combination lies in that the method of Butin allows the user to input select keywords or questions and have them be queued up for the user based upon the selected words. Since the user can select text to be queued up and displayed in the results, one of ordinary skill in the art would understand that this is functionally equivalent to highlighting text and recording the highlighted text because in both cases text is newly selected and displayed, and the results are recorded by the system. (Butin, [0093], [0100] and [0101])
Regarding claim 16 The combination of Jin, Klementiev, Sano, Fang, Otterness and Butin teaches; The script creation method for robotic process automation according to claim 12, wherein a newly added rectangular frame in each of the changes between the segmentation nodes, which are adjacent, is recorded to obtain a circle action(Butin, [0156] The system defines an rectangle area around the mouse pointer to obtain the mouse action, which is functionally equivalent to a circle action since this could obtain any pattern the mouse clicks make)
The combination of Jin, Klementiev, Sano, Fang, Otterness and Butin would have been obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. The motivation for the combination lies in that the use of a predefined analysis as described in Butin would allow the mouse action or pattern of mouse clicks to be determined more accurately. (Butin, [0154]- [0156])
Regarding claim 17 The combination of Jin, Klementiev, Sano, Fang, Otterness and Butin teaches; The script creation method for robotic process automation according to claim 12, wherein a newly added highlighted area in each of the changes between the segmentation nodes, which are adjacent, is recorded to obtain a text highlight action (Butin, [0093] the user is able to input new text queries to be analyzed at different points in using the system, this is functionally equivalent to highlighting because new text is being selected/input to be recognized,[0100] text can be searched for and matched based upon the user’s query [0101] resultant text may be colored to show the user the result (text highlight action)).
The combination of Jin, Klementiev, Sano, Fang, Otterness and Butin would have been obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. Motivation for the combination lies in that the method of Butin allows the user to input select keywords or questions and have them be queued up for the user based upon the selected words. Since the user can select text to be queued up and displayed in the results, one of ordinary skill in the art would understand that this is functionally equivalent to highlighting text and recording the highlighted text because in both cases text is newly selected and displayed, and the results are recorded by the system. (Butin, [0093], [0100] and [0101])
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 8,725,845 B2, Moorer.
Teaches a robotic automation system in which hardware in a user’s home screen is remote monitored and scripts are generated to be executed on a terminal. (Pertinent to claims 1-5)
US 2010/0149192 A1, Kota.
Teaches a method of creating and parsing an action script to analyze image data. (pertinent to claims 1-3)
US 20130019170 A1, Mounty.
Teaches screen recording and filtering of frames from a set of frame data (pertinent to claims 1 and 11).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JORDAN M ELLIOTT whose telephone number is (703)756-5463. The examiner can normally be reached M-F 8AM-5PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached on (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.M.E./Examiner, Art Unit 2666
/EMILY C TERRELL/Supervisory Patent Examiner, Art Unit 2666
Prosecution Insights