DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 03 July 2024 was filed prior to the mailing date of this office correspondence. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Specification
Abstract: The abstract of the disclosure is objected to because the abstract does not present a summary of the instant application. The presented abstract is the summary the allowed Patent US 12,063,745. A corrected abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text. See MPEP § 608.01(b).
Claim Objections
Claims 1 and 11 are objected to because of the following informalities:
Claim 1, line 5-7: “capturing, with an artificial neural network of a processing unit, at least one image in a digitized form depicting a location in or on the handling system or in the environment of the handling system;” should read:
-- capturing, with an image acquisition unit comprising an artificial neural network of a processing unit, at least one image in a digitized form depicting a location in or on the handling system or in the environment of the handling system; --
In claim 11, line 5: “a robot arm an end effector,” should read:
-- a robot arm, an end effector, --
In claim 11, line 8: “an acquisition unit for capturing at least one image;” should read:
-- an acquisition unit comprising an image acquisition unit for capturing at least one image;
In claim 11, lines 28-29: “pass on to a user who draws conclusions from this information for his actions operating the handling system or machine;” should read:
-- pass this information for operating the handling system or machine;
(Also, see Note 2 below)
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-11 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
In claim 1, line 23, the recited limitation “or any combination thereof.” renders claim indefinite because the metes and bounds of the claim is unclear. If the processing unit uses the information for the recited limitation in lines 12-22 “directly controlling the robot, the moving element, the robot arm, the end effector, the positioning unit, or any combination thereof; supportively controlling the robot,…or any combination thereof; aligning the robot,…or any combination thereof; training the robot,…or any combination thereof; monitoring the robot,…or any combination thereof;”, it is unclear what are the combinations that applicant is seeking protection from the recited “or any combination thereof.” in line 23.
As best understood, it appears that the limitation “directly controlling the robot, the moving element, the robot arm, the end effector, the positioning unit, or any combination thereof; supportively controlling the robot, the moving element, the robot arm, the end effector, the positioning unit, or any combination thereof; aligning the robot, the moving element, the robot arm, the end effector, the positioning unit, or any combination thereof; training the robot, the moving element, the robot arm, the end effector, the positioning unit, or any combination thereof; monitoring the robot, the moving element, the robot arm, the end effector, the positioning unit, or any combination thereof; or any combination thereof.” actually intends that:
directly controlling the robot, supportively controlling the robot, aligning the robot, training the robot, or monitoring the robot; or
controlling the moving element, the robot arm, the end effector, or the positioning unit.
The same issues is there for the limitation in claim 11, line 32, the recited limitation “or any combination thereof.”
Claims 2-10 depend on claim 1. Therefore, claims 1-11 are rejected.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 6-7 and 9-11 are rejected under 35 U.S.C. 103 as being unpatentable over Tremblay (US 20210118166) in view of Pannese (US 20070282480).
[AltContent: textbox (moving element)][AltContent: arrow][AltContent: textbox (end effector)][AltContent: ]
PNG
media_image1.png
370
554
media_image1.png
Greyscale
Annotated Fig. 1A, Tremblay.
Regarding claim 1, Tremblay teaches, a method for monitoring or controlling handling systems (Figs. 1A to 7), comprising:
providing a robot (robot 104, Fig. 1A), a moving element, robot arm (articulated limbs 108), an end effector (see annotated Fig. 1A), a positioning unit (base 106), or any combination thereof, having means for gripping (see annotated Fig. 1A, robot 104 can have various articulated limbs 108 or components, para. [0046]);
capturing, with an artificial neural network of a processing unit, at least one image in a digitized form depicting a location in or on the handling system or in the environment of the handling system (a camera 102 might be used to capture images, or video, or an autonomous object, such as a robot 104… captured image 202 of a robot can be provided as input to a trained neural network 204, Fig. 2, para. [0044, 0047]);
analyzing, with the artificial neural network, the at least one image; and generating, with the artificial neural network, an information data set and/or a control command (neural network 204 can be trained specifically for a type of robot 204,…these synthetic images can then serve as training data for a neural network, as images include representations of a type of robot in specific poses, and corresponding pose data can serve as ground truth data for training, para. [0047, 0050]);
wherein the processing unit uses the information data set and/or the control command for:
directly controlling the robot, the moving element, the robot arm, the end effector, the positioning unit, or any combination thereof; supportively controlling the robot, the moving element, the robot arm, the end effector, the positioning unit, or any combination thereof; aligning the robot, the moving element, the robot arm, the end effector, the positioning unit, or any combination thereof; training the robot, the moving element, the robot arm, the end effector, the positioning unit, or any combination thereof; monitoring the robot, the moving element, the robot arm, the end effector, the positioning unit, or any combination thereof; or any combination thereof (a neural network is to be trained for a specific type of robot, or autonomous object, a respective model and kinematic data can be provided as input, and control interface 308 can specify different poses for that robot…these synthetic images can then serve as training data for a neural network, as images include representations of a type of robot in specific poses, and corresponding pose data can serve as ground truth data for training, para. [0050], a tool can be used to generate these images that allows for scripting of robotic joint controls, para. [0056]).
Tremblay does not teach, the robot transporting and/or depositing a substrate. However, Pannese teaches a method for monitoring or controlling a handling system 100 in Figs. 1 to 10 for processing a wafer 104, including image inputs to a plurality of neural networks (para, [0112]); providing a robot (see Fig. 1, para. [0048]), a moving element, robot arm, an end effector, a positioning unit, or any combination thereof (Fig. 1, handling hardware 112 may include one or more robotic arms, transport carts, elevators, transfer stations and the like, as well as combinations of these, para. [0048]), having means for gripping, transporting and/or depositing a substrate (handling hardware 112 operates to manipulate wafers 104 within the system 100, such as by moving a wafer 104 between two of the process tools 110, or to/from the load lock 116, para. [0048]). From the teachings of Tremblay para. [0047-0050] and Figs. 1A to 7, a neural network is trained for a specific type of robot, one of ordinary skill in the art would have known that, controlling the robot enables the recited limitations supportively controlling the robot,…aligning the robot,…training the robot…monitoring the robot (see the Note 1 below). Therefore, in view of the teachings of Pannese, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, to modify the method for monitoring or controlling handling system of Tremblay and to include transporting and/or depositing a semiconductor wafer or substrate so that it enables the robot handling system of a semiconductor industry machine to operate robustly in unstructured, dynamic environments, performing tasks such as object grasping and manipulation, human-robot interaction, and collision detection and transition single wafers or groups of wafers between the interior environment and other areas of a fabrication facility as Pannese disclosed in para. [0050].
Note 1: the recited limitation “supportively controlling the robot, the moving element, the robot arm, the end effector, the positioning unit, or any combination thereof; aligning the robot, the moving element, the robot arm, the end effector, the positioning unit, or any combination thereof; training the robot, the moving element, the robot arm, the end effector, the positioning unit, or any combination thereof; monitoring the robot, the moving element, the robot arm, the end effector, the positioning unit, or any combination thereof; or any combination thereof” does not add any distinguishable weight in conjunction with the recited limitation “directly controlling the robot, the moving element, the robot arm, the end effector, the positioning unit, or any combination thereof” because, unless otherwise defined, directly controlling the robot would enable “supportively controlling the robot,…aligning the robot,…training the robot…monitoring the robot”, and are implied in the recited limitation directly controlling the robot. Therefore, Tremblay teaches directly controlling the robot, supportively controlling the robot,…aligning the robot,…training the robot…monitoring the robot. Furthermore, one of ordinary skill in the art would have known that “supportively controlling the robot,…aligning the robot,…training the robot…monitoring the robot” are enabled only after controlling a robot.
Regarding claim 2, Tremblay in view of Pannese teaches the recited limitations with respect to claim 1. Tremblay further teaches, the method according claim 1, wherein the capturing of the at least one image is obtained by an acquisition unit (camera 102, Fig. 1A) or obtained from a database (least some of input or inference data, may also be stored to a local database 620, para. [0067]).
Regarding claim 3, Tremblay does not teach the recited limitation. However, Pannese further teaches, the method according to claim 1, wherein the information data set contains information about a presence of an object in the at least one image, a position of an object in the at least one image, an orientation of an object in the at least one image, or any combination thereof (data may be processed to capture the time of a transition from wafer presence to wafer absence or, conversely, wafer absence to wafer presence, para. [0083]). Therefore, in view of the teachings of Pannese, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, to modify the method for monitoring or controlling handling system of Tremblay and to include transporting and/or depositing a semiconductor wafer or substrate so that it enables the robot handling system of a semiconductor industry machine transporting single wafers or groups of wafers between the interior environment and other areas of a fabrication facility transition.
Regarding claim 4, Tremblay does not teach the recited limitation. However, Pannese further teaches, the method according to claim 1, wherein the information data set contains information about a type of object in the at least one image, including a presence of: trays, cassettes, parts, markings, stickers, labels, reference marks, or any combination thereof (an operating system may gather data from sensors, robot drive encoders, pressure gauges, and the like, and use this acquired data…wafer cassettes, para. [0071]). Therefore, in view of the teachings of Pannese, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, to modify the method for monitoring or controlling handling system of Tremblay and to include transporting and/or depositing a semiconductor wafer or substrate so that it enables the robot handling system of a semiconductor industry machine transporting single wafers or groups of wafers between the interior environment and other areas of a fabrication facility transition.
Regarding claim 6, Tremblay in view of Pannese teaches the recited limitations with respect to claim 1. Tremblay further teaches, the method according to claim 1, wherein the information data set contains information about a presence of processing stations (determine a relative pose and orientation of any autonomous or semi-autonomous object, such as a robot or vehicle, para. [0055]).
Regarding claim 7, Tremblay in view of Pannese teaches the recited limitations with respect to claim 1. Tremblay further teaches, the method according to claim 1, wherein the information data set contains information about spacing between an at least one object in the at least one image to a reference point of the handling system (information can be used to determine a relative distance and orientation between a camera and a robot, as a base coordinate or other feature of this robot can be accurately identified in a camera space, or camera coordinate system, para. [0049]).
Regarding claim 9, Tremblay in view of Pannese teaches the recited limitations with respect to claim 1. Tremblay further teaches, the method according to claim 1, wherein the capturing of the at least one image includes storing the at least one image for use in at least one initial learning process or at least one new learning process, to improve a result of the artificial neural network (a captured image 202 of a robot can be provided as input to a trained neural network 204… these features can be learned through a training process, para. [0047]).
Regarding claim 10, Tremblay in view of Pannese teaches the recited limitations with respect to claim 1. Tremblay further teaches, the method according to claim 1, wherein geometric methods including triangulation, are used to determine the position and/or orientation and/or spacing and/or dimensions of the object, wherein data of the information data set generated by the trained artificial neural network is used (information can be used to determine a relative distance and orientation between a camera and a robot, as a base coordinate or other feature of this robot can be accurately identified, para. [0049], these synthetic images and data can then be provided 410 as training data to be used to train a neural network to infer feature position data for a specified type of robot, para. [0053, 0054]).
Regarding claim 11, Tremblay teaches, a handling system or machine (Figs. 1A to 7), the handling system or machine comprising:
a semi-conductor industry machine (para. [0133]), a robot (see Fig. 1A), a moving element, a robot arm an end effector, a positioning unit (see annotated Fig. 1A)), or any combination thereof;
a processing unit (para. [0068]) including at least one trained artificial neural network (a trained neural network 204, Fig. 2, para. [0044, 0047]); and an acquisition unit for capturing at least one image (a camera 102 might be used to capture images, or video, or an autonomous object, such as a robot 104…a captured image 202 of a robot can be provided as input to a trained neural network 204, Fig. 2, para. [0044, 0047]);
wherein the artificial neural network of the processing unit: captures at least one image in a digitized form depicting a location in or on the handling system or in the environment of the handling system; analyzes the at least one image; and generates an information data set and/or a control command (a neural network is to be trained for a specific type of robot, or autonomous object, a respective model and kinematic data can be provided as input, and control interface 308 can specify different poses for that robot…these synthetic images can then serve as training data for a neural network, as images include representations of a type of robot in specific poses, and corresponding pose data can serve as ground truth data for training….a tool can be used to generate these images that allows for scripting of robotic joint controls, para. [0050, 0056]); and
wherein the processing unit uses the information data set and/or the control command to: directly control the robot arm, the end effector, the positioning unit, or any combination thereof; supportively control the robot arm, the end effector, the positioning unit, or any combination thereof; align the robot arm, the end effector, the positioning unit, or any combination thereof; train the robot arm, the end effector, the positioning unit, or any combination thereof; monitor the robot arm, the end effector, the positioning unit, or any combination thereof (a neural network is to be trained for a specific type of robot, or autonomous object, a respective model and kinematic data can be provided as input, and control interface 308 can specify different poses for that robot…these synthetic images can then serve as training data for a neural network, as images include representations of a type of robot in specific poses, and corresponding pose data can serve as ground truth data for training, para. [0050], a tool can be used to generate these images that allows for scripting of robotic joint controls, para. [0056]);
pass on to a higher-level control system; pass on to a user who draws conclusions from this information for his actions operating the handling system or machine; pass this information to control systems or other users (context data for a user may also be stored to a user context data repository 622, which may include data about a user which may be useful as input to a network in generating inferences, or determining data to return to a user after obtaining instances, para. [0067]);
save for later or further evaluation; or any combination thereof (inference and/or training logic 915 may include, without limitation, code and/or data storage 901 to store forward and/or output weight and/or input/output data, and/or other parameters to configure neurons or layers of a neural network trained and/or used for inferencing, …any portion of code and/or data storage 901 may be included with other on-chip or off-chip data storage, including a processor's L1, L2, or L3 cache or system memory, para. [0096, 0257]).
Tremblay does not teach, processing substrates including semiconductor wafers, masks, flat panel displays. However, Pannese teaches a method for monitoring or controlling a handling system of a semiconductor industry machine in Figs. 1 to 10 including inputs to a plurality of neural networks; a robot (see Fig. 1, para. [0048]), a moving element, robot arm, an end effector, a positioning unit, or any combination thereof (Fig. 1, handling hardware 112 may include one or more robotic arms, transport carts, elevators, transfer stations and the like, as well as combinations of these, para. [0048]), having means for gripping, transporting and/or depositing a substrate (handling hardware 112 operates to manipulate wafers 104 within the system 100, such as by moving a wafer 104 between two of the process tools 110, or to/from the load lock 116, para. [0048]). Therefore, in view of the teachings of Pannese, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, to modify the method for monitoring or controlling handling system of Tremblay and to include semiconductor wafer or substrates so that it enables the robot handling system of a semiconductor industry machine to operate robustly in unstructured, dynamic environments, performing tasks such as object grasping and manipulation, human-robot interaction, and collision detection and transition single wafers or groups of wafers between the interior environment and other areas of a fabrication facility as Pannese disclosed in para. [0050].
Note 2: unless otherwise defined, the recited limitation “pass on to a user who draws conclusions from this information for his actions operating the handling system or machine; pass this information to control systems or other users; save for later or further evaluation; or any combination thereof” does not contribute any meaningful weight to the claim because a low-level machine such as a sensor or an alarm of a machine does the recited function.
Claim(s) 5 is rejected under 35 U.S.C. 103 as being unpatentable over Tremblay in view of Pannese as applied to claim 1 above, and further in view of Fukutome (US 20210150241).
Regarding claim 5, modified Tremblay does not teach, information data set contains information about possible obstacles. However, Fukutome teaches a semiconductor processing machine comprising an imaging device and a plurality of neural network processing units, in which, the method according to claim 1, wherein the information data set contains information about possible obstacles in a movement area of the handling system, including doors or load locks (an obstacle exists or not in the direction where the robot 2100 advances with the moving mechanism 2108, para. [0433]). Therefore, in view of the teachings of Fukutome, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, to modify the method for monitoring or controlling handling system of Tremblay and to include an obstacle detection system so that it enables the robot handling system to move safely by recognizing the surrounding environment using the processed image.
Claim(s) 8 is rejected under 35 U.S.C. 103 as being unpatentable over Tremblay in view of Pannese as applied to claim 1 above, and further in view of Koopman (WO 2019162204, see US 20210374936 for English Translation).
Regarding claim 8, modified Tremblay does not teach, the information data set contains information about dimensions of an object. However, Koopman teaches a method for monitoring or controlling handlings systems of a semiconductor industry machine, including providing a robot, capturing image and processing with artificial neural network, in which, the method according to claim 1, wherein the information data set contains information about dimensions of an object in the at least one image, wherein the object includes substrates and/or parts of substrates (one or more physical sizes, one or more physical dimensions, etc., para. [0117]). Therefore, in view of the teachings of Koopman, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, to modify the method for monitoring or controlling handling system of Tremblay and to include a physical size detection so that it enables to select desirable processing steps while manufacturing.
Conclusion
Prior art Kim (US 20200167632) teaches a method for monitoring or controlling handling systems, including providing a robot, capturing images and analyzing, with the artificial neural network, the at least one image; and generating, with the artificial neural network, an information data set and/or a control command.
Prior art Mariyama (US 20190375112) teaches a method for monitoring or controlling handling systems of a semiconductor industry machine, including providing a robot, capturing images and analyzing, with the artificial neural network, the at least one image; and generating, with the artificial neural network, an information data set and/or a control command.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSE K. ABRAHAM whose telephone number is (571)270-1087. The examiner can normally be reached Monday-Friday 8:30-4:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, THOMAS J. HONG can be reached at (571) 272-0993. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOSE K ABRAHAM/Examiner, Art Unit 3729
Prosecution Insights