DETAILED ACTION
This action is in response to communication filed on 24 December 2025. Claims 1-11 are amended. No claim has been added or canceled. Claims 1-11 are pending in the application and have been considered below.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant argues that [“As such, the applied references neither teach nor suggest "a processor configured to store history information obtained as the result of the processing in a storage device in response to the storage condition being determined to be satisfied, wherein the storage condition includes a first condition relating to a parameter applied for image processing for detecting the object from an image obtained by capturing the object by the visual sensor," as recited in claim 1. For at least the foregoing reasons, no combination of the applied references would have rendered claim 1 prima facie obvious” (Page 8)].
The argument described above has been considered, and are persuasive. Therefore, rejection has been withdrawn. However, upon further search and consideration, a new ground of rejection is made, citing the new reference USUDA et al. (US20200342267A1) (see claim 1 rejection below).
Response to Amendment
Based on applicant's amendment, the objection to claim 9 is withdrawn.
Based on applicant's amendment, the rejection of claim 3 under 35 U.S.C. 112(a) is withdrawn.
Based on applicant's amendment, the rejection of claim 9 under 35 U.S.C. 112(b) is withdrawn.
Based on applicant's amendment and response, claims 1-11 are no longer interpreted as invoking 35 USC§ 112(f)/sixth paragraph.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1 is rejected under 35 U.S.C. 103 as being unpatentable over WATANABE et al. (US20050071048A1) in view of USUDA et al. (US20200342267A1).
As to claim 1, WATANABE teaches:
a teaching device (see figs. 1-4, par. 0026, wherein the simulation unit is adapted to be able to load a robot program describing the operation of the robot through the communication line, and has a function to change a teaching content of the robot program on the simulation unit and then apply the changed robot program to the robot; as taught by WATANABE),
comprising: a processor configured to determine whether a storage condition related to a result of processing on an object by a visual sensor is satisfied (see figs. 1-4, pars. 0061-067; for example par. 0061, wherein in the visual sensor control unit 23, the processes executed by the visual sensor 21 and the used images are accumulated in the RAM or the nonvolatile RAM (hard disk, etc.) as a log. When an alarm occurs in the robot control unit 17, the visual sensor control unit 23 receives a notification of the alarm occurrence through the data transmitting means 25 and collects sensor history information including information associated with the latest executed visual sensor processing and related information associated with the previously executed visual sensor processing. The collected sensor history information is recorded in the RAM or the nonvolatile RAM; This sensor history information includes content of the visual sensor processing: the content includes, for example, history data representing a series of processes from a detection of a workpiece W up to a measurement of a position and an orientation of the workpiece W to be grasped; parameters used for the visual sensor processing: the parameters include, for example, an electronic shutter speed of the imaging means, a threshold value used for detecting a workpiece W, an affine transformation parameter value for a model image used in a matching method … Result of the visual sensor processing: the result includes, for example, a position and an orientation of the workpiece W to be held; see also fig. 5, pars. 0056-0068; as taught by WATANABE);
and store history information obtained as the result of the processing in a storage device in response to the storage condition being determined to be satisfied (see fig. 5, steps E1 to E11, for example par. 0086, wherein in Step E7: The image, the content of the process, the parameters, the result of the processing, the time, etc. are recorded as measurement processing information, and the process is returned to step E1; see also par. 0090, wherein in Step E11: The robot history information and the executed command information are grouped and recorded with the visual sensor history information selected and stored in step E10. The alarm ID determined in step D1 can be used for this purpose. When the robot history information, the executed command information and the visual sensor history information are recorded, the process is completed; as taught by WATANABE).
WATANABE does not expressly teach wherein the storage condition includes a first condition relating to a parameter applied for image processing for detecting the object from an image obtained by capturing the object by the visual sensor.
In similar field of endeavor, USUDA teaches wherein the storage condition includes a first condition relating to a parameter applied for image processing for detecting the object from an image obtained by capturing the object by the visual sensor (see figs. 1-11 and 17, e.g. fig. 1, par. 0104, wherein the recognition result correction unit 16 performs processing of correcting the recognition result of the image recognition unit 14 in accordance with the instruction from the user. Data corrected by the recognition result correction unit 16 is assigned a label “correction made” and stored in the learning image storage unit 20. That is, the recognition result correction unit 16 transmits data including an “erroneously determined image” erroneously recognized by the image recognition unit 14 and a supervisory signal indicating the “correct answer” provided by the user to the learning image storage unit 20; see also pars. 0020-0023 and 0105-0110; as taught by USUDA).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the WATANABE apparatus to include the teachings of USUDA wherein the storage condition includes a first condition relating to a parameter applied for image processing for detecting the object from an image obtained by capturing the object by the visual sensor. Such a person would have been motivated to make this combination as it is beneficial to the user to have the convenience and flexibility of storing data where it can be used later on as training data for the device.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over WATANABE et al. (US20050071048A1) in view of USUDA et al. (US20200342267A1) and further view of KOYAMA et al. (US20180136634A1).
As to claim 2, WATANABE and USUDA teach the limitations of claim 1. WATANABE and USUDA do not teach wherein the storage condition further includes a second condition specifying a storage destination for saving the history information, and the processor is configured to save the history information to the storage destination specified by the storage condition.
In similar field of endeavor, KOYAMA teaches wherein the storage condition further includes a second condition specifying a storage destination for saving the history information, and the processor is configured to save the history information to the storage destination specified by the storage condition (see figs. 6A-7C, par. 0044, wherein the setting data is information in which there are correlated or associated with each other, information (names) indicative of the captured image data, types of applications in which the captured image data is used, and location information (e.g., file path information) such as address information indicative of a storage location in the controller 12; as taught by KOYAMA).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the WATANABE and USUDA apparatus to include the teachings of KOYAMA wherein the storage condition further includes a second condition specifying a storage destination for saving the history information, and the processor is configured to save the history information to the storage destination specified by the storage condition. Such a person would have been motivated to make this combination as it is beneficial for the user to have the flexibility of storing the data where it is desired so it can be accessed later based on policies.
Claims 3, 7-8 and 10-11 are rejected under 35 U.S.C. 103 as being unpatentable over WATANABE et al. (US20050071048A1) in view of USUDA et al. (US20200342267A1) and further view of TAKAHASHI et al. (US20170220008A1).
As to claim 3, WATANABE and USUDA teach the limitations of claim 1. WATANABE and USUDA do not expressly teach wherein the storage condition further includes a third condition that specifies, among the history information, a target item to be stored, and the processor is configured to store the target item to be stored.
In similar field of endeavor, TAKAHASHI teaches wherein the storage condition further includes a third condition that specifies, among the history information, a target item to be stored, and the processor is configured to store the target item to be stored (see figs. 1-2, par. 0045, wherein inside information of each of the manufacturing machines 25 to 28 is the information stored in a memory of each manufacturing machine, and includes at least one of a drive parameter, a function parameter, an operation program, and an operation command log. Note that the memory for storing the inside information is not needed to be provided within each manufacturing machine as in the present embodiment. The memory may be provided out of each manufacturing machine, or may be provided in the cell controller 12; see also par. 0057, wherein the database 22 is configured to store the state of an abnormality that has occurred in each manufacturing machine and a cause of the occurrence of the abnormality, which are correlated with each other. This enables the abnormality cause finding unit 16 to find, with reference to an abnormality cause management database, a cause of the abnormality which occurs in either the first manufacturing machine or the second manufacturing machine, or a cause of an abnormality which may occur in the future; as taught by TAKAHASHI).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the WATANABE and USUDA apparatus to include the teachings of TAKAHASHI wherein the storage condition further includes a third condition that specifies, among the history information, a target item to be stored, and the processor is configured to store the target item to be stored. Such a person would have been motivated to make this combination as it is beneficial to have a separate database for outliers. This will help to improve productivity as there could be an alert sent when a new record arrives in the dedicated database so that the cause of the abnormality could be determined to quickly to recover the manufacturing machine (see also TAKAHASHI, pars. 0001-0012).
As to claim 7, WATANABE and USUDA teach the limitations of claim 1. WATANABE and USUDA do not expressly teach wherein the processor is configured to learn the storage condition based on the history information, and apply the storage condition obtained through learning.
In similar field of endeavor, TAKAHASHI teaches:
wherein the processor is configured to learn the storage condition based on the history information (see par. 0028, wherein the cell controller according to the ninth aspect further includes a learning instrument which is configured to perform machine learning using the information stored in the database, the acquired inside information of each of the manufacturing machines, and the device configuration information representing components of each of the manufacturing machines, in order to update the information stored in the database; as taught by TAKAHASHI),
and apply the storage condition obtained through learning (see par. 0059, wherein the learning instrument 24 is provided in the database updating unit 23. The learning instrument 24 learns a combination of the state of the abnormality which has occurred in each manufacturing machine with a cause of the occurrence of the abnormality, which is to be stored in the database 22; as taught by TAKAHASHI).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the WATANABE and USUDA apparatus to include the teachings of TAKAHASHI wherein the processor is configured to learn the storage condition based on the history information, and apply the storage condition obtained through learning. Such a person would have been motivated to make this combination as it is beneficial to have use a learning instrument to teach new states and situations to the device that can be referred to again and again.
As to claim 8, WATANABE, USUDA and TAKAHASHI teach the limitations of claim 7. TAKAHASHI further teaches wherein the processor is configured to perform a first learning using first teacher data including the history information as an input of the first learning and information indicating whether the history information has been saved as an output label of the first learning (see figs. 3-4,pars. 0082-0114, for example see par. 0083, wherein “Supervised learning” is a method in which a large volume of input-output (label) paired data are given to a machine learning apparatus, so that characteristics of these datasets can be learned, and a model for inferring an output value from input data, i.e., the input-output relation can be inductively acquired. This can be achieved using an algorithm, for example, a neural network that will be described later; see also par. 0113, wherein supervised learning is applied to the learning instrument 24, the value functions correspond to learning models, and the rewards correspond to errors; as taught by TAKAHASHI).
WATANABE further teaches and apply a first learning model obtained by the first learning as the storage condition (see pars. 0062-0067, for example see par. 0064, wherein parameters used for the visual sensor processing: the parameters include, for example, a electronic Shutter Speed of the imaging means, a threshold value used for detecting a workpiece W, an affine transformation parameter value for a model image used in a matching method; as taught by WATANABE).
As to claim 10, WATANABE and USUDA teach the limitations of claim 1. WATANABE and USUDA do not expressly teach wherein the processor is configured to detect whether there is an outlier in predetermined data included in the history information, and use whether the outlier is detected as the storage condition.
In similar field of endeavor, TAKAHASHI teaches wherein the processor is configured to detect whether there is an outlier in predetermined data included in the history information (see figs. 1-2, par. 0013, wherein the present invention provides a cell controller which efficiently finds an abnormality caused by the inside information of a manufacturing machine; as taught by TAKAHASHI), and use whether the outlier is detected as the storage condition (see figs. 1-2, par. 0057, wherein the database 22 is configured to store the state of an abnormality that has occurred in each manufacturing machine and a cause of the occurrence of the abnormality, which are correlated with each other. This enables the abnormality cause finding unit 16 to find, with reference to an abnormality cause management database, a cause of the abnormality which occurs in either the first manufacturing machine or the second manufacturing machine, or a cause of an abnormality which may occur in the future; as taught by TAKAHASHI).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the WATANABE and USUDA apparatus to include the teachings of TAKAHASHI wherein the processor is configured to detect whether there is an outlier in predetermined data included in the history information, and use whether the outlier is detected as the storage condition. Such a person would have been motivated to make this combination as it is beneficial to have a database for outliers. This will help to improve productivity as there could be an alert sent when a new record arrives in the dedicated database so that the cause of the abnormality could be determined to quickly to recover the manufacturing machine (see also TAKAHASHI, pars. 0001-0012).
As to claim 11, WATANABE, USUDA teach and TAKAHASHI teach the limitations of claim 10. TAKAHASHI further teaches wherein the is configured to save the history information to a predetermined storage destination in response to the outlier being detected (see figs. 1-2, par. 0057, wherein the database 22 is configured to store the state of an abnormality that has occurred in each manufacturing machine and a cause of the occurrence of the abnormality, which are correlated with each other. This enables the abnormality cause finding unit 16 to find, with reference to an abnormality cause management database, a cause of the abnormality which occurs in either the first manufacturing machine or the second manufacturing machine, or a cause of an abnormality which may occur in the future; as taught by TAKAHASHI).
Claims 9 is rejected under 35 U.S.C. 103 as being unpatentable over WATANABE et al. (US20050071048A1) in view of USUDA et al. (US20200342267A1) and further view of TAKAHASHI et al. (US20170220008A1) and further view of KOYAMA et al. (US20180136634A1)..
As to claim 9, WATANABE, USUDA and TAKAHASHI teach the limitations of claim 8. TAKAHASHI further teaches wherein the processor is configured to further perform a second learning using second teacher data including the history information as an input of the second learning and a storage destination of the history information as an output label of the second learning (see figs. 3-4,pars. 0082-0114, for example see par. 0083, wherein “Supervised learning” is a method in which a large volume of input-output (label) paired data are given to a machine learning apparatus, so that characteristics of these datasets can be learned, and a model for inferring an output value from input data, i.e., the input-output relation can be inductively acquired. This can be achieved using an algorithm, for example, a neural network that will be described later; see also par. 0113, wherein supervised learning is applied to the learning instrument 24, the value functions correspond to learning models, and the rewards correspond to errors; as taught by TAKAHASHI).
WATANABE, USUDA and TAKAHASH do not teach and apply a second learning model obtained by the second learning to determine a storage destination when storing the history information.
In similar field of endeavor, KOYAMA teaches and apply a second learning model obtained by the second learning to determine a storage destination when storing the history information (see figs. 6A-7C, par. 0044, wherein the setting data format rule is a format rule for the setting data so as to enable the captured image data to be used on the side of the controller 12. In order to be capable of using the captured image data on the side of the controller 12, the setting data is defined by setting information for the purpose of setting (storing) the captured image data in the controller 12 (more specifically, in the image storage unit 26 a thereof), or setting information of captured information that is set (stored) in the controller 12. The setting data is information in which there are correlated or associated with each other, information (names) indicative of the captured image data, types of applications in which the captured image data is used, and location information (e.g., file path information) such as address information indicative of a storage location in the controller 12; as taught by KOYAMA).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the WATANABE, USUDA and TAKAHASH apparatus to include the teachings of KOYAMA and apply a second learning model obtained by the second learning to determine a storage destination when storing the history information. Such a person would have been motivated to make this combination as it is beneficial for the user to have the flexibility of storing the data where it is desired so it can be accessed later based on policies.
Claims 4 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over WATANABE et al. (US20050071048A1) in view of USUDA et al. (US20200342267A1) and further view IMAI et al. (US20170282428A1).
As to claim 4, WATANABE and USUDA teach the limitations of claim 1. WATANABE and USUDA do not expressly teach wherein the processor is configured to set the storage condition.
In similar field of endeavor, IMAI teaches wherein the processor is configured to set the storage condition (see fig. 1, par. 0008, wherein an image storage condition setting unit configured to set a condition used for determining whether to store the image of the molded product obtained by the molded product image acquisition unit; as taught by IMAI).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the WATANABE and USUDA apparatus to include the teachings of IMAI wherein the processor is configured to set the storage condition. Such a person would have been motivated to make this combination as it provides a system capable of efficiently storing image data of molded products while avoiding an increase of storage capacity of storage means necessary for data storage (see IMAI, par. 0006).
As to claim 6, WATANABE, KOYAMA and IMAI teach the limitations of claim 4. IMAI further teaches wherein the processor is configured to present a user interface for setting the storage condition on a display and accept setting of the storage condition via the user interface (see fig. 6, par. 0029, wherein the controller 4 includes a display device and manual input means through which various kinds of set values are input, and controls the injection molding main body 3 according to the set values and a control program. The controller 4 also includes a physical quantity acquisition unit 10 a, an image storage condition setting unit 11, and an image storage determination unit 12; see also fig. 3, par. 0034, wherein the image storage condition setting unit 11 may set the conditions for data storage in such a way as to, as shown in FIG. 2, calculate a deviation value by calculating statistical values on each kinds of the physical quantities based on multiple pieces of physical quantity data 20 obtained in past molding cycles, and lead the image storage determination unit 12 to store the molded product image data 21 of the molded product when the deviation value deviates from a predetermined range. The image storage condition setting unit 11 may also set the conditions for data storage in such a way as to, as shown in FIG. 3, preliminarily set a plurality of monitoring ranges regarding the respective physical quantities (such as a monitoring range of physical quantity and a deviation value), and lead the image storage determination unit 12 to determine whether to store the molded product image data 21 in the molded product image storage unit 13 depending on the respective monitoring ranges; as taught by IMAI).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over WATANABE et al. (US20050071048A1) in view of USUDA et al. (US20200342267A1) and further view IMAI et al. (US20170282428A1) and further view of SCHMIDT et al. (US20080178092A1).
As to claim 5, WATANABE, USUDA and IMAI teach the limitations of claim 4. WATANABE, USUDA and IMAI do not expressly teach wherein the processor is configured to accept setting of the storage condition by a text-based command.
In similar field of endeavor, SCHMIDT teaches wherein the processor is configured to accept setting of the storage condition by a text-based command (see par. 0013, wherein a system for monitoring processes of a distributed business application includes an integration builder tool comprising a condition editor. The condition editor has a user interface providing an editing area on a display in which to receive a text-based expression of one or more conditions of a business application process. The editing area further includes an operator palette providing a plurality of operators for use in the text based expression of each of the one or more conditions, and a toolbar providing one or more editing tools for editing the text-based expressions; as taught by SCHMIDT).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the WATANABE, USUDA and IMAI apparatus to include the teachings of SCHMIDT wherein the processor is configured to accept setting of the storage condition by a text-based command. Such a person would have been motivated to make this combination as there is a need to have an extensive, easy to use and more readable condition editor UI where more experienced users can easily enter commands without having to go through graphics menus that could be cumbersome (see also SCHMIDT, par. 0011).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Publication Number
Filing Date
Title
US20080086432A1
2007-05-23
Data classification methods using machine learning techniques
US20200184270A1
2020-02-17
Automatically tagging images to create labeled dataset for training supervised machine learning models
US11922319B2
2019-10-24
Image determination device, training method and non-transitory computer readable medium storing program
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KOOROSH NEHCHIRI whose telephone number is (408)918-7643. The examiner can normally be reached M-F, 11-7 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, William L. Bashore can be reached at 571-272-4088. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KOOROSH NEHCHIRI/Examiner, Art Unit 2174
/WILLIAM L BASHORE/ Supervisory Patent Examiner, Art Unit 2174