DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
National Stage
Examiner acknowledges that the instant application is a 371 national stage entry to PCT/JP2021/044960 filed 12/07/2021. As such, the effective filing date of the instant claims is the filing date of that PCT application, 12/07/2021.
Status of Claims
The amendment filed 06/09/2026 has been entered. Claims 1 and 10 have been amended. Claim 11 has been canceled. Claims 1-10 are now pending.
Response to Arguments
Applicant’s arguments with respect to the 35 USC 102 and 35 USC 103 rejections set forth in the previously mailed office action have been considered but are moot because the amendments to the claim language have necessitated new grounds of rejection set forth below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-3 and 9-10 are rejected under 35 U.S.C. 103 as being unpatentable over Gonzalez et al. (US 20200324409 A1), hereinafter Gonzalez in view of Fujishima et al. (US 20160239019 A1), hereinafter Fujishima.
Regarding claim 1, Gonzalez discloses:
A device configured to verify an operation of an industrial machine that controls the operation based on detection data of a sensor (see at least [0022]: “Turning now to FIG. 1, an enhanced environmental analysis and robotic end effector control process 100 is illustrated. A robot may include sensor array 102, map and semantics generator 104, robot modeler 106, mission planner 108, end effector controller 128 and robotic end effector 132 (e.g., multi-fingered robot end-effectors).”)
the device comprising a processor configured to:
acquire the detection data detected by the sensor when executing an operation program including a plurality of instructions for causing the industrial machine to perform a plurality of the operations respectively (see at least [0123]: “Example 1 includes a computing system comprising one or more sensors to generate sensor data, the sensor data to include image data, a processor coupled to the one or more sensors, and a memory including a set of executable program instructions, which when executed by the processor, cause the computing system to generate a semantic labelled image based on image data from the sensor data, wherein the semantic labelled image is to identify a shape of an object and a semantic label of the object, associate a first set of actions with the object and generate a plan based on an intersection of the first set of actions and a second set of actions to satisfy a command from a user through actuation of one or more end effectors, wherein the second set of actions are to be associated with the command.”)
Gonzalez does not explicitly teach, but Fujishima, in an analogous field of endeavor teaches:
and specifying information for specifying the detection data; and store the specifying information in a data storage location designated by a register code included in the executed instruction to associate the executed instruction and the detection data used for control of the operation performed by the executed instruction with each other via the register code and the specifying information (see at least [0062]: “Next, an example of an input screen 50 displayed on the display unit 22 of the operation terminal 10 will be described with reference to FIG. 2. The input screen 50 illustrated in FIG. 2 serves as an input screen 50 for inputting an instruction to the mobile search device 11. In other words, the input screen 50 serves as an input screen 50 for inputting a search object and a predetermined action. A search object input field 51 and an action input field 52 are displayed on this input screen 50. The search object input field 51 is a field for inputting a search object and displays search objects registered in advance or a search object to be newly registered as selectable options. The action input field 52 is a field for inputting a predetermined action and displays predetermined actions registered in advance or a predetermined action to be newly registered as selectable options. The input of a search object and the input of a predetermined action are not particularly limited and a predetermined text string may be input to the respective fields 51 and 52 for a search. Once a search object and a predetermined action are input on the input screen 50, the terminal-side controller 25 transmits the input search object and predetermined action to the mobile search device 11 as the object data and the action data, respectively.”)
It would have been prima facie obvious for one of ordinary skill in the art before the effective filing date of the claimed invention, with a reasonable expectation for success, to combine the invention of Gonzalez with the data registration as taught by Fujishima. This is because as described in the technical problem section of Fujishima: “As described above, while autonomously moving, a remote control robot device disclosed in Patent Literature 1 avoids an obstacle as an action provided in advance, in a case where no remote control is given by an operator. Here, while a mobile search device such as the remote control robot device is autonomously moving, in order to cause the mobile search device to carry out an action other than the avoidance of an obstacle, an instruction for carrying out a predetermined action relative to a predetermined object needs to be provided in advance. For example, when an instruction is provided for the autonomous movement of the mobile search device, it is typically considered to provide an action program for carrying out a predetermined action relative to a predetermined object (e.g., an action program for gripping a drill) as an executive instruction (e.g., an instruction for gripping a drill). In this case, however, because the instruction is provided for carrying out the predetermined action relative to the predetermined object, when an object differs from the predetermined one, the predetermined action cannot be carried out relative to that different object. Taking this into account, the action programs and the executive instructions need to be prepared by the number of combinations of predetermined objects and predetermined actions. Therefore, it is difficult to use for multiple purposes. Considering this situation, an object of the present invention is to provide an autonomous search system, an operation terminal, a mobile search device, and a search control method that can be used for multiple purposes.”
Regarding claim 2, the combination of Gonzalez and Fujishima teaches the device of claim 1.
Gonzalez further discloses wherein the processor is further configured to acquire the detection data as history data in which the detection data detected when executing the operation program is stored in chronological order (see at least [0080]: “The mission planner 390 may capture and unfold high-level directives from sensor data provided by the sensor array 386 (e.g., “clean the kitchen”). The mission planner 390 may decompose the directive into a fine granular sequence of physical atomic-actions or tasks (e.g., primary task, secondary task, target object part assertion, affordance list, etc.) to accomplish the high level directive. The tasks may be stored in the task information 392.”)
Regarding claim 3, the combination of Gonzalez and Fujishima teaches:
The device of claim 1.
Gonzalez further discloses wherein the industrial machine includes:
the sensor configured to detect a workpiece by imaging the workpiece (see at least [0047]: “Imaging and/or range sensors 302 may provide sensor data 336 to the scene semantic spatial context generator 304. The sensor data may include imaging data (e.g., RGB-D data) and/or range data. Imaging sensors of the imaging and/or range sensors 302 may be devices contained within a composed sensor (e.g., RGB-D camera or camera module). For example, the imaging and/or range sensors 302 may provide three data streams capturing information regarding a content in a field-of view and the time-varying 6D pose of one or more objects.”)
and a robot configured to carry out a predetermined work on the workpiece by performing the operation (see at least [0039]: “The above process 100 may empower autonomous service robots to perform real-world physical-interaction tasks generating and capturing value in semi-structured environments.”)
wherein the operation program includes:
a first operation program including the instruction for causing the sensor to perform the operation of detecting the detection data by imaging the workpiece and a second operation program including the instruction for causing the robot to perform the operation for the predetermined work, based on the detection data detected by executing the first operation program wherein the processor is further configured to associate the instruction included in the second operation program and the detection data used for control of the operation for the predetermined work performed by the instruction included in the second operation program with each other (see at least [0123]: “Example 1 includes a computing system comprising one or more sensors to generate sensor data, the sensor data to include image data, a processor coupled to the one or more sensors, and a memory including a set of executable program instructions, which when executed by the processor, cause the computing system to generate a semantic labelled image based on image data from the sensor data, wherein the semantic labelled image is to identify a shape of an object and a semantic label of the object, associate a first set of actions with the object and generate a plan based on an intersection of the first set of actions and a second set of actions to satisfy a command from a user through actuation of one or more end effectors, wherein the second set of actions are to be associated with the command.”)
Regarding claim 9, the combination of Gonzalez and Fujishima teaches the device of claim 1.
Gonzalez further discloses an industrial machine comprising the device of claim 1 (see at least [0039]: “The above process 100 may empower autonomous service robots to perform real-world physical-interaction tasks generating and capturing value in semi-structured environments.”)
Regarding claim 10, Gonzalez discloses:
A method of verifying an operation of an industrial machine that controls the operation based on detection data of a sensor, the method comprising:
acquiring, by a processor, the detection data detected by the sensor when executing an operation program including a plurality of instructions for causing the industrial machine to perform a plurality of the operations respectively; (see at least [0123]: “Example 1 includes a computing system comprising one or more sensors to generate sensor data, the sensor data to include image data, a processor coupled to the one or more sensors, and a memory including a set of executable program instructions, which when executed by the processor, cause the computing system to generate a semantic labelled image based on image data from the sensor data, wherein the semantic labelled image is to identify a shape of an object and a semantic label of the object, associate a first set of actions with the object and generate a plan based on an intersection of the first set of actions and a second set of actions to satisfy a command from a user through actuation of one or more end effectors, wherein the second set of actions are to be associated with the command.”)
and specifying information for specifying the detection data; and storing, by the processor, the specifying information in a data storage location designated by a register code included in the executed instruction to associate the executed instruction and the detection data used for control of the operation performed by the executed instruction with each other via the register code and the specifying information (see at least [0062]: “Next, an example of an input screen 50 displayed on the display unit 22 of the operation terminal 10 will be described with reference to FIG. 2. The input screen 50 illustrated in FIG. 2 serves as an input screen 50 for inputting an instruction to the mobile search device 11. In other words, the input screen 50 serves as an input screen 50 for inputting a search object and a predetermined action. A search object input field 51 and an action input field 52 are displayed on this input screen 50. The search object input field 51 is a field for inputting a search object and displays search objects registered in advance or a search object to be newly registered as selectable options. The action input field 52 is a field for inputting a predetermined action and displays predetermined actions registered in advance or a predetermined action to be newly registered as selectable options. The input of a search object and the input of a predetermined action are not particularly limited and a predetermined text string may be input to the respective fields 51 and 52 for a search. Once a search object and a predetermined action are input on the input screen 50, the terminal-side controller 25 transmits the input search object and predetermined action to the mobile search device 11 as the object data and the action data, respectively.”)
It would have been prima facie obvious for one of ordinary skill in the art before the effective filing date of the claimed invention, with a reasonable expectation for success, to combine the invention of Gonzalez with the data registration as taught by Fujishima. This is because as described in the technical problem section of Fujishima: “As described above, while autonomously moving, a remote control robot device disclosed in Patent Literature 1 avoids an obstacle as an action provided in advance, in a case where no remote control is given by an operator. Here, while a mobile search device such as the remote control robot device is autonomously moving, in order to cause the mobile search device to carry out an action other than the avoidance of an obstacle, an instruction for carrying out a predetermined action relative to a predetermined object needs to be provided in advance. For example, when an instruction is provided for the autonomous movement of the mobile search device, it is typically considered to provide an action program for carrying out a predetermined action relative to a predetermined object (e.g., an action program for gripping a drill) as an executive instruction (e.g., an instruction for gripping a drill). In this case, however, because the instruction is provided for carrying out the predetermined action relative to the predetermined object, when an object differs from the predetermined one, the predetermined action cannot be carried out relative to that different object. Taking this into account, the action programs and the executive instructions need to be prepared by the number of combinations of predetermined objects and predetermined actions. Therefore, it is difficult to use for multiple purposes. Considering this situation, an object of the present invention is to provide an autonomous search system, an operation terminal, a mobile search device, and a search control method that can be used for multiple purposes.”
Claims 4 and 5 are rejected under 35 U.S.C. 103 as being unpatentable over Gonzalez and Fujishima further in view of Kang et al. (US 20180345557 A1), hereinafter Kang.
Regarding claim 4, the combination of Gonzalez and Fujishima teaches the device of claim 1.
Gonzalez does not explicitly disclose but Kang, in an analogous field of endeavor, teaches:
wherein the operation program includes:
a first instruction for calculating a correction amount for correcting the operation of the industrial machine, using the detection data and a second instruction for correcting the operation of the industrial machine in accordance with the correction amount calculated by executing the first instruction, wherein the association generating unit is configured to associate the first instruction or the second instruction and the detection data with each other via the correction amount (see at least [0052]: “If error data is detected from the operation data of the take-out robots 110 and the information data on the injection molding received from the main sever 220 on the production site, further, the control pendent 300 analyzes the error data, corrects, adds or deletes the operation data of the take-out robots 110 and the information data on the injection molding through the connection to the main server 220, and automatically recovers the trouble of the injection molding machine 100, on the basis of the previously stored remote control access program stored in the memory unit.”)
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, with a reasonable expectation for success, to combine the invention of Gonzalez with the operation correction as taught by Kang. This is because, as stated by Kang [0006] of traditional industrial techniques: “… the conventional systems do not integrally control the processes, perfectly, and accordingly, they cannot immediately recognize and handle the malfunctions of the take-out robot and unexpected accidents happened by a worker's mistake, thereby undesirably reducing productivity, causing difficulties in production management and inconveniences in control, and remarkably lowering an industrial safety level.”
Regarding claim 5, the combination of Gonzalez, Fujishima and Kang teaches the device of claim 4.
Gonzalez does not explicitly disclose but Kang, in an analogous field of endeavor, teaches:
wherein the operation program includes:
wherein the first instruction or the second instruction includes a register code representing a data storage location of the calculated correction amount, wherein the association generating unit is configured to: acquire information for specifying the detection data used for calculation of the correction amount, and associate the first instruction or the second instruction and the detection data with each other via the register code, the correction amount, and the information (see at least [0052]: “If error data is detected from the operation data of the take-out robots 110 and the information data on the injection molding received from the main sever 220 on the production site, further, the control pendent 300 analyzes the error data, corrects, adds or deletes the operation data of the take-out robots 110 and the information data on the injection molding through the connection to the main server 220, and automatically recovers the trouble of the injection molding machine 100, on the basis of the previously stored remote control access program stored in the memory unit.”)
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, with a reasonable expectation for success, to combine the invention of Gonzalez with the operation correction as taught by Kang. This is because, as stated by Kang [0006] of traditional industrial techniques: “… the conventional systems do not integrally control the processes, perfectly, and accordingly, they cannot immediately recognize and handle the malfunctions of the take-out robot and unexpected accidents happened by a worker's mistake, thereby undesirably reducing productivity, causing difficulties in production management and inconveniences in control, and remarkably lowering an industrial safety level.”
Claims 6-8 are rejected under 35 U.S.C. 103 as being unpatentable over Gonzalez and Fujishima in view of Einecke (US 20150128547 A1), hereinafter Einecke.
Regarding claim 6, the combination of Gonzalez and Fujishima teaches the device of claim 1.
Gonzalez does not explicitly disclose, but Kang, in an analogous field of endeavor teaches:
an input receiving unit configured to receive an input for selecting one of the plurality of instructions, and a data output unit configured to output the detection data associated with the one of the plurality of instructions by the association generating unit, in response to the input for selecting the one of the plurality of instructions, received by the input receiving unit (see at least [0059]: “According to a further aspect of the invention a software program product in the form of e.g. a mobile application is proposed. The software program product is adapted to be installed on a remote smart device for operating an autonomous robot, such as an autonomous lawn mower. The software program product is adapted to control a communication interface for receiving an input image, a display for displaying the input image, and input means for allowing a user to input or select a remote control instruction. Therein the software program product is adapted to control the communication interface so as to transmit the remote control instruction inputted or selected via the input means.”)
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, with a reasonable expectation for success, to combine the invention of Gonzalez with the input collection as taught by Einecke. This is because, as discussed within the background of Einecke: “In view of the above-mentioned disadvantages of known autonomous mowers, the present invention is intended to improve the state of the art. A particular aim of the present invention is to provide images to a remote smart device of the user, like a smart phone, a tablet or any other user device. The lawn mower can stream camera images to the remote smart device to improve remote control and particularly to help the user to understand the situation of the mower.” One of ordinary skill in the robotics and industrial systems art would understand that the discussed improvements to household robotics could easily be applied to industrial robotics as well.
Regarding claim 7, the combination of Gonzalez, Fujishima, and Einecke teaches the device of claim 6.
Gonzalez further discloses wherein the device further comprises:
an image generating unit configured to generate setting image data for setting an operation parameter of the industrial machine, wherein the data output unit is configured to output image data of the detection data to the image generating unit, wherein the image generating unit is configured to generate the setting image data in which the image data of the detection data is displayed (see at least [0054]: “The free and occupied map generator 316 may generate sparse dual-space map that may capture and split the occupied and unfilled (free) spaces. This mapping may allow for: i) registering diverse 3D images while exploring various interaction (e.g., grasping) scenarios for a kinematic end effector, ii) determine possible collision-free manipulator 6D poses in the environment and iii) serve as an effective scaffolding data structure to store multiresolution local surface descriptors such as volumetric (e.g., with respect to voxels) semantic labels and other attributes.”)
Regarding claim 8, the combination of Gonzalez, Fujishima and Einecke teaches the device of claim 7.
Gonzalez does not explicitly disclose, but Einecke, in an analogous field of endeavor teaches:
wherein the input receiving unit further receives an input for changing the operation parameter through the setting image data, wherein the device further includes a parameter setting unit configured to change the preset operation parameter in response to the input for changing the operation parameter (see at least [0058]: “The remote control instruction may correspond, as mentioned above, to an instruction regarding the moving means of the autonomous robot, and preferably regarding the driving (move forward or backward) and steering (turn left or right) means. The remote control instruction may also relate to the working means and e.g. the tool, e.g. activate or deactivate the working means or modify a working parameter of the tool. The remote control instruction may also relate to a parameter of the camera, such as the exposure, aperture, shutter, hue, zoom or focus, or to control the orientation of the camera.”)
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, with a reasonable expectation for success, to combine the invention of Gonzalez with the input collection as taught by Einecke. This is because, as discussed within the background of Einecke: “In view of the above-mentioned disadvantages of known autonomous mowers, the present invention is intended to improve the state of the art. A particular aim of the present invention is to provide images to a remote smart device of the user, like a smart phone, a tablet or any other user device. The lawn mower can stream camera images to the remote smart device to improve remote control and particularly to help the user to understand the situation of the mower.” One of ordinary skill in the robotics and industrial systems art would understand that the discussed improvements to household robotics could easily be applied to industrial robotics as well.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ELIZABETH NELESKI whose telephone number is (571)272-6064. The examiner can normally be reached 10 - 6.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, THOMAS WORDEN can be reached at (571) 272-4876. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JASON HOLLOWAY/
Primary Examiner, Art Unit 3658
/E.R.N./Examiner, Art Unit 3658
Prosecution Insights