DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed on 03/17/2026, has been received and made of record. In response to the Non-Final Office Action, dated on 01/02/2026. Claims 1-9, 11 and 15-17 are pending in the current application. Claim 17 is newly added.
Response to Arguments
Applicant’s arguments filed on 03/17/2026 have been fully considered.
In the Arguments/Remarks:
Re: Claim Interpretation
Applicant’s remarks have been fully considered, however are unpersuasive. Claim interpretation under 35 U.S.C. 112(f) is maintained.
Re: Rejection of the Claims Under 35 U.S.C. 103
Applicant’s arguments regarding rejection of the claims under 35 U.S.C. 103 have been fully considered. For the sake of brevity, examiner will address the arguments in regards to independent claim 1, and will also apply to independent claim 11 and the corresponding dependent claims.
After further consideration examiner submits that under the broadest reasonable interpretation (BRI) Sato (US 2016/0334777 A1) discloses or suggests the newly amended claim limitation “and generate a processed image in which a related area is extracted, the extracted related area being an area along at least part of a contour of a tool satisfying the tool condition”. Sato discloses in paragraph 9 “A numerical controller according to the present invention is configured to control a machine tool according to an NC program to machine a workpiece. The machine tool includes an automatic tool changer and an image pickup unit configured to capture an image of a tool. The numerical controller includes: a tool management data storage unit configured to store tool management data that includes data on at least a shape and a size of the tool associated with a tool number” and in paragraph 20 “a numerical controller for controlling a machine tool, which is provided with a camera secured around a tool change position or a camera configured to be moved around the tool change position by a robot, comprises means for retrieving and executing only a tool change command in an NC program, means for controlling the camera to capture an image of a tool every time the tool change command is executed, and means for analyzing the captured image to calculate a shape and a size of the tool. The calculated tool shape and size are collated with a shape and a size in tool management data stored in the numerical controller, and the result of the collation is displayed on a screen of a display device so that an operator can be informed of it.”. Examiner submits that Sato discloses a camera that captures an image of a tool and the image is then analyzed to calculate the shape and size of the tool. Sato depicts in Fig.4 that the processed imaged is being analyzed with the related area to determine/calculate the shape and size of the imaged tool. This surrounding area is along the contour of the tool. The shape and size of the tool once calculated are collated to determine if the tool matches the tool condition [(see at least Sato paragraph 41 “The collating means 130 reads the tool shape (tool type) and size corresponding to the currently selected tool number (or tool number commanded by the tool change command read from the NC program by the tool change command execution means 100) from the tool management data storage unit 210. Then, the collating means 130 collates the read tool shape and size with the tool shape and size data received from the tool data calculation means 120 and outputs the result of the collation to the display means 140. In the collation processing, it is determined that the changed tool is identical to the managed tool if the two tool shapes are identical and if the difference between the tool sizes is not more than a predetermined value (e.g., 5%), for example. If the two tool shapes are not identical or if the difference between the tool sizes is more than the predetermined value, it is determined that the changed tool is not identical to the managed tool.”).”. Examiner submits that under the broadest reasonable interpretation (BRI) of the claim, that Sato does teach/disclose the argued limitation. Therefore, applicant’s arguments are unpersuasive.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked. As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph: (A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function; (B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and (C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function. Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function. Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function. Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or preAIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) are:
Image processing unit - (first stated in claim 1)
Determination unit - (first stated in claim 1)
Brightness value acquisition unit - (first stated in claim 2)
Evaluation data acquisition unit - (first stated in claim 5)
Identification processing unit - (first stated in claim 6)
Condition acquisition unit - (first stated in claim 8)
Examiner notes that the applicant’s specification (specifically paragraph 22) state “The configuration of the tool checking device 50a (50) will be described. The tool checking device 50a according to an embodiment includes a condition acquisition unit 51, a tool movement control unit 52, an imaging control unit 53, an image processing unit SSa (55), a brightness value acquisition unit 56, and a determination unit 59a, 59b (59). The functions of these components are implemented by a processor 91 (see FIG. 4) as will be described later.”
Because these claim limitation(s) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof. If applicant does not intend to have these limitation(s) interpreted under 35 U.S.C. 112(f) or preAIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 8, 11 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Sato (US 2016/0334777 A1) in view of Johnson (US 2020/0090099 A1).
Regarding claim 1, Sato teaches the tool checking device comprising: an imaging control unit configured to control the imaging device configured to capture the first image of the workpiece so as to cause the imaging device to capture a second image of the tool, the second image being different from the first image [(see at least paragraph 9, 28, 33-37) As in 9 “The machine tool includes an automatic tool changer and an image pickup unit configured to capture an image of a tool. The numerical controller includes: a tool management data storage unit configured to store tool management data that includes data on at least a shape and a size of the tool associated with a tool number; a tool change command execution unit configured to retrieve only a tool change command in the NC program and control the automatic tool changer to execute tool change based on the retrieved tool change command; a tool image acquisition unit configured to control the image pickup unit to capture the image of the tool mounted on a tool mounting portion of the machine tool every time the tool is changed by the tool change command execution unit, thereby acquiring the tool image” As in 33 “ the tool image acquisition means 110 controls the image pickup means 200 of the machine tool 2 so as to capture the image of the tool mounted on the spindle 20, acquires the captured tool image, and outputs the acquired image to the tool data calculation means 120. The image pickup means 200 of the machine tool 2 may be composed only of the camera 24 (FIG. 1) secured around the tool change position or configured so that the camera 24 previously mounted on the robot 25 can be moved to the vicinity of the tool change position by controlling the robot 25.”]; an image processing unit configured to perform, on the second image of the tool captured by the imaging device configured to capture the first image of the workpiece, image processing associated with a tool condition regarding a tool type or tool state that needs to be satisfied by the tool, and generate a processed image in which a related area is extracted, the extracted related area being an area along at least part of a contour of a tool satisfying the tool condition [(see at least paragraphs 9, 20, 41) As in 9 “The machine tool includes an automatic tool changer and an image pickup unit configured to capture an image of a tool. The numerical controller includes: a tool management data storage unit configured to store tool management data that includes data on at least a shape and a size of the tool associated with a tool number; a tool change command execution unit configured to retrieve only a tool change command in the NC program and control the automatic tool changer to execute tool change based on the retrieved tool change command; a tool image acquisition unit configured to control the image pickup unit to capture the image of the tool mounted on a tool mounting portion of the machine tool every time the tool is changed by the tool change command execution unit, thereby acquiring the tool image” As in 20 “means for controlling the camera to capture an image of a tool every time the tool change command is executed, and means for analyzing the captured image to calculate a shape and a size of the tool.”]; and a determination unit configured to determine whether the tool satisfies the tool condition, on the basis of the processed image. [(see at least paragraphs 9,41) As in 9 “a collating unit configured to collate the shape and the size of the tool calculated by the tool data calculation unit with a shape and a size associated with the tool number commanded by the tool change command that are stored in the tool management data storage unit; and a display unit configured to display the result of the collation by the collating unit”]
Sato does not explicitly teach a tool attached to a robot arm; a food product as a workpiece and a tool checking device for a robot arm of a workpiece processing system which comprises: the robot arm to which a tool for processing a food product as a workpiece is attached; and an imaging device configured to capture a first image of the workpiece, the workpiece being processed by the tool based on results of an image analysis of the captured first image of the workpiece.
However, Johnson teaches a tool attached to a robot arm [(see at least paragraph 60) “Each end effector utensil can be attached and detached from the robot arm by the robot”]
Johnson teaches a food product as a workpiece [(see at least paragraph 27) “The robot arm 110 includes sensor elements/modules such as stereo vision systems (SVS), 3D vision sensors (e.g., Microsoft Kinect™ or an Intel RealSense™), LIDAR sensors, audio sensors (e.g., microphones), inertial sensors (e.g., internal motion unit (IMU), torque sensor, weight sensor, etc.) for sensing aspects of the environment, including pose (i.e., X, Y, Z coordinates and roll, pitch, and yaw angles) of tools for the robot to mate, shape and volume of foodstuffs in ingredient containers, shape and volume of foodstuffs deposited into food assembly container, moving or static obstacles in the environment, etc.”]
Johnson teaches a tool checking device for a robot arm of a workpiece processing system which comprises: the robot arm to which a tool for processing a food product as a workpiece is attached; and an imaging device configured to capture a first image of the workpiece, the workpiece being processed by the tool based on results of an image analysis of the captured first image of the workpiece [(see at least paragraph 27) “The food preparation area 102 includes a plurality of ingredient containers 106a-d each having a particular foodstuff (e.g., lettuce, chicken, cheese, tortilla chips, guacamole, beans, rice, various sauces or dressings, etc.). Each ingredient container 106a-d stores in situ its corresponding ingredients. Utensils 108a-d may be stored in situ in the ingredient containers or in a stand-alone tool rack 109. The utensils 108a-d can be spoons, ladles, tongs, dishers (scoopers), spatulas, or other utensils. Each utensil 108a-e is configured to mate with and disconnect from a tool changer interface 112 of a robot arm 110. While the term utensil is used throughout this application, a person having ordinary skill in the art can recognize that the principles described in relation to utensils can apply in general to end effectors in other contexts (e.g., end effectors for moving fracturable or deformable materials in construction with an excavator or backhoe, etc.); and a robot arm can be replaced with any computer controlled actuatable system which can interact with its environment to manipulate a deformable material. The robot arm 110 includes sensor elements/modules such as stereo vision systems (SVS), 3D vision sensors (e.g., Microsoft Kinect™ or an Intel RealSense™), LIDAR sensors, audio sensors (e.g., microphones), inertial sensors (e.g., internal motion unit (IMU), torque sensor, weight sensor, etc.) for sensing aspects of the environment, including pose (i.e., X, Y, Z coordinates and roll, pitch, and yaw angles) of tools for the robot to mate, shape and volume of foodstuffs in ingredient containers, shape and volume of foodstuffs deposited into food assembly container, moving or static obstacles in the environment, etc.”]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Sato to incorporate the teachings of Johnson of a tool attached to a robot arm, a food product as a workpiece and a tool checking device for a robot arm of a workpiece processing system which comprises: the robot arm to which a tool for processing a food product as a workpiece is attached; and an imaging device configured to capture a first image of the workpiece, the workpiece being processed by the tool based on results of an image analysis of the captured first image of the workpiece in order to safely manipulate food stuffs which are often fracturable and deformable materials and to automatically and seamlessly switch utensils. [(Johnson 25)]
Regarding claim 8, In view of the above combination of references, Sato further teaches further comprising a condition acquisition unit configured to acquire the tool condition according to a work schedule of the robot arm after the determination unit determines that the tool condition is satisfied, wherein the image processing unit is configured to perform, on the captured image, the image processing associated with the tool condition acquired by the condition acquisition unit among a plurality of the tool conditions prepared in advance, and wherein the determination unit is configured to determine whether the tool condition acquired by the condition acquisition unit is satisfied. [(see at least Fig.6, paragraphs 41-48) As in 41 “Then, the collating means 130 collates the read tool shape and size with the tool shape and size data received from the tool data calculation means 120 and outputs the result of the collation to the display means 140. In the collation processing, it is determined that the changed tool is identical to the managed tool if the two tool shapes are identical and if the difference between the tool sizes is not more than a predetermined value (e.g., 5%), for example. If the two tool shapes are not identical or if the difference between the tool sizes is more than the predetermined value, it is determined that the changed tool is not identical to the managed tool.” As in 43 “For a tool of a tool number 0012 in the example shown in FIG. 5, tool management data on the tool number 0012 stored in the tool management data storage unit 210 and the tool shape are identical and the difference between the tool sizes is within the range of the predetermined value, with respect to the tool shape and size data calculated from the image of the tool mounted on the spindle 20, so that the collation result is displayed as “OK”. For a tool of a tool number 0016, in contrast, the difference between the tool sizes exceeds the predetermined value, although tool management data on the tool number 0016 stored in the tool management data storage unit 210 and the tool shape are identical, with respect to the tool shape and size data calculated from the image of the tool mounted on the spindle 20, so that the collation result is displayed as “NG”.”]
Regarding claim 11, Sato teaches the method comprising: an imaging control step of controlling the imaging device configured to capture the first image of the workpiece so as to cause the imaging device to capture a second image of the tool, the second image being different from the first image [(see at least paragraph 9, 28, 33-37) As in 9 “The machine tool includes an automatic tool changer and an image pickup unit configured to capture an image of a tool. The numerical controller includes: a tool management data storage unit configured to store tool management data that includes data on at least a shape and a size of the tool associated with a tool number; a tool change command execution unit configured to retrieve only a tool change command in the NC program and control the automatic tool changer to execute tool change based on the retrieved tool change command; a tool image acquisition unit configured to control the image pickup unit to capture the image of the tool mounted on a tool mounting portion of the machine tool every time the tool is changed by the tool change command execution unit, thereby acquiring the tool image” As in 33 “ the tool image acquisition means 110 controls the image pickup means 200 of the machine tool 2 so as to capture the image of the tool mounted on the spindle 20, acquires the captured tool image, and outputs the acquired image to the tool data calculation means 120. The image pickup means 200 of the machine tool 2 may be composed only of the camera 24 (FIG. 1) secured around the tool change position or configured so that the camera 24 previously mounted on the robot 25 can be moved to the vicinity of the tool change position by controlling the robot 25.”]; an image processing step of performing, on the second image of the tool captured by the imaging device for capturing configured to capture the first image of the workpiece, image processing associated with a tool condition regarding a tool type or tool state that needs to be satisfied by the tool, and generating a processed image in which a related area is extracted, the extracted related area being an area along at least part of a contour of a tool satisfying the tool condition [(see at least paragraphs 9, 20, 41) As in 9 “The machine tool includes an automatic tool changer and an image pickup unit configured to capture an image of a tool. The numerical controller includes: a tool management data storage unit configured to store tool management data that includes data on at least a shape and a size of the tool associated with a tool number; a tool change command execution unit configured to retrieve only a tool change command in the NC program and control the automatic tool changer to execute tool change based on the retrieved tool change command; a tool image acquisition unit configured to control the image pickup unit to capture the image of the tool mounted on a tool mounting portion of the machine tool every time the tool is changed by the tool change command execution unit, thereby acquiring the tool image” As in 20 “means for controlling the camera to capture an image of a tool every time the tool change command is executed, and means for analyzing the captured image to calculate a shape and a size of the tool.”]; and a determination step of determining whether the tool satisfies the tool condition, on the basis of the processed image. [(see at least paragraphs 9,41) As in 9 “a collating unit configured to collate the shape and the size of the tool calculated by the tool data calculation unit with a shape and a size associated with the tool number commanded by the tool change command that are stored in the tool management data storage unit; and a display unit configured to display the result of the collation by the collating unit”]
Sato does not explicitly teach a tool attached to a robot arm; a food product as a workpiece; and a method for checking a tool for a robot arm of a workpiece processing system which comprises: the robot arm to which the tool for processing a food product as a workpiece is attached; an imaging device configured to capture a first image of the workpiece, the workpiece being processed by the tool based on results of an image analysis of the captured first image of the workpiece.
However, Johnson teaches a tool attached to a robot arm [(see at least paragraph 60) “Each end effector utensil can be attached and detached from the robot arm by the robot”]
Johnson teaches a food product as a workpiece [(see at least paragraph 27) “The robot arm 110 includes sensor elements/modules such as stereo vision systems (SVS), 3D vision sensors (e.g., Microsoft Kinect™ or an Intel RealSense™), LIDAR sensors, audio sensors (e.g., microphones), inertial sensors (e.g., internal motion unit (IMU), torque sensor, weight sensor, etc.) for sensing aspects of the environment, including pose (i.e., X, Y, Z coordinates and roll, pitch, and yaw angles) of tools for the robot to mate, shape and volume of foodstuffs in ingredient containers, shape and volume of foodstuffs deposited into food assembly container, moving or static obstacles in the environment, etc.”]
Johnson teaches a method for checking a tool for a robot arm of a workpiece processing system which comprises: the robot arm to which the tool for processing a food product as a workpiece is attached; an imaging device configured to capture a first image of the workpiece, the workpiece being processed by the tool based on results of an image analysis of the captured first image of the workpiece [(see at least paragraph 27) “The food preparation area 102 includes a plurality of ingredient containers 106a-d each having a particular foodstuff (e.g., lettuce, chicken, cheese, tortilla chips, guacamole, beans, rice, various sauces or dressings, etc.). Each ingredient container 106a-d stores in situ its corresponding ingredients. Utensils 108a-d may be stored in situ in the ingredient containers or in a stand-alone tool rack 109. The utensils 108a-d can be spoons, ladles, tongs, dishers (scoopers), spatulas, or other utensils. Each utensil 108a-e is configured to mate with and disconnect from a tool changer interface 112 of a robot arm 110. While the term utensil is used throughout this application, a person having ordinary skill in the art can recognize that the principles described in relation to utensils can apply in general to end effectors in other contexts (e.g., end effectors for moving fracturable or deformable materials in construction with an excavator or backhoe, etc.); and a robot arm can be replaced with any computer controlled actuatable system which can interact with its environment to manipulate a deformable material. The robot arm 110 includes sensor elements/modules such as stereo vision systems (SVS), 3D vision sensors (e.g., Microsoft Kinect™ or an Intel RealSense™), LIDAR sensors, audio sensors (e.g., microphones), inertial sensors (e.g., internal motion unit (IMU), torque sensor, weight sensor, etc.) for sensing aspects of the environment, including pose (i.e., X, Y, Z coordinates and roll, pitch, and yaw angles) of tools for the robot to mate, shape and volume of foodstuffs in ingredient containers, shape and volume of foodstuffs deposited into food assembly container, moving or static obstacles in the environment, etc.”]
Regarding claim 15, Modified Sato has all of the elements of claim 1 as discussed above.
Sato does not explicitly teach wherein the tool has a movable part, and the tool state is a state determined by a position of the movable part, and wherein the related area is a predefined region in the captured image into which at least a portion of the movable part enters or from which the portion of the movable part exits depending on the tool state.
However, Johnson teaches wherein the tool has a movable part, and the tool state is a state determined by a position of the movable part, and wherein the related area is a predefined region in the captured image into which at least a portion of the movable part enters or from which the portion of the movable part exits depending on the tool state. [(see at least paragraphs 43-49) As in 43 “Given the goal state of a certain spatial distribution of material on the plate and in the bins, and the current world state (e.g., an empty plate or partially empty plate and material in the bins), the system of the present disclosure computes a series or sequence of motion plans which re-arrange a portion of the material from one or more bins into a distribution of the material on the plate designated in a recipe/plan. A goal world state is a goal state of the world, including the meal being prepared. The goal world state can also include the state of the materials used to create the meal and the tools used to create the meal. The current world state includes a current state of the meal being prepared (e.g., the ingredients in the meal preparation area, if any, and the distribution and order of those ingredients), the state of the meal preparation tools, and the state of the materials used to create the meal (e.g., how much of each ingredient is available, and the distribution of the remaining materials/ingredients).” As in 46 “First, a planning or designing user defines an initial “goal state” 202 as a spatial distribution of materials with spatial constraints which are specific to each material and their relation to other materials and the container. The material can be specified by one or more of a density, voxels, surface, or other distribution which encapsulates volume and mass of the material.” As in 49 “The action is performed by a tool to alter the observed state of the materials, with the goal creating the goal state. The action can be a granular action, such as changing a tool, rotating a mechanical arm, moving a mechanical arm, applying torque, scooping, tonging, or otherwise acquiring a material/ingredient, etc”]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Sato to incorporate the teachings of Johnson of wherein the tool has a movable part, and the tool state is a state determined by a position of the movable part, and wherein the related area is a predefined region in the captured image into which at least a portion of the movable part enters or from which the portion of the movable part exits depending on the tool state in order to generate a successful motion plan based off the tool condition. [(Johnson 9)]
Regarding claim 17, In view of the above combination of references, Sato further teaches wherein the tool satisfying the tool condition and the tool attached to the robot arm are the same type of tool as each other. [(see at least paragraph 41) “The collating means 130 reads the tool shape (tool type) and size corresponding to the currently selected tool number (or tool number commanded by the tool change command read from the NC program by the tool change command execution means 100) from the tool management data storage unit 210. Then, the collating means 130 collates the read tool shape and size with the tool shape and size data received from the tool data calculation means 120 and outputs the result of the collation to the display means 140. In the collation processing, it is determined that the changed tool is identical to the managed tool if the two tool shapes are identical and if the difference between the tool sizes is not more than a predetermined value (e.g., 5%), for example. If the two tool shapes are not identical or if the difference between the tool sizes is more than the predetermined value, it is determined that the changed tool is not identical to the managed tool.”]
Claims 2-4 are rejected under 35 U.S.C. 103 as being unpatentable over Sato in view of Johnson and in further view of Tenney (US 2013/0010081 A1).
Regarding claim 2, Modified Sato has all of the elements of claim 1 as discussed above.
Sato does not explicitly teach further comprising a brightness value acquisition unit configured to acquire a brightness value of the processed image, wherein the determination unit is configured to determine whether the tool condition is satisfied on the basis of the brightness value acquired.
However, Tenney teaches further comprising a brightness value acquisition unit configured to acquire a brightness value of the processed image, wherein the determination unit is configured to determine whether the tool condition is satisfied on the basis of the brightness value acquired. [(see at least paragraph 24) “Various tools, including a single needle, punch or cannula, or tool assemblies of various configurations may be used in different embodiments. In some embodiments, the first and second data transitions may comprise one or more contrast, intensity, color, or brightness transitions of the image data, and the detecting step may further comprise processing the image data such that the distal end of the tool is represented as a unique value within the image data, e.g., a unique color within the image data. In some embodiments, the detecting step may further comprise binarizing the image data such that the distal end is represented by a first unique color and background features are represented by a second unique color. In some embodiments, the regression comprises a best-fit algorithm or a least-squares fit that approximately connects respective first and second sets of data transitions.”]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of modified Sato to further incorporate the teachings of Tenney of a brightness value acquisition unit configured to acquire a brightness value of the processed image, wherein the determination unit is configured to determine whether the tool condition is satisfied on the basis of the brightness value acquired in order to have the distal end of the tool is represented as a unique value within the image data. [(Tenney 24)]
Regarding claim 3, Modified Sato has all of the elements of claim 2 as discussed above.
Sato does not explicitly teach wherein the brightness value acquisition unit is configured to acquire a sum of brightness values of the processed image, and wherein the determination unit is configured to determine whether the tool condition is satisfied on the basis of the sum of brightness values acquired.
However, Tenney teaches wherein the brightness value acquisition unit is configured to acquire a sum of brightness values of the processed image, and wherein the determination unit is configured to determine whether the tool condition is satisfied on the basis of the sum of brightness values acquired. [(see at least paragraph 24) “In some embodiments, the first and second data transitions may comprise one or more contrast, intensity, color, or brightness transitions of the image data, and the detecting step may further comprise processing the image data such that the distal end of the tool is represented as a unique value within the image data, e.g., a unique color within the image data. In some embodiments, the detecting step may further comprise binarizing the image data such that the distal end is represented by a first unique color and background features are represented by a second unique color.”]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of modified Sato to further incorporate the teachings of Tenney of the brightness value acquisition unit is configured to acquire a sum of brightness values of the processed image, and wherein the determination unit is configured to determine whether the tool condition is satisfied on the basis of the sum of brightness values acquired in order to binarize the image data such that the distal end is represented by a first unique color and background features are represented by a second unique color. [(Tenney 24)]
Regarding claim 4, Modified Sato has all of the elements of claim 2 as discussed above.
Sato does not explicitly teach wherein the brightness value acquisition unit is configured to acquire a sum X.sub.1 of differences between brightness values identified by the following equation (1) X.sub.1=Σ.sub.i=0.sup.MΣ.sub.j=0.sup.N|B.sub.ij−Bs.sub.ij| where B.sub.i is a brightness value of each pixel of the processed image, i is any natural number equal to or less than the number of pixels in a horizontal direction of the processed image and j is any natural number equal to or less than the number of pixels in a vertical direction of the processed image, and Bs.sub.ij is a brightness value set for each pixel according to the tool condition, and wherein the determination unit is configured to determine whether the tool condition is satisfied on the basis of the sum X.sub.1 of differences between brightness values acquired.
However, Tenney teaches wherein the brightness value acquisition unit is configured to acquire a sum X.sub.1 of differences between brightness values identified by the following equation (1) X.sub.1=Σ.sub.i=0.sup.MΣ.sub.j=0.sup.N|B.sub.ij−Bs.sub.ij| where B.sub.i is a brightness value of each pixel of the processed image, i is any natural number equal to or less than the number of pixels in a horizontal direction of the processed image and j is any natural number equal to or less than the number of pixels in a vertical direction of the processed image, and Bs.sub.ij is a brightness value set for each pixel according to the tool condition, and wherein the determination unit is configured to determine whether the tool condition is satisfied on the basis of the sum X.sub.1 of differences between brightness values acquired. [(see at least paragraph 24) “In some embodiments, the first and second data transitions may comprise one or more contrast, intensity, color, or brightness transitions of the image data, and the detecting step may further comprise processing the image data such that the distal end of the tool is represented as a unique value within the image data, e.g., a unique color within the image data. In some embodiments, the detecting step may further comprise binarizing the image data such that the distal end is represented by a first unique color and background features are represented by a second unique color.”]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of modified Sato to further incorporate the teachings of Tenney of the brightness value acquisition unit is configured to acquire a sum X.sub.1 of differences between brightness values identified by the following equation (1) X.sub.1=Σ.sub.i=0.sup.MΣ.sub.j=0.sup.N|B.sub.ij−Bs.sub.ij| where B.sub.i is a brightness value of each pixel of the processed image, i is any natural number equal to or less than the number of pixels in a horizontal direction of the processed image and j is any natural number equal to or less than the number of pixels in a vertical direction of the processed image, and Bs.sub.ij is a brightness value set for each pixel according to the tool condition, and wherein the determination unit is configured to determine whether the tool condition is satisfied on the basis of the sum X.sub.1 of differences between brightness values acquired in order to binarize the image data such that the distal end is represented by a first unique color and background features are represented by a second unique color. [(Tenney 24)]
Claims 5 is rejected under 35 U.S.C. 103 as being unpatentable over Sato in view of Johnson and in further view of Yamazaki (JP 2020-110920 A).
Regarding claim 5, Modified Sato has all of the elements of claim 1 as discussed above.
Sato does not explicitly teach further comprising: a storage unit storing a trained model configured to output evaluation data regarding whether the tool satisfies the tool condition in response to input of data regarding the processed image; and an evaluation data acquisition unit configured to acquire the evaluation data output from the trained model into which the processed image generated by the image processing unit has been input, wherein the determination unit is configured to determine whether the tool condition is satisfied on the basis of the evaluation data acquired.
However, Yamazaki teaches further comprising: a storage unit storing a trained model configured to output evaluation data regarding whether the tool satisfies the tool condition in response to input of data regarding the processed image; and an evaluation data acquisition unit configured to acquire the evaluation data output from the trained model into which the processed image generated by the image processing unit has been input, wherein the determination unit is configured to determine whether the tool condition is satisfied on the basis of the evaluation data acquired. [(see at least paragraphs 42, 71-80) As in 42 “The learning unit 14 is a part that performs processing related to machine learning. The learning unit 14 includes a learning processing unit 141, a learned model storage unit 142, and an estimation processing unit 143. The learning processing unit 141 is a part that executes machine learning, and for example, performs deep learning (deep learning) using a convolutional neural network (convolutional neural network). The trained model storage unit 142 is a part that stores the parameters of the learning model that the learning processing unit 141 is learning in machine learning and the parameters of the trained model. The estimation processing unit 143 performs estimation using the trained model stored in the trained model storage unit 142 in order to select the extraction position by the selection processing unit 11. The detailed contents of machine learning performed by each of these parts will be described later in the item.” As in 72 “The learning processing unit 141 uses the teaching position vicinity image data included in the matching point cloud information stored in the selection data storage unit 111 as input data, and the evaluation value given to the matching point cloud information as a label. Supervised learning. As a method of supervised learning, the learning processing unit 141 performs deep learning (deep learning) using a convolutional neural network (CNN), which is a neural network suitable for learning targeting image data. Do. Therefore, a convolution neural network having three or more layers and including at least one image convolution operation is prepared. However, this does not mean that the machine learning applied in the first embodiment is limited to the convolution neural network. A deep learning model other than the convolution neural network, machine learning using a linear model, or the like may be applied to the first embodiment.” As in 73 “Here, the convolution neural network has a structure including a convolution layer, a pooling layer, a fully connected layer, and an output layer. However, this is just a structural example for explanation, and for example, the pooling layer may be omitted. Further, for example, as described above, when learning is performed using an image as a label, a deconvolution layer may be further provided.”]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of modified Sato to incorporate the teachings of Yamazaki of a storage unit storing a trained model configured to output evaluation data regarding whether the tool satisfies the tool condition in response to input of data regarding the processed image; and an evaluation data acquisition unit configured to acquire the evaluation data output from the trained model into which the processed image generated by the image processing unit has been input, wherein the determination unit is configured to determine whether the tool condition is satisfied on the basis of the evaluation data acquired in order to improve the selection accuracy of the extraction position of the work/tool. [(Yamazaki 70)]
Claims 6-7 are rejected under 35 U.S.C. 103 as being unpatentable over Sato in view of Johnson and in further view of Nakano II (JP2014-168823A).
Regarding claim 6, Modified Sato has all of the elements of claim 1 as discussed above.
Sato does not explicitly teach wherein the image processing unit is configured to perform, on the captured image, the image processing associated with the tool condition of the tool having an outer surface with a mark, wherein the tool checking device further comprises an identification processing unit configured to execute a process of identifying the mark on the processed image generated, and wherein the determination unit is configured to determine whether the tool condition is satisfied on the basis of a result of the process by the identification processing unit.
However, Nakano II teaches wherein the image processing unit is configured to perform, on the captured image, the image processing associated with the tool condition of the tool having an outer surface with a mark, wherein the tool checking device further comprises an identification processing unit configured to execute a process of identifying the mark on the processed image generated, and wherein the determination unit is configured to determine whether the tool condition is satisfied on the basis of a result of the process by the identification processing unit. [(see at least paragraphs 50-58) As in 51 “the tool number and the tool dimension (correction value) are printed on the label 33 which can be recognized by an image such as a matrix type two-dimensional code by the label printer 14 shown in FIG. Further, information is written to an electrically readable tag 34 such as an IC tag by the tag writer 15” As in 58 “Next, in step S460 of FIG. 15, the integrated information generation unit A 46a generates integrated information including the information recorded in the label / tag and the image information as shown in FIG. Here, the label is a matrix type two-dimensional code or the like, and the information written in the image can be read by the image, so even if the information read by the tool information reading unit A is not referred to via the database, It is also possible to read and reference information from the image. That is, since the tool number is recorded on the label, it is determined whether or not it is a desired tool by comparing the read tool number with the tool number determined from the image of the actual tool. Is possible.”] Examiner notes the tag/label is being interpreted as a mark.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of modified Sato to incorporate the teachings of Nakano II of the image processing unit is configured to perform, on the captured image, the image processing associated with the tool condition of the tool having an outer surface with a mark, wherein the tool checking device further comprises an identification processing unit configured to execute a process of identifying the mark on the processed image generated, and wherein the determination unit is configured to determine whether the tool condition is satisfied on the basis of a result of the process by the identification processing unit in order for the mark/label to be recognized within an image. [(Nakano II 51)]
Regarding claim 7, Modified Sato has all of the elements of claim 6 as discussed above.
Sato does not explicitly teach wherein the identification processing unit is configured to execute a process of identifying a character as the mark on the processed image.
However, Nakano II teaches wherein the identification processing unit is configured to execute a process of identifying a character as the mark on the processed image. [(see at least Fig.17, paragraphs 51-52) As in 51 “First, at step S400 in FIG. 15, the tool number and the tool dimension (correction value) are printed on the label 33 which can be recognized by an image such as a matrix type two-dimensional code by the label printer 14 shown in FIG. Further, information is written to an electrically readable tag 34”]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of modified Sato to incorporate the teachings of Nakano II of wherein the identification processing unit is configured to execute a process of identifying a character as the mark on the processed image in order for the mark/label to be recognized within the processed image. [(Nakano II 51)]
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Sato in view of Johnson and in further view of Koda (JPH 6-134638 A).
Regarding claim 9, Modified Sato has all of the elements of claim 1 as discussed above.
Sato does not explicitly teach wherein the image processing unit is configured to apply masking to the captured image using a reference image associated with the tool condition, and generate an image in which the related area associated with the tool condition is extracted as the processed image.
However, Koda teaches wherein the image processing unit is configured to apply masking to the captured image using a reference image associated with the tool condition, and generate an image in which the related area associated with the tool condition is extracted as the processed image. [(see at least paragraph 12) “Tool T1 has been reached the tool side position 4 is photographed by the photographing device 5. Image of the captured tool T1 is transferred from the photographing apparatus 5 as image information to the image processing unit 8a of the information processing apparatus such as a personal computer 8, it is processed according to the processing flow described below. The information in the PC 8, may be displayed on the display of the monitor television 9. First, as shown schematically in FIG. 3, from the original image in the absence of tool T1 on the tool side position 4 (Fig. A), subtracting the original image (Fig. B) when there is a tool T1, so the difference by performing the image processing, determine the image obtained by extracting only a tool T1 portion (Fig. c). Then, after the image processing of binarization processing or the like in the image processing unit 8a the image data, extracting and calculating the characteristic amount of the tool T1 from the image data. This process is performed by the tool feature amount calculating section 8b of the personal computer 8. Tool characteristic amount arithmetic operation section 8b extracts and computing a characteristic quantity by counting the number of pixels from the image data at a predetermined position. The feature data of extracted and computed tool T1 is stored temporarily sent to the matching recognition unit 8c of the personal computer 8. In parallel with this, NC device 3 specifies the tool number T1 in the master data registration section 2, call the master data P1 for tool T1, which had been measured beforehand registering, transferring it to the matching recognition unit 8c make. Then, the matching recognition unit 8c, by collating the characteristic amount data on the tool T1 extracted and calculated by the tool characteristic amount arithmetic operation section 8 b, and a master data P1 for tool T1, photographed tool T1 is NC device 3 than is to recognize whether or not to match the tool T1 specified by. The recognition result is sent from the verification recognition section 8c to the NC device 3. NC apparatus 3 on the basis of the recognition result, the processing to determine whether to continue, if they match allowed to proceed as it is processing, if it does not match interrupt the processing. Incidentally, verification recognized, in this embodiment, adopts a so-called feature extraction method performed by extracting the feature amount of the tool T, so-called pattern matching method performed for each matrix of pixels or more pixels, and other socalled such as the minimum distance can be performed by employing a known image pattern recognition techniques.”]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of modified Sato to incorporate the teachings of Koda of the image processing unit is configured to apply masking to the captured image using a reference image associated with the tool condition, and generate an image in which the related area associated with the tool condition is extracted as the processed image in order to process and monitor the wear of the tool and its condition. [(Koda 5)]
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Sato in view of Johnson and in further view of Nakano (US 2014/0148939 A1).
Regarding claim 16, Modified Sato has all of the elements of claim 1 as discussed above.
Sato does not explicitly teach wherein the imaging control unit is further configured to control the imaging device configured to capture the first image of the food product as the workpiece such that the first image of the food product as the workpiece captured by the imaging device excludes the image of the tool attached to the robot arm.
Johnson teaches the food product as the workpiece. [(see at least paragraph 27) “The robot arm 110 includes sensor elements/modules such as stereo vision systems (SVS), 3D vision sensors (e.g., Microsoft Kinect™ or an Intel RealSense™), LIDAR sensors, audio sensors (e.g., microphones), inertial sensors (e.g., internal motion unit (IMU), torque sensor, weight sensor, etc.) for sensing aspects of the environment, including pose (i.e., X, Y, Z coordinates and roll, pitch, and yaw angles) of tools for the robot to mate, shape and volume of foodstuffs in ingredient containers, shape and volume of foodstuffs deposited into food assembly container, moving or static obstacles in the environment, etc.”]
However, Nakano teaches wherein the imaging control unit is further configured to control the imaging device configured to capture the first image of the workpiece such that the first image of the workpiece captured by the imaging device excludes the image of the tool attached to the robot arm. [(see at least Figs.2-4, paragraph 6) “Specifically, there is provided the tool observation method comprising the steps of: imaging a state of a tool for cutting a workpiece, with an imaging unit; and observing the tool based on this image information, wherein a plurality of images of the workpiece are captured while rotating or moving the tool at least before or after machining the workpiece with the tool, and wherein a focused image among the plurality of images is selectively used for observation.”]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of modified Sato by further incorporating the teachings of Johnson of the food product as the workpiece in order to safely manipulate food stuffs which are often fracturable and deformable materials and to automatically and seamlessly switch utensils [(Johnson 25)] and to incorporate the teachings of Nakano of wherein the imaging control unit is further configured to control the imaging device configured to capture the first image of the workpiece such that the first image of the workpiece captured by the imaging device excludes the image of the tool attached to the robot arm in order to ensure the workpiece is accurately handled. [(Nakano 2)]
The Examiner has cited particular paragraphs or columns and line numbers in the references applied to the claims above for the convenience of the Applicant. Although the specified citations are representative of the teachings of the art and are applied to specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested of the Applicant in preparing responses, to fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the Examiner. See MPEP 2141.02 [R-07.2015] VI. A prior art reference must be considered in its entirety, i.e., as a whole, including portions that would lead away from the claimed Invention. W.L. Gore & Associates, Inc. v. Garlock, Inc., 721 F.2d 1540, 220 USPQ 303 (Fed. Cir. 1983), cert, denied, 469 U.S. 851 (1984). See also MPEP §2123.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOHAMMED YOUSEF ABUELHAWA whose telephone number is (571)272-3219. The examiner can normally be reached Monday-Friday 8:30-5:00 with flex.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Wade Miles can be reached at 571-270-7777. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MOHAMMED YOUSEF ABUELHAWA/Examiner, Art Unit 3656
/WADE MILES/Supervisory Patent Examiner, Art Unit 3656
Prosecution Insights