Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This Office action is in response to the amendment filed on 03/26/2026. Claims 1-21 are currently pending with claims 1, 5, 7, 11, 13, 16, and 20-21 being amended.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 10/21/2025 has been received. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Amendment
The amendments to the claims submitted on 03/23/2026 overcome the claim objections set forth in the previous Office action except for those set forth in the claim objection section.
Response to Arguments
Applicant’s arguments, see claims and remarks, filed 03/23/2026, with respect to the rejection of claims 5-8, 16, and 20-21 under 35 U.S.C. 112(b) have been fully considered and are persuasive. The rejection of claims 5-8, 16, and 20-21 under 35 U.S.C. 112(b) has been withdrawn.
Applicant's arguments filed 03/23/2026 regarding the interpretation of claim limitations under 35 U.S.C. 112(f) have been fully considered but they are not persuasive.
The terms “calibrator” and “inspector” are not known structures capable of performing the claimed functionality. While these claim limitations do not use the word “means” or “step”, they are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier.
Because this/these claim limitations are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have these limitations interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Applicant's arguments filed 03/23/2026 regarding the rejection of claims 1, 7-13, and 17-21 under 35 U.S.C. 103 in view of Poelman and Attar as well as the rejection of claims 2-4, 6, and 14-15 in view of Poelman, Attar, and Pagnon have been fully considered but they are not persuasive.
The validation process is not defined except for the use of partial assembly and under broadest reasonable interpretation may encompass the validation/verification that the assembly being performed meets requirements of the product model. For example, validation may be that the model was successfully followed up to this point of inspection. Presence of an error during validation/inspection of a partially assembled product would be a method of validating the product model (see Attar Paragraph 0425, “(v) building part re-alignment, or otherwise robot pose correction; and/or (vi) post-assembly inspection of building structures—images of assembled, or partially assembled, building structures can be inspected to automatically identify defects in requiring modification.”). This demonstrates that the system identifies places where the product model/assembly data could be optimized. If no errors are present then the model/assembly instructions are successful. If there are errors or faults then the model/assembly instructions may be considered flawed. This may trigger a fault which causes the system to re-evaluate the robotic cell and assembly data. Then a user may make “a request for modifying the assembly sequence” (See Attar Paragraph 0022) which can then be deployed to any subset of robotic cells. There is no requirement in the currently provided claim language that the product model is modified if an error is detected/identified.
The Applicant further argues that the “partially assembled” state is different from a “partial insertion of components”. This is not present in the currently provided claim language. A partially assembled product encompasses a product in any state after assembly begins until assembly has been completed. Attar acquires images of the product while the product is only partially assembled and uses those images for analyzing the assembly.
The Applicant has further stated that the method includes “validating special data coordinates in a produce model using a partial assembly method” as well as “execution of a physical partial assembly method to validate the accuracy of a virtual product model”. The remarks argue that the words “virtual” and “product model” do not appear in the cited art. The currently provided claim language does not use the word “virtual” at any point and does not require this. The currently provided claim language of the independent claims does not define the “product model” beyond the term. Therefore, this term may be interpreted as any representation/model of the product. The prior art primary reference (Poelman) utilizes a CAD model of an end product and a “recipe”. The system uses this information to determine a sequence of operations which will allow the robot to assemble the product. Further, the secondary reference (Attar) teaches a “design file” and “assembly data” which includes a “CAD model file of the building structure” (See Attar Paragraph 0183). This further demonstrates the use of a virtual representation/model of the product as well as the assembly process for use in operation of the system.
Claim Objections
Claim 16 is objected to because of the following informalities:
“such that navigation to and insertion of an element at an incorrect location” should be corrected to “such that navigation to, and insertion of, an element at an incorrect location” or similar.
Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
Regarding claim 13, “calibrator” will be interpreted under 112(f) because of the following three-prong analysis:
Prong 1: The claim uses the nonce term “calibrator”.
Prong 2: The claim uses functional language to modify the nonce term.
Prong 3: Sufficient structure for performing the function is not recited within the claim.
This limitation is being interpreted according to the specification (paragraph 0047) as a processor.
Regarding claim 13, “product model generator” will be interpreted under 112(f) because of the following three-prong analysis:
Prong 1: The claim uses the nonce term “generator”.
Prong 2: The claim uses functional language to modify the nonce term.
Prong 3: Sufficient structure for performing the function is not recited within the claim.
This limitation is being interpreted according to the specification (paragraphs 0047-0048) as a processor.
Regarding claim 13, “validator” will be interpreted under 112(f) because of the following three-prong analysis:
Prong 1: The claim uses the nonce term “validator”.
Prong 2: The claim uses functional language to modify the nonce term.
Prong 3: Sufficient structure for performing the function is not recited within the claim.
This limitation is being interpreted according to the specification (paragraphs 0047 and 0049) as a processor.
Regarding claim 17, “inspector” will be interpreted under 112(f) because of the following three-prong analysis:
Prong 1: The claim uses the nonce term “inspector”.
Prong 2: The claim uses functional language to modify the nonce term.
Prong 3: Sufficient structure for performing the function is not recited within the claim.
This limitation is being interpreted according to the specification (paragraphs 0047 and 0049) as a processor.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 7-13, and 17-21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Poelman et al. (US 20220147026 A1), hereinafter Poelman in view of Attar et al. (US 20240017408 A1), hereinafter Attar.
Regarding claim 1, Poelman teaches:
1. (Currently Amended) A method of automated
calibrating a robotic cell (Paragraph 0067, "The workspace in one embodiment may include other elements such as the workpiece, and the conveyor belt(s) 226 which move work pieces to and from the robotic cell. Other elements may include a tray 227 on which a workpiece, or other parts to be added to the workpiece are located, and the path from the tray feeder. Other elements may include the end of arm tool 228, and optionally a tool switch if other tools are present. A “tool switch” refers to a special mounting interface on the robotic arm that allows the end-of-arm tool to be changed (either manually or automatically via software control). This enables the auto-calibration routine to calibrate multiple tools. In one embodiment, this also enables the use of a special calibration tool for the robot, enabling the system to calibrate and then re-mount an end-of-arm tool. Other elements in the work space may include safety systems such as light curtains, an electrical system, illumination systems, sensors, cable management systems, and other elements which are within the workspace of the robotic cell, and thus may be considered as part of the robotic cell system.") for generating a product model, (Paragraph 0149, "FIG. 11 is a simplified block diagram of the process which utilizes the micro factory comprising one or more robotic cells to create assembled final products. In some embodiments, the robotic cells may be inserted into a traditional manufacturing line, to take over some sub-portions of the manufacturing. The process includes the layers of manufacturing, from recipe creation via recipe creator C10 to fabrication/assembly via micro factory C50. Although a complete process is shown, from initial concept/functional design through completed manufacturing, one of skill in the art would understand that the system may implement a subset of these processes and include a subset of these features.") the robotic cell including an end-of-arm sensor; (Paragraph 0062, "FIG. 1E is a diagram showing one embodiment of the elements of defining a holistic view of a workspace of the robotic cell. The workspace is defined in one embodiment by a frame and enclosure within which the robotic arm can move to take action on work pieces. In one embodiment, the workspace includes two or more cameras to provide a stereoscopic view, or multi-camera three dimensional view, of the space. In one embodiment, there is an additional camera on a side of the frame. In one embodiment, there is a camera at the end of arm. Additional cameras may also be used. The different cameras with different viewpoints give different benefits.")
utilizing the robotic cell to generate the product model of the device for assembly; (Paragraph 0149, "FIG. 11 is a simplified block diagram of the process which utilizes the micro factory comprising one or more robotic cells to create assembled final products. In some embodiments, the robotic cells may be inserted into a traditional manufacturing line, to take over some sub-portions of the manufacturing. The process includes the layers of manufacturing, from recipe creation via recipe creator C10 to fabrication/assembly via micro factory C50. Although a complete process is shown, from initial concept/functional design through completed manufacturing, one of skill in the art would understand that the system may implement a subset of these processes and include a subset of these features.")
…
making the product model available for use by robotic cells for navigation, validation, and/or inspection. (Please see Figure 1A as well as Paragraph 0047, "In one embodiment, the robotic cells A10 are controlled by software. In one embodiment, the configuration and control data for the robotic cells A10 are applied to the cell from memory A20. In one embodiment, the memory A20 may be part of a remote system, coupled to the robotic cells A10 via network A05. The configuration data A25 defines the software configuration for each robotic cell A10 and the manufacturing line. Each robotic cell is calibrated prior to use. In one embodiment, the robotic cells A10 are also continuously calibrated, as will be described below in more detail.")
Poelman does not specifically teach validation using partial assembly. However, Attar, in the same field of endeavor of robotics, teaches:
… validating the product model using a partial assembly method; (Paragraph 0425, "In the illustrated example, the position sensor system can include one or more imaging sensors 1502, 1504 located over, or around the assembly robots 204. In some example cases, images captured by imaging sensors 1502, 1504 can be used for: (i) real-time collision avoidance and safety—monitoring for obstacles in the assembly robot's motion pathway that may otherwise obstruct motion of the assembly robot. Obstacles may include other assembly robots, or otherwise any dynamic objects moving in the environment (e.g., near-by human operators); (ii) object detection and verification—capturing images to locate objects (or building parts) requiring pick-up and assembly, and further verifying that the correct object has been picked-up, or the object is picked-up in the correct orientation; (iii) object grasp verification—verifying that a picked-up object is grasped correctly by the assembly robot; and/or (iv) object quality inspection—after (or before) a building part is picked-up, it may be imaged to detect any flaws that compromise the quality of the building part. Building parts that are poor quality may not be used for assembly. For example, as shown in images 1702 and 1704 (FIG. 17), this can include detecting various cracks and fissures; (v) building part re-alignment, or otherwise robot pose correction; and/or (vi) post-assembly inspection of building structures—images of assembled, or partially assembled, building structures can be inspected to automatically identify defects in requiring modification.") …
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic system and control methods as taught by Poelman with the ability to validate and verify the process using partial assembly methods as taught by Attar. This would ensure that defects and instances where the system does not meet expectations may be quickly and automatically addressed.
Regarding claim 7, where all the limitations of claim 1 are discussed above, Poelman further teaches:
7. (Currently Amended) The method of claim 1, further comprising:
using a feedback sensor to fine-tune navigation offsets, (Paragraphs 0024-0025, "By building up a holistic or system view of a robotic cell and its working area the auto-calibration system can provide high accuracy and precision, an iterative feedback loop, constant pose refinement, and camera refinement. Furthermore, in one embodiment, the auto-calibration system can accomplish this with off-the shelf pieces, rather than customized assemblies. In one embodiment, a plurality of low cost cameras may be used, in combination with geometric fiducial constraints, to provide high accuracy at a reduced cost and complexity. Auto-calibration may be used to encompass the entirety of the robotic cell environment, including all elements in a system, also referred to as a holistic view, a system view, or solving as a system. For example, in a modular robotic assembly system this may include one or more cameras and/or other sensors, a robotic arm, end of arm tools, the tray on which parts and additional tools are located, the conveyor which moves work pieces into and out of the robotic assembly system, and the individual work pieces. In one embodiment, the holistic view of the working area includes all elements, whether they are stationary or movable, and tracks and/or calibrates all elements.
Auto-calibration is possible in one embodiment using a combination of camera lens calibration, frame registration, end of arm tool contact calibration, robot pose error compensation, and calibration monitoring and using a feed forward control and constant correction to maintain calibration over time. Camera lens calibration calibrates the camera's sensor and lens distortions and errors, which is used for calibration of the other elements. Frame registration establishes the relative positions and orientations of everything within the workspace with respect to each other or to a shared coordinate system. End of arm tool contact calibration in one embodiment is referred to as tool center point (TCP) calibration, which calibrates the tool tip locations of interest, or tool lines of interest, or tool contact planes of interest that interface the tool with a product or work piece. The end of arm tool contact calibration may have a position in space as well as an orientation. The combination of all positions and orientations of interest for a component is called a pose. The contact point may be a dynamic fluid or plasma, as in soldering or welding. Robot pose error compensation provides an adjustment to compensate for the effect of the accumulated inaccuracies of manufacturing, wear, and other causes of inaccuracies in the components of the robotic arm or other component. Robot pose errors may be caused a variety of sources such as mechanical hysteresis, backlash, thermal expansion, and loading. In one embodiment, calibration monitoring provides continuous monitoring of the calibration state of the robotic cell system, and constant correction. Thus, the system calibrates the dynamic and static components in the robotic cell, and uses a learning process to model the dynamic and static components.") the feedback sensor comprising one or more of: a force sensor and an endoscopic camera. (Paragraph 0031, "In one embodiment, in addition to cameras, additional sensors such as magnetic sensors, infrared (IR) sensors, and other sensors may also be mounted within, or outside, the robotic cell to monitor the working area. Other sensors which may be part of the robotic cell may include one or more of sound sensors, vibration sensors, motor torque sensors, force sensors (load cells), motion sensors for sensing velocity and/or acceleration, light sensors, and temperature sensors. In one embodiment, the system integrates all of this sensor data to accurately represent the environment as a systemic representation." Examiner Note: An endoscopic camera is generally understood as a camera being a part of an endoscope which is used by doctors to capture imaging of internal organs and tissues.)
Regarding claim 8, where all the limitations of claim 7 are discussed above, Poelman further teaches:
8. (Original) The method of claim 7, wherein the feedback sensor (Paragraph 0031, "In one embodiment, in addition to cameras, additional sensors such as magnetic sensors, infrared (IR) sensors, and other sensors may also be mounted within, or outside, the robotic cell to monitor the working area. Other sensors which may be part of the robotic cell may include one or more of sound sensors, vibration sensors, motor torque sensors, force sensors (load cells), motion sensors for sensing velocity and/or acceleration, light sensors, and temperature sensors. In one embodiment, the system integrates all of this sensor data to accurately represent the environment as a systemic representation.") is used for one or more of: fine-tuning navigation offsets, applying micro-adjustments to an estimated position during an assembly step, (Paragraphs 0024-0025, "By building up a holistic or system view of a robotic cell and its working area the auto-calibration system can provide high accuracy and precision, an iterative feedback loop, constant pose refinement, and camera refinement. Furthermore, in one embodiment, the auto-calibration system can accomplish this with off-the shelf pieces, rather than customized assemblies. In one embodiment, a plurality of low cost cameras may be used, in combination with geometric fiducial constraints, to provide high accuracy at a reduced cost and complexity. Auto-calibration may be used to encompass the entirety of the robotic cell environment, including all elements in a system, also referred to as a holistic view, a system view, or solving as a system. For example, in a modular robotic assembly system this may include one or more cameras and/or other sensors, a robotic arm, end of arm tools, the tray on which parts and additional tools are located, the conveyor which moves work pieces into and out of the robotic assembly system, and the individual work pieces. In one embodiment, the holistic view of the working area includes all elements, whether they are stationary or movable, and tracks and/or calibrates all elements.
Auto-calibration is possible in one embodiment using a combination of camera lens calibration, frame registration, end of arm tool contact calibration, robot pose error compensation, and calibration monitoring and using a feed forward control and constant correction to maintain calibration over time. Camera lens calibration calibrates the camera's sensor and lens distortions and errors, which is used for calibration of the other elements. Frame registration establishes the relative positions and orientations of everything within the workspace with respect to each other or to a shared coordinate system. End of arm tool contact calibration in one embodiment is referred to as tool center point (TCP) calibration, which calibrates the tool tip locations of interest, or tool lines of interest, or tool contact planes of interest that interface the tool with a product or work piece. The end of arm tool contact calibration may have a position in space as well as an orientation. The combination of all positions and orientations of interest for a component is called a pose. The contact point may be a dynamic fluid or plasma, as in soldering or welding. Robot pose error compensation provides an adjustment to compensate for the effect of the accumulated inaccuracies of manufacturing, wear, and other causes of inaccuracies in the components of the robotic arm or other component. Robot pose errors may be caused a variety of sources such as mechanical hysteresis, backlash, thermal expansion, and loading. In one embodiment, calibration monitoring provides continuous monitoring of the calibration state of the robotic cell system, and constant correction. Thus, the system calibrates the dynamic and static components in the robotic cell, and uses a learning process to model the dynamic and static components.") and applying offsets to compensate for vision drift during a production process. (Paragraph 0138, "At block 930, after the initial calibration, the system performs calibration refinement steps during idle cycles. This ensures that robot pose error over time, which may be the result of changes in temperature, drift due to movement, etc. are accounted for in the holistic representation and thus expectation.")
Regarding claim 9, where all the limitations of claim 1 are discussed above, Poelman further teaches:
9. (Original) The method of claim 1, further comprising performing an inspection of the product model comprising:
identifying regions of interest; (Paragraph 0027, "In one embodiment, one or more fiducials which are observable from various vantage points in the work area, are affixed to the robotic cell structure. In one embodiment, the fiducials are distributed through the work area (or volumetric distribution), so that each camera always sees at least one fiducial affixed to the work area, in addition to any fiducials on the robotic arm and/or a calibration board or other assembled element held by the robotic arm. However, even such “rigid” parts aren't truly rigid because robotic cells are physically moved, or someone may hit the cell or otherwise shift the robotic cell structure in a way to displace the rigid parts. Additionally, wear and tear may cause movement or displacement. The present system, in one embodiment, can identify such displacements so that the robotic cell structure may be re-calibrated when such a shift occurs.")
taking a plurality of images of the regions of interest; (Paragraph 0073, "The localization process 274 localizes one or more cameras relative to other features of the cell, such as a robotic arm and/or other elements such as fiducials, fixtures, pallets, parts, feeders, trays, visual features, points of interest, etc. within the robotic cell. In one embodiment, the system uses a virtual origin frame to localize the cameras in space, relative to the robotic cell. In one embodiment, the process then creates a map, including accuracy v. precision curves. The map is continuously updated, in one embodiment. The frame registry provides positional data. In one embodiment, the system provides flexible storage and retrieval of images used to compute positional data.")
performing model inspection verification on the images to verify location predictions; (Paragraph 0088, "At block 440, the process determines whether the observed real-world situation is different from the expectation based on the holistic conception of the robotic cell, that is whether there are discrepancies in the robotic arm position between the predicted position and the observed position. In one embodiment, the system compares the real world pose and movement pattern (ex. approach vector for each joint) of the robotic arm, tool, and/or work piece to the expectation based on the holistic 3D conception of the robotic system. If the discrepancy is within an acceptable threshold, the current step in the process is executed, at block 450. At block 460, the process determines whether the routine has been completed. If not, the process returns to block 430 and continues observing the position/movement of components. Note that while this is illustrated as a flowchart in the real world the monitoring is continuous, and in one embodiment an interrupt is triggered when a difference is detected between the expectation and the reality.")
determining that the location predictions are correct (Paragraphs 0079-0081, "At block 340, the calibration routine is identified for the cell. In one embodiment, the calibration routine is based on the production line and cell configuration. For example, for a robotic cell that does pick-and-place, the calibration routine may be different than for a robotic cell that tightens a screw, because the range of motion and the path taken by the robotic arm and the work pieces is different. The auto-calibration process is executed.
At block 350, based on the data from the calibration process the system creates a holistic conception of the robotic cell's workspace, including the robotic arm, end-of-arm tool, work pieces within the workspace, etc. The holistic conception is used to set expectations for the movements and commands.
At block 355, a virtual representation of the robotic cells is created, based on the holistic conception. The virtual representation of the cell is designed to be closely aligned to the actual cell so that the mapped actions are performed similarly. This enables the system to build expectations for the position and movement pattern of the robotic arm, as well as the other elements within the workspace. The system utilizes this expectation to validate the operation of the cell.") and approving the product model for deployment. (Paragraph 0082, "At block 360, a recipe is deployed and the line is run. A recipe in one embodiment is the sequence of commands sent to a robotic cell to execute one or more actions. A recipe may be static or dynamic, simple, or complex. A simple static recipe may be “move conveyor at time X by Y inches.” A complex recipe may include obtaining data from a camera, and conditionally inserting a screw, if a part is configured correctly.")
Regarding claim 10, where all the limitations of claim 9 are discussed above, Poelman further teaches:
10. (Original) The method of claim 9, wherein when the location predictions are not correct, images with the incorrect prediction are collected and used to improve the predictions by training a machine learning system. (Paragraph 0089, "Once the routine is completed, in one embodiment, at block 470 a feedback loop is used to provide feedback to the system. The feedback loop provides data to the expectation and machine learning systems. The system may also recalibrate, if appropriate. In one embodiment, the system may utilize continuous micro-recalibrations, to keep the holistic conception completely aligned with the system. The process then ends, at block 475.")
Regarding claim 11, where all the limitations of claim 1 are discussed above, Poelman further teaches:
11. (Currently Amended) The method of claim 1, further comprising:
uploading the product model to a second robotic cell having a particular configuration; (Paragraph 0046-0049, "FIG. 1A is an overview block diagram of a system for a robotic factory. FIG. 1A is a simplified block diagram of one embodiment of a system in which robotic cells may be implemented. In one embodiment, robotic cells A10 include one or more individual robotic cells which together form the software defined manufacturing line, or micro factory A12. In one embodiment, individual robotic cells A10 may be linked via conveyors, and reverse conveyors, so that a single item being manufactured or assembled through the micro factory passes through one or more robotic cells A10 (or multiple times through one or more cells A10). The robotic cells A10 may provide manufacturing, assembly, inspection, and/or testing of products. For simplicity the term “manufacturing” will be used, however it should be understood that this term is being used for any process which is part of making a product, including inspection, manufacturing, validation, and testing.
In one embodiment, the robotic cells A10 are controlled by software. In one embodiment, the configuration and control data for the robotic cells A10 are applied to the cell from memory A20. In one embodiment, the memory A20 may be part of a remote system, coupled to the robotic cells A10 via network A05. The configuration data A25 defines the software configuration for each robotic cell A10 and the manufacturing line. Each robotic cell is calibrated prior to use. In one embodiment, the robotic cells A10 are also continuously calibrated, as will be described below in more detail. In one embodiment, the robotic cells A10 collect operational data while being calibrated, tested, and used. This operational data A30 is stored in memory A20 and used by machine learning system A35. In one embodiment, local storage A15 provides backup for configuration data for the robotic cell, as well as operational data produced by the robotic cell while it is in use. Local storage A15 in one embodiment acts as a buffer for memory A20. In one embodiment, if the robotic cell A10 becomes disconnected from the network A05, it may continue to operate and collect real time operational data, using local storage A15. This also enables offline operation of a robotic cell A10.
In one embodiment, because the cells are software configured, a single robotic cell A10 may perform multiple stages in the manufacturing process and may be reconfigured during the manufacturing process. In one embodiment, this also enables the substitution of robotic cells A10 in a micro factory during manufacturing without extensive reconfiguration. In one embodiment, this also permits the addition of cells into a micro factory.")
…
inspecting the device; (Paragraph 0046, "FIG. 1A is an overview block diagram of a system for a robotic factory. FIG. 1A is a simplified block diagram of one embodiment of a system in which robotic cells may be implemented. In one embodiment, robotic cells A10 include one or more individual robotic cells which together form the software defined manufacturing line, or micro factory A12. In one embodiment, individual robotic cells A10 may be linked via conveyors, and reverse conveyors, so that a single item being manufactured or assembled through the micro factory passes through one or more robotic cells A10 (or multiple times through one or more cells A10). The robotic cells A10 may provide manufacturing, assembly, inspection, and/or testing of products. For simplicity the term “manufacturing” will be used, however it should be understood that this term is being used for any process which is part of making a product, including inspection, manufacturing, validation, and testing.") and
when the partial assembly method and the inspecting show that the assembly was successful, (Paragraphs 0112-0114, "At block 675, the process determines whether the elements are sufficiently accurate to meet an accuracy threshold. In one embodiment, the system may refine the calibrations multiple times. In one embodiment, the system may provide an initial calibration, and then test whether the quality thresholds are met, and refine as necessary. The test whether the calibration, pose estimates, and errors are within the parameters may be performed after any of the steps above, as well as here.
Once the calibrations are sufficiently refined, at block 680 multi-parameter optimization is performed in one embodiment.
At block 685, in one embodiment, the system is validated with an external fiducial. In one embodiment, that involves utilizing one or more separate fiducials or natural visual features to verify that each camera lens, camera pose, and tool contact point calibration are all accurate, and the error model accounts for inaccuracies. The process then ends at block 690.") approving the product model for use by the robotic cells having the particular configuration. (Paragraph 0082, "At block 360, a recipe is deployed and the line is run. A recipe in one embodiment is the sequence of commands sent to a robotic cell to execute one or more actions. A recipe may be static or dynamic, simple, or complex. A simple static recipe may be “move conveyor at time X by Y inches.” A complex recipe may include obtaining data from a camera, and conditionally inserting a screw, if a part is configured correctly.")
Poelman does not specifically teach validation using partial assembly. However, Attar, in the same field of endeavor of robotics, teaches:
… running the partial assembly method for all components; (Paragraph 0425, "In the illustrated example, the position sensor system can include one or more imaging sensors 1502, 1504 located over, or around the assembly robots 204. In some example cases, images captured by imaging sensors 1502, 1504 can be used for: (i) real-time collision avoidance and safety—monitoring for obstacles in the assembly robot's motion pathway that may otherwise obstruct motion of the assembly robot. Obstacles may include other assembly robots, or otherwise any dynamic objects moving in the environment (e.g., near-by human operators); (ii) object detection and verification—capturing images to locate objects (or building parts) requiring pick-up and assembly, and further verifying that the correct object has been picked-up, or the object is picked-up in the correct orientation; (iii) object grasp verification—verifying that a picked-up object is grasped correctly by the assembly robot; and/or (iv) object quality inspection—after (or before) a building part is picked-up, it may be imaged to detect any flaws that compromise the quality of the building part. Building parts that are poor quality may not be used for assembly. For example, as shown in images 1702 and 1704 (FIG. 17), this can include detecting various cracks and fissures; (v) building part re-alignment, or otherwise robot pose correction; and/or (vi) post-assembly inspection of building structures—images of assembled, or partially assembled, building structures can be inspected to automatically identify defects in requiring modification.") …
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic system and control methods as taught by Poelman with the ability to validate and verify the process using partial assembly methods as taught by Attar. This would ensure that defects and instances where the system does not meet expectations may be quickly and automatically addressed.
Regarding claim 12, where all the limitations of claim 11 are discussed above, Poelman further teaches:
12. (Original) The method of claim 11, further comprising:
when the inspecting ends in a failure, (Paragraph 0038, "The ecosystem has a variety of potential sources of errors—for example lens distortions for each camera, robot motion backlash, hysteresis, thermal expansion, etc. The auto-calibration system solves for these errors at the system level, holistically as an ecosystem, taking into account contributions from all potential error sources. For example, if the tool isn't where it is expected to be, the error might be due to lens distortion, or due to robot motion backlash, or both. To decide how much error is caused by the lens vs. backlash, the system can utilize data from other sensors in the system—but those other sensors may have their own errors. Thus, solving for the source of one error depends on other sensors with other errors. It is a system of simultaneous equations and constraints.") refining regions of interest (Paragraph 0039, "A constraint might be that the system can assume that the fiducials are on flat surfaces—this enables verifying whether there is lens distortion. If a lens is distorted, it would make the fiducial look like it is on a non-flat surface. The system can then correct for the lens distortion. The algorithm can solve the compound system by a series of multiple sensor observations and constraints as a global ecosystem and refine (iteratively and/or directly). One way to describe this is “global error minimization and optimization by holistic mathematical analysis of observations and constraints across multiple objects and features of interest.” This framework can be used for camera, robot, and end of arm tool contact calibration. In one embodiment, this framework is used for optimization by pose error minimization and re-projection error minimization.") and taking additional images of the refined region of interest. (Paragraph 0042, "In one embodiment, the system continuously performs mini-calibrations. In one embodiment this is done during idle cycles of the processor, with data captured by the cameras and/or sensors. In one embodiment, a watch-dog process monitors the mini-calibrations to identify when a full re-calibration is needed. In one embodiment, over time the data is used in a machine learning system to fine-tune when calibration is needed, and identify what changes or occurrences lead to a need for recalibration. This may be used to improve the robotic cell itself, or address issues that knock cells out of calibration. In one embodiment, a user interface provides a way to display the calibration output, and statistics. Over time, this may lead to improved cell design, as issues which cause the cell to become unstable or require recalibration are identified and eliminated.")
Regarding claim 13, Poelman further teaches:
13. (Currently Amended) A system to enable automated assembly of a device using a robotic cell comprising:
calibrator to calibrate a robotic cell (Paragraph 0067, "The workspace in one embodiment may include other elements such as the workpiece, and the conveyor belt(s) 226 which move work pieces to and from the robotic cell. Other elements may include a tray 227 on which a workpiece, or other parts to be added to the workpiece are located, and the path from the tray feeder. Other elements may include the end of arm tool 228, and optionally a tool switch if other tools are present. A “tool switch” refers to a special mounting interface on the robotic arm that allows the end-of-arm tool to be changed (either manually or automatically via software control). This enables the auto-calibration routine to calibrate multiple tools. In one embodiment, this also enables the use of a special calibration tool for the robot, enabling the system to calibrate and then re-mount an end-of-arm tool. Other elements in the work space may include safety systems such as light curtains, an electrical system, illumination systems, sensors, cable management systems, and other elements which are within the workspace of the robotic cell, and thus may be considered as part of the robotic cell system.") the robotic cell including an end-of-arm sensor; (Paragraph 0062, "FIG. 1E is a diagram showing one embodiment of the elements of defining a holistic view of a workspace of the robotic cell. The workspace is defined in one embodiment by a frame and enclosure within which the robotic arm can move to take action on work pieces. In one embodiment, the workspace includes two or more cameras to provide a stereoscopic view, or multi-camera three dimensional view, of the space. In one embodiment, there is an additional camera on a side of the frame. In one embodiment, there is a camera at the end of arm. Additional cameras may also be used. The different cameras with different viewpoints give different benefits.")
a product model generator to generate the product model of the device for assembly; (Paragraph 0149, "FIG. 11 is a simplified block diagram of the process which utilizes the micro factory comprising one or more robotic cells to create assembled final products. In some embodiments, the robotic cells may be inserted into a traditional manufacturing line, to take over some sub-portions of the manufacturing. The process includes the layers of manufacturing, from recipe creation via recipe creator C10 to fabrication/assembly via micro factory C50. Although a complete process is shown, from initial concept/functional design through completed manufacturing, one of skill in the art would understand that the system may implement a subset of these processes and include a subset of these features.")
…
a model store to make the product model available for use by robotic cells for navigation, validation, and/or inspection. (Please see Figure 1A as well as Paragraph 0047, "In one embodiment, the robotic cells A10 are controlled by software. In one embodiment, the configuration and control data for the robotic cells A10 are applied to the cell from memory A20. In one embodiment, the memory A20 may be part of a remote system, coupled to the robotic cells A10 via network A05. The configuration data A25 defines the software configuration for each robotic cell A10 and the manufacturing line. Each robotic cell is calibrated prior to use. In one embodiment, the robotic cells A10 are also continuously calibrated, as will be described below in more detail.")
Poelman does not specifically teach validation using partial assembly. However, Attar, in the same field of endeavor of robotics, teaches:
… a validator to validate the product model using a partial assembly method; (Paragraph 0425, "In the illustrated example, the position sensor system can include one or more imaging sensors 1502, 1504 located over, or around the assembly robots 204. In some example cases, images captured by imaging sensors 1502, 1504 can be used for: (i) real-time collision avoidance and safety—monitoring for obstacles in the assembly robot's motion pathway that may otherwise obstruct motion of the assembly robot. Obstacles may include other assembly robots, or otherwise any dynamic objects moving in the environment (e.g., near-by human operators); (ii) object detection and verification—capturing images to locate objects (or building parts) requiring pick-up and assembly, and further verifying that the correct object has been picked-up, or the object is picked-up in the correct orientation; (iii) object grasp verification—verifying that a picked-up object is grasped correctly by the assembly robot; and/or (iv) object quality inspection—after (or before) a building part is picked-up, it may be imaged to detect any flaws that compromise the quality of the building part. Building parts that are poor quality may not be used for assembly. For example, as shown in images 1702 and 1704 (FIG. 17), this can include detecting various cracks and fissures; (v) building part re-alignment, or otherwise robot pose correction; and/or (vi) post-assembly inspection of building structures—images of assembled, or partially assembled, building structures can be inspected to automatically identify defects in requiring modification.") …
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic system and control methods as taught by Poelman with the ability to validate and verify the process using partial assembly methods as taught by Attar. This would ensure that defects and instances where the system does not meet expectations may be quickly and automatically addressed.
Regarding claim 17, where all the limitations of claim 13 are discussed above, Poelman further teaches:
17. (Original) The system of claim 13, further comprising:
an inspector to perform an inspection of the product model comprising:
identifying regions of interest; (Paragraph 0027, "In one embodiment, one or more fiducials which are observable from various vantage points in the work area, are affixed to the robotic cell structure. In one embodiment, the fiducials are distributed through the work area (or volumetric distribution), so that each camera always sees at least one fiducial affixed to the work area, in addition to any fiducials on the robotic arm and/or a calibration board or other assembled element held by the robotic arm. However, even such “rigid” parts aren't truly rigid because robotic cells are physically moved, or someone may hit the cell or otherwise shift the robotic cell structure in a way to displace the rigid parts. Additionally, wear and tear may cause movement or displacement. The present system, in one embodiment, can identify such displacements so that the robotic cell structure may be re-calibrated when such a shift occurs.")
taking a plurality of images of the regions of interest; (Paragraph 0073, "The localization process 274 localizes one or more cameras relative to other features of the cell, such as a robotic arm and/or other elements such as fiducials, fixtures, pallets, parts, feeders, trays, visual features, points of interest, etc. within the robotic cell. In one embodiment, the system uses a virtual origin frame to localize the cameras in space, relative to the robotic cell. In one embodiment, the process then creates a map, including accuracy v. precision curves. The map is continuously updated, in one embodiment. The frame registry provides positional data. In one embodiment, the system provides flexible storage and retrieval of images used to compute positional data.")
performing model inspection verification on the images to verify accuracy of location predictions; (Paragraph 0088, "At block 440, the process determines whether the observed real-world situation is different from the expectation based on the holistic conception of the robotic cell, that is whether there are discrepancies in the robotic arm position between the predicted position and the observed position. In one embodiment, the system compares the real world pose and movement pattern (ex. approach vector for each joint) of the robotic arm, tool, and/or work piece to the expectation based on the holistic 3D conception of the robotic system. If the discrepancy is within an acceptable threshold, the current step in the process is executed, at block 450. At block 460, the process determines whether the routine has been completed. If not, the process returns to block 430 and continues observing the position/movement of components. Note that while this is illustrated as a flowchart in the real world the monitoring is continuous, and in one embodiment an interrupt is triggered when a difference is detected between the expectation and the reality.")
determining that the predictions are correct (Paragraphs 0079-0081, "At block 340, the calibration routine is identified for the cell. In one embodiment, the calibration routine is based on the production line and cell configuration. For example, for a robotic cell that does pick-and-place, the calibration routine may be different than for a robotic cell that tightens a screw, because the range of motion and the path taken by the robotic arm and the work pieces is different. The auto-calibration process is executed.
At block 350, based on the data from the calibration process the system creates a holistic conception of the robotic cell's workspace, including the robotic arm, end-of-arm tool, work pieces within the workspace, etc. The holistic conception is used to set expectations for the movements and commands.
At block 355, a virtual representation of the robotic cells is created, based on the holistic conception. The virtual representation of the cell is designed to be closely aligned to the actual cell so that the mapped actions are performed similarly. This enables the system to build expectations for the position and movement pattern of the robotic arm, as well as the other elements within the workspace. The system utilizes this expectation to validate the operation of the cell.") and approving the product model for deployment. (Paragraph 0082, "At block 360, a recipe is deployed and the line is run. A recipe in one embodiment is the sequence of commands sent to a robotic cell to execute one or more actions. A recipe may be static or dynamic, simple, or complex. A simple static recipe may be “move conveyor at time X by Y inches.” A complex recipe may include obtaining data from a camera, and conditionally inserting a screw, if a part is configured correctly.")
Regarding claim 18, where all the limitations of claim 13 are discussed above, Poelman further teaches:
18. (Original) The system of claim 17, wherein when the predictions are not correct, images with the incorrect prediction are collected and used to improve the predictions by training a machine learning system. (Paragraph 0089, "Once the routine is completed, in one embodiment, at block 470 a feedback loop is used to provide feedback to the system. The feedback loop provides data to the expectation and machine learning systems. The system may also recalibrate, if appropriate. In one embodiment, the system may utilize continuous micro-recalibrations, to keep the holistic conception completely aligned with the system. The process then ends, at block 475.")
Regarding claim 19, where all the limitations of claim 17 are discussed above, Poelman further teaches:
19. (Original) The system of claim 17, further comprising, the inspector further to add more images (Paragraph 0042, "In one embodiment, the system continuously performs mini-calibrations. In one embodiment this is done during idle cycles of the processor, with data captured by the cameras and/or sensors. In one embodiment, a watch-dog process monitors the mini-calibrations to identify when a full re-calibration is needed. In one embodiment, over time the data is used in a machine learning system to fine-tune when calibration is needed, and identify what changes or occurrences lead to a need for recalibration. This may be used to improve the robotic cell itself, or address issues that knock cells out of calibration. In one embodiment, a user interface provides a way to display the calibration output, and statistics. Over time, this may lead to improved cell design, as issues which cause the cell to become unstable or require recalibration are identified and eliminated.") and refining the region of interest, (Paragraph 0039, "A constraint might be that the system can assume that the fiducials are on flat surfaces—this enables verifying whether there is lens distortion. If a lens is distorted, it would make the fiducial look like it is on a non-flat surface. The system can then correct for the lens distortion. The algorithm can solve the compound system by a series of multiple sensor observations and constraints as a global ecosystem and refine (iteratively and/or directly). One way to describe this is “global error minimization and optimization by holistic mathematical analysis of observations and constraints across multiple objects and features of interest.” This framework can be used for camera, robot, and end of arm tool contact calibration. In one embodiment, this framework is used for optimization by pose error minimization and re-projection error minimization.") when one or more of the inspections end in a failure. (Paragraph 0038, "The ecosystem has a variety of potential sources of errors—for example lens distortions for each camera, robot motion backlash, hysteresis, thermal expansion, etc. The auto-calibration system solves for these errors at the system level, holistically as an ecosystem, taking into account contributions from all potential error sources. For example, if the tool isn't where it is expected to be, the error might be due to lens distortion, or due to robot motion backlash, or both. To decide how much error is caused by the lens vs. backlash, the system can utilize data from other sensors in the system—but those other sensors may have their own errors. Thus, solving for the source of one error depends on other sensors with other errors. It is a system of simultaneous equations and constraints.")
Regarding claim 20, where all the limitations of claim 13 are discussed above, Poelman further teaches:
20. (Currently Amended) The system of claim 13 further comprising:
a feedback sensor to fine-tune navigation offsets, (Paragraphs 0024-0025, "By building up a holistic or system view of a robotic cell and its working area the auto-calibration system can provide high accuracy and precision, an iterative feedback loop, constant pose refinement, and camera refinement. Furthermore, in one embodiment, the auto-calibration system can accomplish this with off-the shelf pieces, rather than customized assemblies. In one embodiment, a plurality of low cost cameras may be used, in combination with geometric fiducial constraints, to provide high accuracy at a reduced cost and complexity. Auto-calibration may be used to encompass the entirety of the robotic cell environment, including all elements in a system, also referred to as a holistic view, a system view, or solving as a system. For example, in a modular robotic assembly system this may include one or more cameras and/or other sensors, a robotic arm, end of arm tools, the tray on which parts and additional tools are located, the conveyor which moves work pieces into and out of the robotic assembly system, and the individual work pieces. In one embodiment, the holistic view of the working area includes all elements, whether they are stationary or movable, and tracks and/or calibrates all elements.
Auto-calibration is possible in one embodiment using a combination of camera lens calibration, frame registration, end of arm tool contact calibration, robot pose error compensation, and calibration monitoring and using a feed forward control and constant correction to maintain calibration over time. Camera lens calibration calibrates the camera's sensor and lens distortions and errors, which is used for calibration of the other elements. Frame registration establishes the relative positions and orientations of everything within the workspace with respect to each other or to a shared coordinate system. End of arm tool contact calibration in one embodiment is referred to as tool center point (TCP) calibration, which calibrates the tool tip locations of interest, or tool lines of interest, or tool contact planes of interest that interface the tool with a product or work piece. The end of arm tool contact calibration may have a position in space as well as an orientation. The combination of all positions and orientations of interest for a component is called a pose. The contact point may be a dynamic fluid or plasma, as in soldering or welding. Robot pose error compensation provides an adjustment to compensate for the effect of the accumulated inaccuracies of manufacturing, wear, and other causes of inaccuracies in the components of the robotic arm or other component. Robot pose errors may be caused a variety of sources such as mechanical hysteresis, backlash, thermal expansion, and loading. In one embodiment, calibration monitoring provides continuous monitoring of the calibration state of the robotic cell system, and constant correction. Thus, the system calibrates the dynamic and static components in the robotic cell, and uses a learning process to model the dynamic and static components.") the feedback sensor comprising one or more of: a force sensor and an endoscopic camera, (Paragraph 0031, "In one embodiment, in addition to cameras, additional sensors such as magnetic sensors, infrared (IR) sensors, and other sensors may also be mounted within, or outside, the robotic cell to monitor the working area. Other sensors which may be part of the robotic cell may include one or more of sound sensors, vibration sensors, motor torque sensors, force sensors (load cells), motion sensors for sensing velocity and/or acceleration, light sensors, and temperature sensors. In one embodiment, the system integrates all of this sensor data to accurately represent the environment as a systemic representation." Examiner Note: An endoscopic camera is generally understood as a camera being a part of an endoscope which is used by doctors to capture imaging of internal organs and tissues.) wherein the feedback sensor (Paragraph 0031, "In one embodiment, in addition to cameras, additional sensors such as magnetic sensors, infrared (IR) sensors, and other sensors may also be mounted within, or outside, the robotic cell to monitor the working area. Other sensors which may be part of the robotic cell may include one or more of sound sensors, vibration sensors, motor torque sensors, force sensors (load cells), motion sensors for sensing velocity and/or acceleration, light sensors, and temperature sensors. In one embodiment, the system integrates all of this sensor data to accurately represent the environment as a systemic representation.") is used for one or more of: fine-tuning navigation offsets,
applying micro-adjustments to an estimated position during an assembly step, (Paragraphs 0024-0025, "By building up a holistic or system view of a robotic cell and its working area the auto-calibration system can provide high accuracy and precision, an iterative feedback loop, constant pose refinement, and camera refinement. Furthermore, in one embodiment, the auto-calibration system can accomplish this with off-the shelf pieces, rather than customized assemblies. In one embodiment, a plurality of low cost cameras may be used, in combination with geometric fiducial constraints, to provide high accuracy at a reduced cost and complexity. Auto-calibration may be used to encompass the entirety of the robotic cell environment, including all elements in a system, also referred to as a holistic view, a system view, or solving as a system. For example, in a modular robotic assembly system this may include one or more cameras and/or other sensors, a robotic arm, end of arm tools, the tray on which parts and additional tools are located, the conveyor which moves work pieces into and out of the robotic assembly system, and the individual work pieces. In one embodiment, the holistic view of the working area includes all elements, whether they are stationary or movable, and tracks and/or calibrates all elements.
Auto-calibration is possible in one embodiment using a combination of camera lens calibration, frame registration, end of arm tool contact calibration, robot pose error compensation, and calibration monitoring and using a feed forward control and constant correction to maintain calibration over time. Camera lens calibration calibrates the camera's sensor and lens distortions and errors, which is used for calibration of the other elements. Frame registration establishes the relative positions and orientations of everything within the workspace with respect to each other or to a shared coordinate system. End of arm tool contact calibration in one embodiment is referred to as tool center point (TCP) calibration, which calibrates the tool tip locations of interest, or tool lines of interest, or tool contact planes of interest that interface the tool with a product or work piece. The end of arm tool contact calibration may have a position in space as well as an orientation. The combination of all positions and orientations of interest for a component is called a pose. The contact point may be a dynamic fluid or plasma, as in soldering or welding. Robot pose error compensation provides an adjustment to compensate for the effect of the accumulated inaccuracies of manufacturing, wear, and other causes of inaccuracies in the components of the robotic arm or other component. Robot pose errors may be caused a variety of sources such as mechanical hysteresis, backlash, thermal expansion, and loading. In one embodiment, calibration monitoring provides continuous monitoring of the calibration state of the robotic cell system, and constant correction. Thus, the system calibrates the dynamic and static components in the robotic cell, and uses a learning process to model the dynamic and static components.") and applying offsets to compensate for vision drift during a production process. (Paragraph 0138, "At block 930, after the initial calibration, the system performs calibration refinement steps during idle cycles. This ensures that robot pose error over time, which may be the result of changes in temperature, drift due to movement, etc. are accounted for in the holistic representation and thus expectation.")
Regarding claim 21, Poelman further teaches:
21. (Currently Amended) A method of automated assembly of a device using a robotic cell comprising:
calibrating a robotic cell (Paragraph 0067, "The workspace in one embodiment may include other elements such as the workpiece, and the conveyor belt(s) 226 which move work pieces to and from the robotic cell. Other elements may include a tray 227 on which a workpiece, or other parts to be added to the workpiece are located, and the path from the tray feeder. Other elements may include the end of arm tool 228, and optionally a tool switch if other tools are present. A “tool switch” refers to a special mounting interface on the robotic arm that allows the end-of-arm tool to be changed (either manually or automatically via software control). This enables the auto-calibration routine to calibrate multiple tools. In one embodiment, this also enables the use of a special calibration tool for the robot, enabling the system to calibrate and then re-mount an end-of-arm tool. Other elements in the work space may include safety systems such as light curtains, an electrical system, illumination systems, sensors, cable management systems, and other elements which are within the workspace of the robotic cell, and thus may be considered as part of the robotic cell system.") for generating a product model, (Paragraph 0149, "FIG. 11 is a simplified block diagram of the process which utilizes the micro factory comprising one or more robotic cells to create assembled final products. In some embodiments, the robotic cells may be inserted into a traditional manufacturing line, to take over some sub-portions of the manufacturing. The process includes the layers of manufacturing, from recipe creation via recipe creator C10 to fabrication/assembly via micro factory C50. Although a complete process is shown, from initial concept/functional design through completed manufacturing, one of skill in the art would understand that the system may implement a subset of these processes and include a subset of these features.") the robotic cell including an end-of-arm sensor; (Paragraph 0062, "FIG. 1E is a diagram showing one embodiment of the elements of defining a holistic view of a workspace of the robotic cell. The workspace is defined in one embodiment by a frame and enclosure within which the robotic arm can move to take action on work pieces. In one embodiment, the workspace includes two or more cameras to provide a stereoscopic view, or multi-camera three dimensional view, of the space. In one embodiment, there is an additional camera on a side of the frame. In one embodiment, there is a camera at the end of arm. Additional cameras may also be used. The different cameras with different viewpoints give different benefits.")
utilizing the robotic cell to generate the product model of the device for assembly, (Paragraph 0149, "FIG. 11 is a simplified block diagram of the process which utilizes the micro factory comprising one or more robotic cells to create assembled final products. In some embodiments, the robotic cells may be inserted into a traditional manufacturing line, to take over some sub-portions of the manufacturing. The process includes the layers of manufacturing, from recipe creation via recipe creator C10 to fabrication/assembly via micro factory C50. Although a complete process is shown, from initial concept/functional design through completed manufacturing, one of skill in the art would understand that the system may implement a subset of these processes and include a subset of these features.") the product model including a plurality of regions of interest, (Paragraph 0027, "In one embodiment, one or more fiducials which are observable from various vantage points in the work area, are affixed to the robotic cell structure. In one embodiment, the fiducials are distributed through the work area (or volumetric distribution), so that each camera always sees at least one fiducial affixed to the work area, in addition to any fiducials on the robotic arm and/or a calibration board or other assembled element held by the robotic arm. However, even such “rigid” parts aren't truly rigid because robotic cells are physically moved, or someone may hit the cell or otherwise shift the robotic cell structure in a way to displace the rigid parts. Additionally, wear and tear may cause movement or displacement. The present system, in one embodiment, can identify such displacements so that the robotic cell structure may be re-calibrated when such a shift occurs.") each of the regions of interest corresponding to an area for component insertion during the assembly; (Paragraph 0082, "At block 360, a recipe is deployed and the line is run. A recipe in one embodiment is the sequence of commands sent to a robotic cell to execute one or more actions. A recipe may be static or dynamic, simple, or complex. A simple static recipe may be “move conveyor at time X by Y inches.” A complex recipe may include obtaining data from a camera, and conditionally inserting a screw, if a part is configured correctly.")
…
adding robotic cell specific compensation, based on the validating; (Paragraphs 0024-0025, "By building up a holistic or system view of a robotic cell and its working area the auto-calibration system can provide high accuracy and precision, an iterative feedback loop, constant pose refinement, and camera refinement. Furthermore, in one embodiment, the auto-calibration system can accomplish this with off-the shelf pieces, rather than customized assemblies. In one embodiment, a plurality of low cost cameras may be used, in combination with geometric fiducial constraints, to provide high accuracy at a reduced cost and complexity. Auto-calibration may be used to encompass the entirety of the robotic cell environment, including all elements in a system, also referred to as a holistic view, a system view, or solving as a system. For example, in a modular robotic assembly system this may include one or more cameras and/or other sensors, a robotic arm, end of arm tools, the tray on which parts and additional tools are located, the conveyor which moves work pieces into and out of the robotic assembly system, and the individual work pieces. In one embodiment, the holistic view of the working area includes all elements, whether they are stationary or movable, and tracks and/or calibrates all elements.
Auto-calibration is possible in one embodiment using a combination of camera lens calibration, frame registration, end of arm tool contact calibration, robot pose error compensation, and calibration monitoring and using a feed forward control and constant correction to maintain calibration over time. Camera lens calibration calibrates the camera's sensor and lens distortions and errors, which is used for calibration of the other elements. Frame registration establishes the relative positions and orientations of everything within the workspace with respect to each other or to a shared coordinate system. End of arm tool contact calibration in one embodiment is referred to as tool center point (TCP) calibration, which calibrates the tool tip locations of interest, or tool lines of interest, or tool contact planes of interest that interface the tool with a product or work piece. The end of arm tool contact calibration may have a position in space as well as an orientation. The combination of all positions and orientations of interest for a component is called a pose. The contact point may be a dynamic fluid or plasma, as in soldering or welding. Robot pose error compensation provides an adjustment to compensate for the effect of the accumulated inaccuracies of manufacturing, wear, and other causes of inaccuracies in the components of the robotic arm or other component. Robot pose errors may be caused a variety of sources such as mechanical hysteresis, backlash, thermal expansion, and loading. In one embodiment, calibration monitoring provides continuous monitoring of the calibration state of the robotic cell system, and constant correction. Thus, the system calibrates the dynamic and static components in the robotic cell, and uses a learning process to model the dynamic and static components.")
performing a final inspection of the product model; (Paragraph 0046, "FIG. 1A is an overview block diagram of a system for a robotic factory. FIG. 1A is a simplified block diagram of one embodiment of a system in which robotic cells may be implemented. In one embodiment, robotic cells A10 include one or more individual robotic cells which together form the software defined manufacturing line, or micro factory A12. In one embodiment, individual robotic cells A10 may be linked via conveyors, and reverse conveyors, so that a single item being manufactured or assembled through the micro factory passes through one or more robotic cells A10 (or multiple times through one or more cells A10). The robotic cells A10 may provide manufacturing, assembly, inspection, and/or testing of products. For simplicity the term “manufacturing” will be used, however it should be understood that this term is being used for any process which is part of making a product, including inspection, manufacturing, validation, and testing." As well as Paragraph 0083, “A recipe may include an entire assembly process, or a step in a process. As the recipe is run, an expectation of position and movement based on the holistic representation is used to check the accuracy of the process. If any discrepancy is identified it can therefore be immediately addressed.”) and
releasing the product model. (Paragraph 0082, "At block 360, a recipe is deployed and the line is run. A recipe in one embodiment is the sequence of commands sent to a robotic cell to execute one or more actions. A recipe may be static or dynamic, simple, or complex. A simple static recipe may be “move conveyor at time X by Y inches.” A complex recipe may include obtaining data from a camera, and conditionally inserting a screw, if a part is configured correctly.")
Poelman does not specifically teach validation using partial assembly. However, Attar, in the same field of endeavor of robotics, teaches:
… validating the product model using a physical validation by partial insertion of an element into each of the areas; (Paragraph 0425, "In the illustrated example, the position sensor system can include one or more imaging sensors 1502, 1504 located over, or around the assembly robots 204. In some example cases, images captured by imaging sensors 1502, 1504 can be used for: (i) real-time collision avoidance and safety—monitoring for obstacles in the assembly robot's motion pathway that may otherwise obstruct motion of the assembly robot. Obstacles may include other assembly robots, or otherwise any dynamic objects moving in the environment (e.g., near-by human operators); (ii) object detection and verification—capturing images to locate objects (or building parts) requiring pick-up and assembly, and further verifying that the correct object has been picked-up, or the object is picked-up in the correct orientation; (iii) object grasp verification—verifying that a picked-up object is grasped correctly by the assembly robot; and/or (iv) object quality inspection—after (or before) a building part is picked-up, it may be imaged to detect any flaws that compromise the quality of the building part. Building parts that are poor quality may not be used for assembly. For example, as shown in images 1702 and 1704 (FIG. 17), this can include detecting various cracks and fissures; (v) building part re-alignment, or otherwise robot pose correction; and/or (vi) post-assembly inspection of building structures—images of assembled, or partially assembled, building structures can be inspected to automatically identify defects in requiring modification.") …
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic system and control methods as taught by Poelman with the ability to validate and verify the process using partial assembly methods as taught by Attar. This would ensure that defects and instances where the system does not meet expectations may be quickly and automatically addressed.
Claim(s) 2-4, 6, and 14-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Poelman in view of Attar and in further view of Pagnon et al. (US 20250262700 A1), hereinafter Pagnon.
Regarding claim 2, where all the limitations of claim 1 are discussed above, Poelman further teaches:
2. (Original) The method of claim 1, wherein generating the product model comprises:
identifying a reference point on the device and device scale; (Paragraph 0027, "In one embodiment, one or more fiducials which are observable from various vantage points in the work area, are affixed to the robotic cell structure. In one embodiment, the fiducials are distributed through the work area (or volumetric distribution), so that each camera always sees at least one fiducial affixed to the work area, in addition to any fiducials on the robotic arm and/or a calibration board or other assembled element held by the robotic arm. However, even such “rigid” parts aren't truly rigid because robotic cells are physically moved, or someone may hit the cell or otherwise shift the robotic cell structure in a way to displace the rigid parts. Additionally, wear and tear may cause movement or displacement. The present system, in one embodiment, can identify such displacements so that the robotic cell structure may be re-calibrated when such a shift occurs.")
…
Poelman does not specifically teach generating and analyzing a 3D model of the object/environment. However, Pagnon, in the same field of endeavor of robotic control, teaches:
… running a model creation client to create a 3D model representation of the device; (Paragraph 0085, "The robotic tool system 100 includes a scanner 115, coupled to a controller (not shown), the scanner 115 configured to generated a three dimensional model of the item 105 and its surroundings, including one or more obstacles 110.")
computing locations of components from the reference point; (Paragraph 0087, "The point cloud generated by the scanner 115 and/or controller may then be pre-processed to remove noise, to stitch point cloud data from multiple reference points, or perform any other processing or filtering of the data. In one embodiment, the scanner 115 may incorporate several scanning technologies (e.g. optical and laser scanning data), to generate the point cloud.")
verifying the computed locations of the components using a computer aided design (CAD) model of the device. (Paragraphs 0170-0171, "The system may align features of the scan data to the CAD model using one or more bounding boxes. The bounding box may be used to search for corresponding points in the point cloud.
The system may measure a deviation between the object and the model and refine the model according to the deviation. The refined model advantageously better matches the object than the original model.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic system and control methods as taught by Poelman with the ability to generate and analyze a 3D model of the target object and environment as taught by Pagnon. This would allow for simple and efficient analysis of the items and environment which the robot may then interact with safely in a plurality of different applications (See Pagnon, Paragraph 0083).
Regarding claim 3, where all the limitations of claim 2 are discussed above, Poelman further teaches:
3. (Original) The method of claim 2, wherein the reference point comprises a fiducial on a component. (Paragraph 0027, "In one embodiment, one or more fiducials which are observable from various vantage points in the work area, are affixed to the robotic cell structure. In one embodiment, the fiducials are distributed through the work area (or volumetric distribution), so that each camera always sees at least one fiducial affixed to the work area, in addition to any fiducials on the robotic arm and/or a calibration board or other assembled element held by the robotic arm. However, even such “rigid” parts aren't truly rigid because robotic cells are physically moved, or someone may hit the cell or otherwise shift the robotic cell structure in a way to displace the rigid parts. Additionally, wear and tear may cause movement or displacement. The present system, in one embodiment, can identify such displacements so that the robotic cell structure may be re-calibrated when such a shift occurs.")
Regarding claim 4, where all the limitations of claim 2 are discussed above, Poelman does not specifically teach verifying the product model using partial assembly. However, Attar, in the same field of endeavor of robotics, teaches:
4. (Original) The method of claim 2, wherein generating the product model further comprises:
verifying robustness of the product model using the partial assembly method. (Paragraph 0425, "In the illustrated example, the position sensor system can include one or more imaging sensors 1502, 1504 located over, or around the assembly robots 204. In some example cases, images captured by imaging sensors 1502, 1504 can be used for: (i) real-time collision avoidance and safety—monitoring for obstacles in the assembly robot's motion pathway that may otherwise obstruct motion of the assembly robot. Obstacles may include other assembly robots, or otherwise any dynamic objects moving in the environment (e.g., near-by human operators); (ii) object detection and verification—capturing images to locate objects (or building parts) requiring pick-up and assembly, and further verifying that the correct object has been picked-up, or the object is picked-up in the correct orientation; (iii) object grasp verification—verifying that a picked-up object is grasped correctly by the assembly robot; and/or (iv) object quality inspection—after (or before) a building part is picked-up, it may be imaged to detect any flaws that compromise the quality of the building part. Building parts that are poor quality may not be used for assembly. For example, as shown in images 1702 and 1704 (FIG. 17), this can include detecting various cracks and fissures; (v) building part re-alignment, or otherwise robot pose correction; and/or (vi) post-assembly inspection of building structures—images of assembled, or partially assembled, building structures can be inspected to automatically identify defects in requiring modification.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic system and control methods as taught by Poelman with the ability to validate and verify the process using partial assembly methods as taught by Attar. This would ensure that defects and instances where the system does not meet expectations may be quickly and automatically addressed.
Regarding claim 6, where all the limitations of claim 4 are discussed above, Poelman further teaches:
6. (Original) The method of claim 4, further comprising:
… apply a robotic cell specific compensation to the product model (Paragraphs 0024-0025, "By building up a holistic or system view of a robotic cell and its working area the auto-calibration system can provide high accuracy and precision, an iterative feedback loop, constant pose refinement, and camera refinement. Furthermore, in one embodiment, the auto-calibration system can accomplish this with off-the shelf pieces, rather than customized assemblies. In one embodiment, a plurality of low cost cameras may be used, in combination with geometric fiducial constraints, to provide high accuracy at a reduced cost and complexity. Auto-calibration may be used to encompass the entirety of the robotic cell environment, including all elements in a system, also referred to as a holistic view, a system view, or solving as a system. For example, in a modular robotic assembly system this may include one or more cameras and/or other sensors, a robotic arm, end of arm tools, the tray on which parts and additional tools are located, the conveyor which moves work pieces into and out of the robotic assembly system, and the individual work pieces. In one embodiment, the holistic view of the working area includes all elements, whether they are stationary or movable, and tracks and/or calibrates all elements. Auto-calibration is possible in one embodiment using a combination of camera lens calibration, frame registration, end of arm tool contact calibration, robot pose error compensation, and calibration monitoring and using a feed forward control and constant correction to maintain calibration over time. Camera lens calibration calibrates the camera's sensor and lens distortions and errors, which is used for calibration of the other elements. Frame registration establishes the relative positions and orientations of everything within the workspace with respect to each other or to a shared coordinate system. End of arm tool contact calibration in one embodiment is referred to as tool center point (TCP) calibration, which calibrates the tool tip locations of interest, or tool lines of interest, or tool contact planes of interest that interface the tool with a product or work piece. The end of arm tool contact calibration may have a position in space as well as an orientation. The combination of all positions and orientations of interest for a component is called a pose. The contact point may be a dynamic fluid or plasma, as in soldering or welding. Robot pose error compensation provides an adjustment to compensate for the effect of the accumulated inaccuracies of manufacturing, wear, and other causes of inaccuracies in the components of the robotic arm or other component. Robot pose errors may be caused a variety of sources such as mechanical hysteresis, backlash, thermal expansion, and loading. In one embodiment, calibration monitoring provides continuous monitoring of the calibration state of the robotic cell system, and constant correction. Thus, the system calibrates the dynamic and static components in the robotic cell, and uses a learning process to model the dynamic and static components.") …
Poelman does not specifically teach validation using partial assembly. However, Attar, in the same field of endeavor of robotics, teaches:
… when the partial assembly method indicates a mismatch, … and re-attempting the verifying. (Paragraph 0425, "In the illustrated example, the position sensor system can include one or more imaging sensors 1502, 1504 located over, or around the assembly robots 204. In some example cases, images captured by imaging sensors 1502, 1504 can be used for: (i) real-time collision avoidance and safety—monitoring for obstacles in the assembly robot's motion pathway that may otherwise obstruct motion of the assembly robot. Obstacles may include other assembly robots, or otherwise any dynamic objects moving in the environment (e.g., near-by human operators); (ii) object detection and verification—capturing images to locate objects (or building parts) requiring pick-up and assembly, and further verifying that the correct object has been picked-up, or the object is picked-up in the correct orientation; (iii) object grasp verification—verifying that a picked-up object is grasped correctly by the assembly robot; and/or (iv) object quality inspection—after (or before) a building part is picked-up, it may be imaged to detect any flaws that compromise the quality of the building part. Building parts that are poor quality may not be used for assembly. For example, as shown in images 1702 and 1704 (FIG. 17), this can include detecting various cracks and fissures; (v) building part re-alignment, or otherwise robot pose correction; and/or (vi) post-assembly inspection of building structures—images of assembled, or partially assembled, building structures can be inspected to automatically identify defects in requiring modification.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic system and control methods as taught by Poelman with the ability to validate and verify the process using partial assembly methods as taught by Attar. This would ensure that defects and instances where the system does not meet expectations may be quickly and automatically addressed.
Regarding claim 14, where all the limitations of claim 13 are discussed above, Poelman further teaches:
14. (Original) The system of claim 13, wherein the product model generator is further to:
identify a reference point on the device and device scale; (Paragraph 0027, "In one embodiment, one or more fiducials which are observable from various vantage points in the work area, are affixed to the robotic cell structure. In one embodiment, the fiducials are distributed through the work area (or volumetric distribution), so that each camera always sees at least one fiducial affixed to the work area, in addition to any fiducials on the robotic arm and/or a calibration board or other assembled element held by the robotic arm. However, even such “rigid” parts aren't truly rigid because robotic cells are physically moved, or someone may hit the cell or otherwise shift the robotic cell structure in a way to displace the rigid parts. Additionally, wear and tear may cause movement or displacement. The present system, in one embodiment, can identify such displacements so that the robotic cell structure may be re-calibrated when such a shift occurs.")
…
Poelman does not specifically teach generating and analyzing a 3D model of the object/environment. However, Pagnon, in the same field of endeavor of robotic control, teaches:
… run a model creation client to create a 3D model representation of the device; (Paragraph 0085, "The robotic tool system 100 includes a scanner 115, coupled to a controller (not shown), the scanner 115 configured to generated a three dimensional model of the item 105 and its surroundings, including one or more obstacles 110.")
compute locations of components from the reference point; (Paragraph 0087, "The point cloud generated by the scanner 115 and/or controller may then be pre-processed to remove noise, to stitch point cloud data from multiple reference points, or perform any other processing or filtering of the data. In one embodiment, the scanner 115 may incorporate several scanning technologies (e.g. optical and laser scanning data), to generate the point cloud.")
verify the computed locations of the components using a computer aided design (CAD) model of the device. (Paragraphs 0170-0171, "The system may align features of the scan data to the CAD model using one or more bounding boxes. The bounding box may be used to search for corresponding points in the point cloud.
The system may measure a deviation between the object and the model and refine the model according to the deviation. The refined model advantageously better matches the object than the original model.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic system and control methods as taught by Poelman with the ability to generate and analyze a 3D model of the target object and environment as taught by Pagnon. This would allow for simple and efficient analysis of the items and environment which the robot may then interact with safely in a plurality of different applications (See Pagnon, Paragraph 0083).
Regarding claim 15, where all the limitations of claim 14 are discussed above, Poelman further teaches:
15. (Original) The system of claim 14, wherein the reference point comprises a fiducial on a component. (Paragraph 0027, "In one embodiment, one or more fiducials which are observable from various vantage points in the work area, are affixed to the robotic cell structure. In one embodiment, the fiducials are distributed through the work area (or volumetric distribution), so that each camera always sees at least one fiducial affixed to the work area, in addition to any fiducials on the robotic arm and/or a calibration board or other assembled element held by the robotic arm. However, even such “rigid” parts aren't truly rigid because robotic cells are physically moved, or someone may hit the cell or otherwise shift the robotic cell structure in a way to displace the rigid parts. Additionally, wear and tear may cause movement or displacement. The present system, in one embodiment, can identify such displacements so that the robotic cell structure may be re-calibrated when such a shift occurs.")
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Poelman in view of Attar and Pagnon and in further view of Liu et al. (US 20230286145 A1), hereinafter Liu.
Regarding claim 5, where all the limitations of claim 1 are discussed above, Poelman further teaches:
5. (Currently Amended) The method of claim 1further comprising:
the product model of the device (Paragraphs 0053, “In one embodiment, CAD/Generative Design tools A60 may be used to create a CAD design for the end product to be made. In one embodiment, when using CAD/Generative Design tools A60 the system may take into account the manufacturing/assembly limitations of the robotic cells A10 in designing the end product. In one embodiment, the CAD/Generative Design tools A60 may receive data from development tools A40, and may iterate the end product design based on issues identified through the development tools A40. The output of the development tools A40 is a sequence of operations for each robotic cell.” and 0082-0083, “At block 360, a recipe is deployed and the line is run. A recipe in one embodiment is the sequence of commands sent to a robotic cell to execute one or more actions. A recipe may be static or dynamic, simple, or complex. A simple static recipe may be “move conveyor at time X by Y inches.” A complex recipe may include obtaining data from a camera, and conditionally inserting a screw, if a part is configured correctly.
A recipe may include an entire assembly process, or a step in a process. As the recipe is run, an expectation of position and movement based on the holistic representation is used to check the accuracy of the process. If any discrepancy is identified it can therefore be immediately addressed.) including one or more computed locations where assembly elements will be inserted; (Paragraph 0032, “In one embodiment, the system sets up a point of origin (0, 0, 0) frame of reference for the cell, to which all locations are referenced. In one embodiment, the point of origin is set up with respect to the most rigid part of the cell. This point of origin is the reference point for the cell. In one embodiment, it is established using a fiducial on the most rigid part of the frame, and one or more cameras attached to the most rigid part of the frame. In one embodiment, additional cameras may be mounted on other parts of the robotic cell. In one embodiment, the system acquires accurate coordinate locations for the centers of the fiducials with overview and/or internal measurement technology. In another embodiment, the system may utilize natural or man-made visual features, or other points of a fiducial. The coordinate locations for the fiducials are with respect to the point of origin frame reference, also referred to as the virtual origin frame (0,0,0), in one embodiment.” And Paragraph 0025, “Auto-calibration is possible in one embodiment using a combination of camera lens calibration, frame registration, end of arm tool contact calibration, robot pose error compensation, and calibration monitoring and using a feed forward control and constant correction to maintain calibration over time. Camera lens calibration calibrates the camera's sensor and lens distortions and errors, which is used for calibration of the other elements. Frame registration establishes the relative positions and orientations of everything within the workspace with respect to each other or to a shared coordinate system. End of arm tool contact calibration in one embodiment is referred to as tool center point (TCP) calibration, which calibrates the tool tip locations of interest, or tool lines of interest, or tool contact planes of interest that interface the tool with a product or work piece. The end of arm tool contact calibration may have a position in space as well as an orientation. The combination of all positions and orientations of interest for a component is called a pose. The contact point may be a dynamic fluid or plasma, as in soldering or welding. Robot pose error compensation provides an adjustment to compensate for the effect of the accumulated inaccuracies of manufacturing, wear, and other causes of inaccuracies in the components of the robotic arm or other component. Robot pose errors may be caused a variety of sources such as mechanical hysteresis, backlash, thermal expansion, and loading. In one embodiment, calibration monitoring provides continuous monitoring of the calibration state of the robotic cell system, and constant correction. Thus, the system calibrates the dynamic and static components in the robotic cell, and uses a learning process to model the dynamic and static components.” And 0073, “The localization process 274 localizes one or more cameras relative to other features of the cell, such as a robotic arm and/or other elements such as fiducials, fixtures, pallets, parts, feeders, trays, visual features, points of interest, etc. within the robotic cell. In one embodiment, the system uses a virtual origin frame to localize the cameras in space, relative to the robotic cell. In one embodiment, the process then creates a map, including accuracy v. precision curves. The map is continuously updated, in one embodiment. The frame registry provides positional data. In one embodiment, the system provides flexible storage and retrieval of images used to compute positional data.”)
… corresponding each of the computed locations on the device, based on the product model of the device, …
Poelman does not specifically teach validation using partial assembly or partial insertion of a component without complete insertion. However, Attar, in the same field of endeavor of robotics, teaches:
… wherein the partial assembly method comprises … (Paragraph 0425, "In the illustrated example, the position sensor system can include one or more imaging sensors 1502, 1504 located over, or around the assembly robots 204. In some example cases, images captured by imaging sensors 1502, 1504 can be used for: (i) real-time collision avoidance and safety—monitoring for obstacles in the assembly robot's motion pathway that may otherwise obstruct motion of the assembly robot. Obstacles may include other assembly robots, or otherwise any dynamic objects moving in the environment (e.g., near-by human operators); (ii) object detection and verification—capturing images to locate objects (or building parts) requiring pick-up and assembly, and further verifying that the correct object has been picked-up, or the object is picked-up in the correct orientation; (iii) object grasp verification—verifying that a picked-up object is grasped correctly by the assembly robot; and/or (iv) object quality inspection—after (or before) a building part is picked-up, it may be imaged to detect any flaws that compromise the quality of the building part. Building parts that are poor quality may not be used for assembly. For example, as shown in images 1702 and 1704 (FIG. 17), this can include detecting various cracks and fissures; (v) building part re-alignment, or otherwise robot pose correction; and/or (vi) post-assembly inspection of building structures—images of assembled, or partially assembled, building structures can be inspected to automatically identify defects in requiring modification.")
However, Liu, in the same field of endeavor of robotics, teaches:
… partially inserting the assembly element … without complete insertion, such that navigation to, and insertion of, the assembly element at an incorrect location does not damage the device or the assembly element. … (Paragraph 0035, “As described above, the system determines the state of assembly of the workpiece W1 with the part W2 by combining the first determination about the engagement state based on the change in the force F and the second determination about the appropriateness of the insertion orientation of the workpiece W1 based on the moment M. When no significant change is observed in the force F during engagement between the engagement portion 23 and the receiving portion 24 due to manufacturing variations in the workpiece W1 and the part W2 or due to tilting of the workpiece W1, the second determination can determine whether the insertion is complete. The system can thus prevent excess pressing of the workpiece W1 and avoid damaging parts. The system evaluates the appropriateness of the insertion orientation of the workpiece W1 in the second determination to prevent excess interference between the workpiece W1 and the part W2 and to avoid damaging these parts, as well as, for example, to correct the insertion orientation of the workpiece W1.” as well as Paragraphs 0057-0059, “When the moment Mx(t) is greater than or equal to the second threshold Th2, and the predetermined time Tht has elapsed, or when Mx(t)≥Th2 and t≥Tht, the controller 12 determines that the workpiece W1 is in state 7 (step S57). In state 7, the insertion orientation of the workpiece W1 is out of the allowable range and the recovery operation for correction has been unsuccessful. The controller 12 determines the assembly of the workpiece W1 as being unsuccessful and stops the robot 10 (step S58).
The system determines the state of assembly of the workpiece W1 with the part W1 by combining the first determination with the force Fz and the second determination with the moment Mx. Thus, completion (success) or a failure of the assembly may be determined with high accuracy independently of manufacturing variations in the workpiece W1 and the part W2, or tilting of the workpiece W1. When the insertion orientation of the workpiece W1 is inappropriate, correction is attempted automatically. The system thus causes the robot 10 to automatically assemble parts with the fastener structure including the engaging portion and the receiving portion.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic system and control methods as taught by Poelman with the ability to validate and verify the process using partial assembly methods as taught by Attar and further with the ability to only partially insert components while monitoring for success as taught by Liu. This would ensure that defects and instances where the system does not meet expectations may be quickly and automatically addressed while preventing the use of excess force (See Liu, Paragraph 0035).
Allowable Subject Matter
Claim 16 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The Examiner has cited particular paragraphs or columns and line numbers in the referencesapplied to the claims above for the convenience of the Applicant. Although the specified citations arerepresentative of the teachings of the art and are applied to specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested of the Applicant in preparing responses, to fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the Examiner. See MPEP 2141.02 [R-07.2015] VI. A prior art reference must be considered in its entirety, i.e., as a whole, including portions that would lead away from the claimed Invention. W.L. Gore & Associates, Inc. v. Garlock, Inc., 721 F.2d 1540, 220 USPQ 303 (Fed. Cir. 1983), cert, denied, 469 U.S. 851 (1984). See also MPEP §2123.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
/H.J.K./Examiner, Art Unit 3657
/ADAM R MOTT/Supervisory Patent Examiner, Art Unit 3657
Prosecution Insights