DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
This office action is in response to amendments filed 08/06/2025. Claims 1, 4-33 are pending.
Applicant' s arguments and amendments to Claim(s) 4 with respect to Claim objections have been fully considered and are persuasive. The objections to Claim(s) 4 have been withdrawn.
Applicant’s arguments, amendments, and approved terminal disclaimer to the claims with respect to rejections of Claims 1, 5, 9, 11, 30, 32-33 under Double Patenting have been fully considered and are persuasive. The rejections of Claims 1, 5, 9, 11, 30, 32-33 under Double Patenting have been withdrawn.
Applicant’s arguments and amendments to the claims with respect to prior art rejections of Claims 1, 4-11, 12-14, 30-31 under 35 USC 102/103 have been fully considered and are persuasive. The rejections of Claims 1, 4-11, 12-14, 30-31 under 35 USC 102/103 have been withdrawn. However, upon further consideration, a new rejection is made in view of Bronfeld et al ( US 6308144, hereinafter Bronfeld)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 4-5, 8-11, 14, 30 is/are rejected under 35 U.S.C. 103 as being unpatentable over Buehler et al (US 20130346348, hereinafter Buehler) in view of Bronfeld et al (US 6308144, hereinafter Bronfeld)
Regarding Claim 1, Buehler teaches:
A robot teaching system (see at least "FIG. 2 illustrates an exemplary control system 200 in block-diagram form. In this system 200, the robot controller and high-level computational functionality are implemented on a general-purpose computer 210, whereas several ARM processors 220 (reduced instruction set computer (RISC) processors developed by ARM Holdings, Cambridge, UK) serve as the joint-level controllers." in par. 0041 and “To facilitate the robot's autonomous performance of tasks in execution mode, the task-execution module 268 may utilize representations of tasks, and of objects to which the tasks pertain, that have been specified during training mode.” In par. 0043) , comprising:
a 3D scanning device; (see at least " Further, the robot 100 includes five cameras. It has one camera 109 in each of its two wrists so that the robot 100 can "see" objects it is about to pick up and adjust its grippers 106 accordingly. Further, it has two cameras 110, side by side, in its chest that provide a wide-angle view of the workspace and allow the robot 100 to visually detect equipment or objects to be manipulated. Finally, the robot 100 has a camera 111 above the screen 108 that is oriented outwardly to detect people in the environment. The robot 100 may also include range sensors in each wrist, and a ring of sonar sensors in its head that are used to detect moving objects in the environment; " in par. 0036 and “The robot may identify and circumscribe the object of interest using foreground/background segmentation techniques based on color, texture, edges, or a combination thereof (e.g., as implemented in various widely available, open-source tools). If the robot has depth-sensing capabilities, depth information may likewise be used.” In par. 0057 and “Sensor input 500 includes, typically, a video stream of images from one or more robot cameras, e.g., from a camera integrated into an end-effector and pointing at the object of interest, and three-dimensional positional information about the location (and/or orientation) of the camera in a fixed coordinate system associated with the robot.” In par. 0061)
a user input device; (see at least "traditional input devices 230 (e.g., a keyboard or a mouse)," in par. 0041 and “Some of the hardware components, such as the robot's wrist cuffs, navigators, and screen, are specially dedicated to user interaction, whereas others, such as cameras and appendages, have separate, independent functions (e.g., to capture images of an object or manipulate the object), but double as user-input devices (e.g., by interpreting visual information such as hand gestures or user-manipulation of the appendage as user input). Collectively, these hardware and software components are hereinafter referred to as the "user interface."” In par. 0047)
a computation device; and (see at least "a general-purpose computer 210," in par. 0041)
a storage device in communication with the computation device, the storage device storing instructions that when executed by the computation device results in a method comprising: (see at least "system memory 224, and non-volatile mass storage devices (such as one or more hard disks and/or optical storage units) 226. " in par. 0041)
causing the 3D scanning device to capture a first 3D scan of a workpiece; (see at least "In a first step 600, the robot identifies the object of interest based on user input; this step may be accomplished in the same or a similar manner as described above with FIG. 4. The user may begin the training process by indicating the object of interest, for example, by moving the robot's arm to position the camera in the gripper above and viewing the object. Once the user releases the arm, the robot may search the camera image for a group of pixels that most likely corresponds to the object of interest, overlay an object outline on the image, move its camera to optimal viewing height, and move its arm into a position and orientation that are horizontally centered above and aligned with the object." in par. 0076)
recording a first spatial point in response to user input via the user input device; (see at least “For example, to initially indicate the object of interest, the user may bring the robot's end-effector into physical contact with the object. The robot may then move away from the object to get a better view, and use the visual methods described above in conjunction with knowledge of the approximate location of the object to select the object in the view. " in par. 0059)
determining at least one best-fit geometry of at least a portion of the first 3D scan; (see at least "If the parameters thus determined are within specified ranges around the parameters of the representation, a match is declared. (Typically, the match is determined based on shape only--not considering size--as the size of an object within the image varies with the distance from the camera.) The edges may then be analyzed further to determine the position, orientation, and size of the ellipse." in par. 0063)
defining a robot instruction based on the first geometric point; and (see at least "Following the user's approval of the selected visual model, the robot may overlay virtual grippers on the image (or video stream) of the object to indicate where the grasp point would be, given the current arm location and orientation (step 606). " in par. 0078 and “For example, knowing the shape of an object may help the robot choose an appropriate grasping strategy. If the representation describes, for instance, an object with a hollow (e.g., cylindrical) body and a handle (e.g., akin to a mug), the robot may attempt to pick up the object by its handle.” In par. 0080)
using a machine learning model to predict a quality resulting from the robot instruction. (see at least "Following the user's approval of the selected visual model, the robot may overlay virtual grippers on the image (or video stream) of the object to indicate where the grasp point would be, given the current arm location and orientation (step 606). Thus, without the need to actually lower the gripper and bring it in contact with the object (which could be a cumbersome and, in some instance, even dangerous procedure), the robot can give the user a sense of the likelihood of a successful pick-up. If the virtual gripper is not positioned around the proper part of the object (as determined in step 608), the user may adjust the arm to roughly establish the proper grasp point (step 610). The robot may then fine-tune the arm location to optimize the gripper location relative to the rough grasp point on the object (step 612). This process of coarse and fine adjustments of the gripper by the human trainer and the robot, respectively, may be iterated until the user is satisfied with the grasp point and gripper orientation shown in the visual display." in par. 0078 and “Container objects, for example, may have associated routines for opening or closing the containers, which vary between classes of containers. Thus, the robot's end-effector would undergo different motions to open a cylindrical container with a lid than it would to open a cardboard box. The robot may learn these special routines during training, and associate them with particular visual representations. Alternatively, the routines may be programmed into the robot at the outset and altered, as necessary, during training as noted above.” In par. 0080)
Buehler does not appear to explicitly teach all of the following, but Bronfeld does teach:
projecting the first spatial point to a first geometric point having a predefined relationship to an individual one of the best-fit geometries (see at least “In many cases, the user will wish to align sketch geometry with points of interest on the model, such as edges, corners, or faces of a primitive (e.g., the box discussed above) of the model …Additionally, an embodiment of the invention can "snap" an element of the sketched object to a projection to the sketcher plane of a point of interest within the model geometry onto the sketcher plane, i.e., can align the element precisely with the point of interest when the element is located within a specified range about the point of interest.” In col. 6 lines 25-41 and " One type of an association relationship supported by the present invention is that a reference point of the sketched object in the sketcher plane can be positioned at a projection from the point of interest on the 3D model, with the projection being defined by a vector that is perpendicular to the sketcher plane, wherein the vector passes through the point of interest on the 3D model and also through the reference point on the sketched object." In col. 6 lines 53-59);
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system taught by Buehler to incorporate the teachings of Bronfeld wherein the user interface input of a location is snapped to a point of interest on the 3D model by projecting the point onto the 3D model, in order to arrive at snapping the user input to points of interest on the object representation on the user interface taught by Buehler. The motivation to incorporate the teachings of Bronfeld would be to increase the precision with which user input can be collected while reducing the amount of manual alignment needed (see col. 6 lines 25-41)
Regarding Claim 4, Buehler as modified by Bronfeld teaches:
The system of claim 1 (see Claim 1 analysis).
Buehler further teaches: wherein the user input device is a graphical user interface (see at least "Further, the robot includes a user interface having a robot-controlled visual display for displaying a camera view of the object and, overlaid thereon, robot-generated graphics indicative of the identification, and one or more input devices for receiving feedback from the trainer indicating whether the identification is correct." in par. 0021).
Regarding Claim 5, Buehler as modified by Bronfeld teaches:
The system of claim 1,
Buehler further teaches: wherein the user input device is a process agnostic pointing device (see at least "The user may also indicate the object of interest simply by pointing at it with his finger or by shining a laser onto the object. " in par. 0057) .
Regarding Claim 8, Buehler as modified by Bronfeld teaches:
The system of claim 1,
Buehler further teaches: wherein the method executed by the computation device further comprises collecting 3D information about the workpiece, in addition to the first 3D scan captured by the 3D scanning device, by bringing a portion of a robot tool in contact with the workpiece. (see at least "For example, the robot appendage may be brought into physical contact with the object of interest so as to provide the mechanical input" in par. 0017 and “For example, to initially indicate the object of interest, the user may bring the robot's end-effector into physical contact with the object. The robot may then move away from the object to get a better view, and use the visual methods described above in conjunction with knowledge of the approximate location of the object to select the object in the view.” In par. 0059 and “Sensor input 500 includes, typically, a video stream of images from one or more robot cameras, e.g., from a camera integrated into an end-effector and pointing at the object of interest, and three-dimensional positional information about the location (and/or orientation) of the camera in a fixed coordinate system associated with the robot.” In par. 0061) .
Regarding Claim 9, Buehler teaches:
A robot teaching system (see at least "FIG. 2 illustrates an exemplary control system 200 in block-diagram form. In this system 200, the robot controller and high-level computational functionality are implemented on a general-purpose computer 210, whereas several ARM processors 220 (reduced instruction set computer (RISC) processors developed by ARM Holdings, Cambridge, UK) serve as the joint-level controllers." in par. 0041 and “To facilitate the robot's autonomous performance of tasks in execution mode, the task-execution module 268 may utilize representations of tasks, and of objects to which the tasks pertain, that have been specified during training mode.” In par. 0043), comprising:
a 3D scanning device (see at least " Further, the robot 100 includes five cameras. It has one camera 109 in each of its two wrists so that the robot 100 can "see" objects it is about to pick up and adjust its grippers 106 accordingly. Further, it has two cameras 110, side by side, in its chest that provide a wide-angle view of the workspace and allow the robot 100 to visually detect equipment or objects to be manipulated. Finally, the robot 100 has a camera 111 above the screen 108 that is oriented outwardly to detect people in the environment. The robot 100 may also include range sensors in each wrist, and a ring of sonar sensors in its head that are used to detect moving objects in the environment; " in par. 0036 and “The robot may identify and circumscribe the object of interest using foreground/background segmentation techniques based on color, texture, edges, or a combination thereof (e.g., as implemented in various widely available, open-source tools). If the robot has depth-sensing capabilities, depth information may likewise be used.” In par. 0057 and “Sensor input 500 includes, typically, a video stream of images from one or more robot cameras, e.g., from a camera integrated into an end-effector and pointing at the object of interest, and three-dimensional positional information about the location (and/or orientation) of the camera in a fixed coordinate system associated with the robot.” In par. 0061);
a user input device (see at least "traditional input devices 230 (e.g., a keyboard or a mouse)," in par. 0041 and “Some of the hardware components, such as the robot's wrist cuffs, navigators, and screen, are specially dedicated to user interaction, whereas others, such as cameras and appendages, have separate, independent functions (e.g., to capture images of an object or manipulate the object), but double as user-input devices (e.g., by interpreting visual information such as hand gestures or user-manipulation of the appendage as user input). Collectively, these hardware and software components are hereinafter referred to as the "user interface."” In par. 0047);
a computation device (see at least "a general-purpose computer 210," in par. 0041); and
a storage device in communication with the computation device, the storage device storing instructions that when executed by the computation device results in a method comprising: (see at least "system memory 224, and non-volatile mass storage devices (such as one or more hard disks and/or optical storage units) 226. " in par. 0041)
causing the 3D scanning device to capture a 3D scan of a workpiece (see at least "In a first step 600, the robot identifies the object of interest based on user input; this step may be accomplished in the same or a similar manner as described above with FIG. 4. The user may begin the training process by indicating the object of interest, for example, by moving the robot's arm to position the camera in the gripper above and viewing the object. Once the user releases the arm, the robot may search the camera image for a group of pixels that most likely corresponds to the object of interest, overlay an object outline on the image, move its camera to optimal viewing height, and move its arm into a position and orientation that are horizontally centered above and aligned with the object." in par. 0076);
determining one or more best-fit geometries of at least a portion of the 3D scan (see at least "If the parameters thus determined are within specified ranges around the parameters of the representation, a match is declared. (Typically, the match is determined based on shape only--not considering size--as the size of an object within the image varies with the distance from the camera.) The edges may then be analyzed further to determine the position, orientation, and size of the ellipse." in par. 0063);
selecting one or more robot parameters in response to user input via the user input device (see at least "Following the user's approval of the selected visual model, the robot may overlay virtual grippers on the image (or video stream) of the object to indicate where the grasp point would be, given the current arm location and orientation (step 606). " in par. 0078 and “For example, knowing the shape of an object may help the robot choose an appropriate grasping strategy. If the representation describes, for instance, an object with a hollow (e.g., cylindrical) body and a handle (e.g., akin to a mug), the robot may attempt to pick up the object by its handle.” In par. 0080) ;
defining a robot instruction based on the one or more best-fit geometries, the geometric point, and the one or more selected robot parameters (see at least "Following the user's approval of the selected visual model, the robot may overlay virtual grippers on the image (or video stream) of the object to indicate where the grasp point would be, given the current arm location and orientation (step 606). " in par. 0078 and “For example, knowing the shape of an object may help the robot choose an appropriate grasping strategy. If the representation describes, for instance, an object with a hollow (e.g., cylindrical) body and a handle (e.g., akin to a mug), the robot may attempt to pick up the object by its handle.” In par. 0080); and
recording the results of the robot instruction to train a machine learning model (see at least "In some embodiments, the visual representation of an object informs the robot beyond the purpose of detecting and localizing that type of object in images. For example, knowing the shape of an object may help the robot choose an appropriate grasping strategy. If the representation describes, for instance, an object with a hollow (e.g., cylindrical) body and a handle (e.g., akin to a mug), the robot may attempt to pick up the object by its handle. More generally, representations of objects, or classes of objects, may have specific routines for picking up or otherwise manipulating the object associated with them. Container objects, for example, may have associated routines for opening or closing the containers, which vary between classes of containers. Thus, the robot's end-effector would undergo different motions to open a cylindrical container with a lid than it would to open a cardboard box. The robot may learn these special routines during training, and associate them with particular visual representations." in par. 0080).
Buehler does not appear to explicitly teach all of the following, but Bronfeld does teach:
projecting a spatial point, recorded in response to user input via the user input device, to a geometric point having a predefined relationship to an individual one of the best-fit geometries (see at least “In many cases, the user will wish to align sketch geometry with points of interest on the model, such as edges, corners, or faces of a primitive (e.g., the box discussed above) of the model …Additionally, an embodiment of the invention can "snap" an element of the sketched object to a projection to the sketcher plane of a point of interest within the model geometry onto the sketcher plane, i.e., can align the element precisely with the point of interest when the element is located within a specified range about the point of interest.” In col. 6 lines 25-41 and " One type of an association relationship supported by the present invention is that a reference point of the sketched object in the sketcher plane can be positioned at a projection from the point of interest on the 3D model, with the projection being defined by a vector that is perpendicular to the sketcher plane, wherein the vector passes through the point of interest on the 3D model and also through the reference point on the sketched object." In col. 6 lines 53-59);
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system taught by Buehler to incorporate the teachings of Bronfeld wherein the user interface input of a location is snapped to a point of interest on the 3D model by projecting the point onto the 3D model, in order to arrive at snapping the user input to points of interest on the object representation on the user interface taught by Buehler. The motivation to incorporate the teachings of Bronfeld would be to increase the precision with which user input can be collected while reducing the amount of manual alignment needed (see col. 6 lines 25-41)
Regarding Claim 10, Buehler as modified by Bronfeld teaches:
the system of claim 9,
Buehler further teaches: wherein the user input device is a graphical user interface (see at least "Further, the robot includes a user interface having a robot-controlled visual display for displaying a camera view of the object and, overlaid thereon, robot-generated graphics indicative of the identification, and one or more input devices for receiving feedback from the trainer indicating whether the identification is correct." in par. 0021).
Regarding Claim 11, Buehler as modified by Bronfeld teaches:
The system of Claim 9,
Buehler further teaches: wherein the user input device is a process agnostic pointing device (see at least " The user may also indicate the object of interest simply by pointing at it with his finger or by shining a laser onto the object. " in par. 0057)
Regarding Claim 14, Buehler as modified by Bronfeld teaches:
the system of claim 9,
Buehler further teaches: wherein the method executed by the computation device further comprises collecting 3D information about the workpiece, in addition to the first 3D scan captured by the 3D scanning device, by bringing a portion of a robot tool in contact with the workpiece. (see at least "For example, the robot appendage may be brought into physical contact with the object of interest so as to provide the mechanical input" in par. 0017 and “For example, to initially indicate the object of interest, the user may bring the robot's end-effector into physical contact with the object. The robot may then move away from the object to get a better view, and use the visual methods described above in conjunction with knowledge of the approximate location of the object to select the object in the view.” In par. 0059 and “Sensor input 500 includes, typically, a video stream of images from one or more robot cameras, e.g., from a camera integrated into an end-effector and pointing at the object of interest, and three-dimensional positional information about the location (and/or orientation) of the camera in a fixed coordinate system associated with the robot.” In par. 0061) .
Regarding Claim 30, Buehler teaches:
A robot teaching system (see at least "FIG. 2 illustrates an exemplary control system 200 in block-diagram form. In this system 200, the robot controller and high-level computational functionality are implemented on a general-purpose computer 210, whereas several ARM processors 220 (reduced instruction set computer (RISC) processors developed by ARM Holdings, Cambridge, UK) serve as the joint-level controllers." in par. 0041 and “To facilitate the robot's autonomous performance of tasks in execution mode, the task-execution module 268 may utilize representations of tasks, and of objects to which the tasks pertain, that have been specified during training mode.” In par. 0043), comprising:
at least one 3D scanning device (see at least " Further, the robot 100 includes five cameras. It has one camera 109 in each of its two wrists so that the robot 100 can "see" objects it is about to pick up and adjust its grippers 106 accordingly. Further, it has two cameras 110, side by side, in its chest that provide a wide-angle view of the workspace and allow the robot 100 to visually detect equipment or objects to be manipulated. Finally, the robot 100 has a camera 111 above the screen 108 that is oriented outwardly to detect people in the environment. The robot 100 may also include range sensors in each wrist, and a ring of sonar sensors in its head that are used to detect moving objects in the environment; " in par. 0036 and “The robot may identify and circumscribe the object of interest using foreground/background segmentation techniques based on color, texture, edges, or a combination thereof (e.g., as implemented in various widely available, open-source tools). If the robot has depth-sensing capabilities, depth information may likewise be used.” In par. 0057 and “Sensor input 500 includes, typically, a video stream of images from one or more robot cameras, e.g., from a camera integrated into an end-effector and pointing at the object of interest, and three-dimensional positional information about the location (and/or orientation) of the camera in a fixed coordinate system associated with the robot.” In par. 0061) ;
a user input device (see at least "traditional input devices 230 (e.g., a keyboard or a mouse)," in par. 0041 and “Some of the hardware components, such as the robot's wrist cuffs, navigators, and screen, are specially dedicated to user interaction, whereas others, such as cameras and appendages, have separate, independent functions (e.g., to capture images of an object or manipulate the object), but double as user-input devices (e.g., by interpreting visual information such as hand gestures or user-manipulation of the appendage as user input). Collectively, these hardware and software components are hereinafter referred to as the "user interface."” In par. 0047);
a computation device (see at least "a general-purpose computer 210," in par. 0041); and
a storage device in communication with the computation device, the storage device storing instructions that when executed by the computation device results in a method comprising: (see at least "system memory 224, and non-volatile mass storage devices (such as one or more hard disks and/or optical storage units) 226. " in par. 0041)
recording, prior to capturing 3D information about a workpiece, at least one spatial point in response to user input via the user input device (see at least “For example, to initially indicate the object of interest, the user may bring the robot's end-effector into physical contact with the object. The robot may then move away from the object to get a better view, and use the visual methods described above in conjunction with knowledge of the approximate location of the object to select the object in the view. " in par. 0059) ;
causing the at least one 3D scanning device to capture 3D information of at least a portion of the workpiece in proximity to the at least one spatial point (see at least “For example, to initially indicate the object of interest, the user may bring the robot's end-effector into physical contact with the object. The robot may then move away from the object to get a better view, and use the visual methods described above in conjunction with knowledge of the approximate location of the object to select the object in the view. " in par. 0059) ;
determining at least one best-fit geometry of at least a portion of the 3D information of at least a portion of a workpiece scan (see at least "If the parameters thus determined are within specified ranges around the parameters of the representation, a match is declared. (Typically, the match is determined based on shape only--not considering size--as the size of an object within the image varies with the distance from the camera.) The edges may then be analyzed further to determine the position, orientation, and size of the ellipse." in par. 0063);
defining a robot instruction based on the at least one geometric point (see at least "Following the user's approval of the selected visual model, the robot may overlay virtual grippers on the image (or video stream) of the object to indicate where the grasp point would be, given the current arm location and orientation (step 606). " in par. 0078 and “For example, knowing the shape of an object may help the robot choose an appropriate grasping strategy. If the representation describes, for instance, an object with a hollow (e.g., cylindrical) body and a handle (e.g., akin to a mug), the robot may attempt to pick up the object by its handle.” In par. 0080).
Buehler does not appear to explicitly teach all of the following, but Bronfeld does teach:
projecting the at least one spatial point to at least one geometric point having a predefined relationship to an individual one of the best-fit geometries (see at least “In many cases, the user will wish to align sketch geometry with points of interest on the model, such as edges, corners, or faces of a primitive (e.g., the box discussed above) of the model …Additionally, an embodiment of the invention can "snap" an element of the sketched object to a projection to the sketcher plane of a point of interest within the model geometry onto the sketcher plane, i.e., can align the element precisely with the point of interest when the element is located within a specified range about the point of interest.” In col. 6 lines 25-41 and " One type of an association relationship supported by the present invention is that a reference point of the sketched object in the sketcher plane can be positioned at a projection from the point of interest on the 3D model, with the projection being defined by a vector that is perpendicular to the sketcher plane, wherein the vector passes through the point of interest on the 3D model and also through the reference point on the sketched object." In col. 6 lines 53-59);
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system taught by Buehler to incorporate the teachings of Bronfeld wherein the user interface input of a location is snapped to a point of interest on the 3D model by projecting the point onto the 3D model, in order to arrive at snapping the user input to points of interest on the object representation on the user interface taught by Buehler. The motivation to incorporate the teachings of Bronfeld would be to increase the precision with which user input can be collected while reducing the amount of manual alignment needed (see col. 6 lines 25-41)
Claim(s) 6-7, 12-13, 31 is/are rejected under 35 U.S.C. 103 as being unpatentable over Buehler et al (US 20130346348, hereinafter Buehler) in view of Bronfeld et al ( US 6308144, hereinafter Bronfeld) and Cardenas Bernal (US 20180350056, hereinafter Cardenas Bernal).
Regarding Claim 6, Buehler as modified by Bronfeld teaches the system of claim 1 ,
Buehler and Bronfeld do not appear to explicitly teach all of the following, but Cardenas Bernal does teach:
wherein a scan is made after the robot instructions are executed and a comparison between work intended to be performed and actual work performed will be shown to the user (see at least " In some embodiments, the process is used to inspect the accuracy of assembly including determining whether joints are assembled within tolerances and for performing dimensional quality inspection" in par. 0020 and “By comparing an actual spot weld to the overlaid user interface component representing the reference location of the spot weld, the user of the device can visually inspect the quality of a spot weld.” In par. 0024 and “As yet another example, a user can interact with the user interface to inspect a part or assembly. For example, certain mechanical joints may be selected via the user interface and marked as non-acceptable if they are not within the acceptable tolerances. The marked features may also be exported and used to re-calibrate robots used to perform the operation by adjusting for any identified deviations.” In par. 0027) .
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system taught by Buehler as modified by Bronfeld to incorporate the teachings of Cardenas Bernal wherein the user inspects a scan of mechanical joints made by a robot with a model of the intended work overlaid and marks whether or not they are acceptable. The motivation to incorporate the teachings of Cardenas Bernal would be to improve the calibration of the robot performing work on the workpiece (see par. 0080)
Regarding Claim 7, Buehler as modified by Bronfeld and Cardenas Bernal teaches the system of claim 6,
Buehler and Bronfeld do not appear to explicitly teach all of the following, but Cardenas Bernal does teach:
wherein the comparison shown to the user has visually recognizable attributes defined by thresholds which determine if a difference is of sufficient size to be shown by the visually recognizable attributes (see at least " In the example shown, user interface components 911 and 913 depict locations on object of interest 901 where the joint is correctly marked. In some embodiments, the user interface component depicts a correctly marked joint when the user interface component overlaps the entirety of the marked joint location. In some embodiments, the user interface component depicts a correctly marked joint when the user interface component overlaps the center of the marked joint location. User interface components 911 and 913 include representations of a tolerance measurement for each joint. For example, in some embodiments, the size of the user interface component represents an allowable deviation from the center of the joint. In some embodiments, user interface components 911 and 913 represent correctly marked joints and are displayed as circular shapes where the volume of the circular shapes represents the allowable deviation before the marked joint is incorrect. In various embodiments, the circular shapes are rendered as spherical visual indicators. In some embodiments, the radius of circular shapes represents an allowable deviance from a reference property. In some embodiments, user interface components 911 and 913 represent correctly marked joints and are displayed as circles where the area of the circle represents the allowable deviation before the marked joint is incorrect." in par. 0081) .
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system taught by Buehler as modified by Bronfeld to incorporate the teachings of Cardenas Bernal wherein a visual indication of acceptable tolerances is overlaid on the workpiece being inspected. The motivation to incorporate the teachings of Cardenas Bernal would be to improve the calibration of the robot performing work on the workpiece (see par. 0080).
Regarding Claim 12, Buehler as modified by Bronfeld teaches:
The system of claim 9,
Buehler and Bronfeld do not appear to explicitly teach all of the following, but Cardenas Bernal does teach:
wherein a scan is made after the robot instructions are executed and a comparison between work intended to be performed and actual work performed will be shown to the user (see at least " In some embodiments, the process is used to inspect the accuracy of assembly including determining whether joints are assembled within tolerances and for performing dimensional quality inspection" in par. 0020 and “By comparing an actual spot weld to the overlaid user interface component representing the reference location of the spot weld, the user of the device can visually inspect the quality of a spot weld.” In par. 0024 and “As yet another example, a user can interact with the user interface to inspect a part or assembly. For example, certain mechanical joints may be selected via the user interface and marked as non-acceptable if they are not within the acceptable tolerances. The marked features may also be exported and used to re-calibrate robots used to perform the operation by adjusting for any identified deviations.” In par. 0027) .
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system taught by Buehler as modified by Bronfeld to incorporate the teachings of Cardenas Bernal wherein the user inspects a scan of mechanical joints made by a robot with a model of the intended work overlaid and marks whether or not they are acceptable. The motivation to incorporate the teachings of Cardenas Bernal would be to improve the calibration of the robot performing work on the workpiece (see par. 0080)
Regarding Claim 13, Buehler as modified by Bronfeld and Cardenas Bernal teaches the system of Claim 12,
Buehler and Bronfeld does not appear to explicitly teach all of the following, but Cardenas Bernal does teach:
wherein the comparison shown to the user has visually recognizable attributes defined by thresholds which determine if a difference is of sufficient size to be shown by the visually recognizable attributes (see at least " In the example shown, user interface components 911 and 913 depict locations on object of interest 901 where the joint is correctly marked. In some embodiments, the user interface component depicts a correctly marked joint when the user interface component overlaps the entirety of the marked joint location. In some embodiments, the user interface component depicts a correctly marked joint when the user interface component overlaps the center of the marked joint location. User interface components 911 and 913 include representations of a tolerance measurement for each joint. For example, in some embodiments, the size of the user interface component represents an allowable deviation from the center of the joint. In some embodiments, user interface components 911 and 913 represent correctly marked joints and are displayed as circular shapes where the volume of the circular shapes represents the allowable deviation before the marked joint is incorrect. In various embodiments, the circular shapes are rendered as spherical visual indicators. In some embodiments, the radius of circular shapes represents an allowable deviance from a reference property. In some embodiments, user interface components 911 and 913 represent correctly marked joints and are displayed as circles where the area of the circle represents the allowable deviation before the marked joint is incorrect." in par. 0081) .
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system taught by Buehler as modified by Bronfeld to incorporate the teachings of Cardenas Bernal wherein a visual indication of acceptable tolerances is overlaid on the workpiece being inspected. The motivation to incorporate the teachings of Cardenas Bernal would be to improve the calibration of the robot performing work on the workpiece (see par. 0080)
Regarding Claim 31, Buehler as modified by Bronfeld teaches:
the system of claim 30,
Buehler and Bronfeld does not appear to explicitly teach all of the following, but Cardenas Bernal does teach:
wherein a second 3D scanning device captures second 3D information of at least a portion of the workpiece and the 3D information is updated with the second 3D information to create merged 3D information about at least a portion of a workpiece (see at least " In some embodiments, a surface point is determined based on the features of the object of interest. In various embodiments, the same surface point is analyzed from different perspectives such as from two different cameras or via two different images once the camera has moved. In some embodiments, features are matched across two corresponding images and 3D coordinates of the surface points are determined. In some embodiments, the 3D coordinates are determined by triangulating corresponding surface points of different matched images. In various embodiments, multiple readings of the same point are utilized." in par. 0041).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system taught by Buehler as modified by Bronfeld to incorporate the teachings of Cardenas Bernal wherein the machine vision system merges 3D data from multiple cameras to determine surface points of a detected object. The motivation to incorporate the teachings of Cardenas Bernal would be to increase the chances of correctly modeling the object (see par. 0044)
Allowable Subject Matter
Claims 15-29, 32-33 allowed.
REASONS FOR ALLOWANCE
The following is an examiner’s statement of reasons for allowance: The closest prior art comes from Buehler, Bronfeld, and Cardenas Bernal for all claims.
For claim 15 and dependents, the prior art does not appear to teach the combination of the following limitations: “display to the user the first geometric point in real time with the user input; causing the second 3D scanning device to capture additional geometric information of a region of the workpiece to which the first spatial point had been proximal; updating the first set of 3D information with the additional geometric information to create merged 3D information; projecting the first spatial point to a second geometric point having a predefined relationship to an individual one of the best-fit geometries of the merged 3D information; and defining a robot instruction based on the second geometric point.” in addition to all of the other limitations in the independent claims.
For claims 32 and 33, the prior art does not appear to teach “creating a 3D representation of the workpiece from the two or more best-fit geometries, the two or more best- fit geometries including a plane or cylinder;” in combination with the external axis device and all of the other limitations in the independent claims.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DYLAN M KATZ whose telephone number is (571)272-2776. The examiner can normally be reached Mon-Thurs. 8:00-6:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abby Lin can be reached on (571) 270-3976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DYLAN M KATZ/Examiner, Art Unit 3657