DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 05/23/2025 was filed. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Specification
The disclosure is objected to because of the following informalities:
In Paragraph 61, Line 1, “workers are available” should read as “worker is available”
In Paragraph 98, Line 8, “can consolidated” should read “can be consolidated” or “can consolidate”
In Paragraph 121, Line 1, “an selection” should read “a selection”
Appropriate correction is required.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference characters not mentioned in the description: 804, 1002, and 1009. Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference character “303” has been used to designate both “workspace” and “optimization procedure” in Fig. 1B, and reference character “302” has been used to designate both “cameras” in Fig. 3 and “annotation tool” in Fig. 4. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The drawings are objected to because “task optimization procedure 115” and “annotator 113” are mislabeled as 105 and 103 respectively in Fig. 1B. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-9, 11, 13-15, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Lonsberry (US20220266453A1) in view of Bell (US10529143B2).
Re Claim 1, Lonsberry discloses a method for specifying a robot task, the method comprising: (Paragraph 0071, “FIG. 9 is a flow diagram of a method 900 for performing autonomous welds, in accordance with various examples. More specifically, FIG. 9 is a flowchart of a method 900 for operating and controlling welding robots (e.g., robot 110 in FIG. 1), according to some examples.”)
capturing, via one or more cameras, one or more images of a robot workspace, where the one or more cameras are mounted in an environment of the robot workspace; (Paragraph 0030, “FIG. 3 is a schematic diagram of an autonomous robotic welding system 300, in accordance with various examples. The system 300 is an example of the system 100 of FIG. 1 and the system 200 of FIG. 2, with like numerals referring to like components… In operation, the sensors 302 collect 2D images of the workspace 301 and provide the 2D images to a controller (not expressly shown in FIG. 3).”; Paragraph 0021, “The sensors 102 are configured to capture information about the workspace 101. In examples, the sensors 102 are image sensors that are configured to capture visual information (e.g., two-dimensional (2D) images) about the workspace 101. For instance, the sensors 102 may include cameras (e.g., cameras with built-in laser)…”)
displaying a visual representation of the robot workspace to a user based on the one or more captured images; (Paragraph 0030, “The system 300 further includes a UI 306 coupled to the workspace 301. In operation, the sensors 302 collect 2D images of the workspace 301 and provide the 2D images to a controller (not expressly shown in FIG. 3). The controller generates 3D representations (e.g., point clouds) of the workspace 301, such as the fixtures 316, a part supported by the fixtures 316, and/or other structures within the workspace 301.”; Paragraph 0051, “In examples, the user interface 106 can provide the user with an option to view candidate seams. For example, the user interface 106 may provide a graphical representation of a part 114 and/or candidate seams on a part 114.”)
receiving, from the user, one or more annotations associated with the visual representation… (Paragraph 0054, “The user can annotate seams that are to be welded in the representation via the user interface 106.”)
determining a set of waypoints based on the one or more annotations via an optimization process; (Paragraph 40, “Accordingly, the controller 108 compares a first seam (e.g., a candidate seam on a part 114 that has been verified as an actual seam) to a second seam (e.g., a seam annotated (e.g., by an operator/user) on the CAD model corresponding to the first seam) to determine differences between the first and second seams. Seams on the CAD model may be annotated as described above. The first seam and the second seam can be in nearly the same location, in instances in which the CAD model and/or controller 108 accurately predicts the location of the candidate seam. Alternatively, the first seam and the second seam can partially overlap, in instances in which the CAD model and/or controller 108 is partially accurate. The controller 108 may perform a comparison of the first seam and the second seam. This comparison of first seam and the second seam can be based in part on shape and relative location in space of both the seams. Should the first seam and the second seam be relatively similar in shape and be proximal to each other, the second seam can be identified as being the same as the first seam. In this way, the controller 108 can account for the topography of the surfaces on the part that are not accurately represented in the CAD models. In this manner, the controller 108 can identify candidate seams and can sub-select or refine or update candidate seams relative to the part using a CAD model of the part. Each candidate seam can be a set of updated points that represents the position and orientation of the candidate seam relative to the part.”)
and obtaining the robot task based on the set of waypoints and the one or more annotations. (Paragraph 76, “In some examples, the welding instructions can be based on the type of seam (e.g., butt joint, corner joint, edge joint, lap joint, tee joint, and/or the like). In some examples, the welding instructions can be updated based on input from a user via a user interface (e.g., user interface 106 in FIG. 1). The user can select a candidate seam to be welded from all the available candidate seams via the user interface. Path planning can be performed for the selected candidate seam and welding instructions can be generated for the selected candidate seam.”) but does not explicitly disclose …wherein the one or more annotations include at least one of graphical annotations and natural language annotations;
However, Bell teaches a system/method for generating a three-dimensional composite scene of an environment using a robotic arm, and discloses …wherein the one or more annotations include at least one of graphical annotations and natural language annotations; (Paragraph 113, “A reference dataset 401 containing auxiliary 3D data that is spatially aligned 403 (using the 3D alignment techniques described herein or other techniques) to known captured 3D data of the object or environment being viewed 402 may be used as a source of information to display in an overlay to, in combination with, or in replacement of, the current captured scene 402 as seen by the outward-facing camera or 3D capture device 304. Types of auxiliary datasets include, but are not limited to: Marks or labels identifying captured objects or locations; these marks may be manually or automatically (via image or object recognition algorithms) made during a current or prior capture of the environment or via a user of an external tool manipulating the captured 3D data, for example using the annotation interface described herein.”; Examiner is treating the marks as graphical annotations and the labels as natural language annotations)
Thus, it would be obvious to a person of ordinary skill in the art at the time of the effective filing of the application to modify Lonsberry’s autonomous welding robot method with Bell’s manual auxiliary dataset because it would allow the user to provide the robot with both structural and contextual feedback regarding its welding operations.
Re Claim 2, Lonsberry discloses …of one or more cameras mounted on a robot (Paragraph 20, “In some examples, the robot 110 may include one or more sensors 102. For instance, one or more sensors 102 may be positioned on an arm (e.g., on a weld head attached to the arm) of the robot 110.”; Paragraph 21, “The sensors 102 are configured to capture information about the workspace 101. In examples, the sensors 102 are image sensors that are configured to capture visual information (e.g., two-dimensional (2D) images) about the workspace 101. For instance, the sensors 102 may include cameras (e.g., cameras with built-in laser)…”) but does not explicitly disclose wherein the visual representation comprises one or more live camera feeds…
However, Bell teaches a system/method for generating a three-dimensional composite scene of an environment using a robotic arm, and discloses wherein the visual representation comprises one or more live camera feeds… (Paragraph 41, “A “live” view 203 that shows distance and/or color data as may be currently seen by a 3D capture device may be implemented in the course of the present invention. Such an implementation may show a live video feed from a color camera that is part of the 3D capture device.”; Paragraph 55, “The 3D capture hardware may be attached or coupled (either permanently or detachably) to any one of a variety of types of robots or other mechanized implementation rather than be manipulated by a human user. Possible implementations include, but are by no means limited to: The 3D capture hardware is placed at the tip of a robotic arm on a fixed platform.”; Examiner is treating the 3D capture device as the camera disclosed in Lonsberry)
Thus, it would be obvious to a person of ordinary skill in the art at the time of the effective filing of the application to modify Lonsberry’s autonomous welding robot method with Bell’s live video feed because it would provide the robot’s user with a dynamic view of the welding operation.
Re Claim 3, Lonsberry discloses wherein the visual representation comprises a 3D representation based on the captured images. (Paragraph 0030, “The system 300 further includes a UI 306 coupled to the workspace 301. In operation, the sensors 302 collect 2D images of the workspace 301 and provide the 2D images to a controller (not expressly shown in FIG. 3). The controller generates 3D representations (e.g., point clouds) of the workspace 301, such as the fixtures 316, a part supported by the fixtures 316, and/or other structures within the workspace 301.”; Paragraph 0051, “In examples, the user interface 106 can provide the user with an option to view candidate seams. For example, the user interface 106 may provide a graphical representation of a part 114 and/or candidate seams on a part 114.”)
Re Claim 4, Lonsberry discloses wherein the set of waypoints comprises a sequence of locations in the robot workspace that can be seen by the one or more cameras. (Paragraph 21, “The sensors 102 are configured to capture information about the workspace 101. In examples, the sensors 102 are image sensors that are configured to capture visual information (e.g., two-dimensional (2D) images) about the workspace 101. For instance, the sensors 102 may include cameras (e.g., cameras with built-in laser)…”; Paragraph 0022, “To generate 3D representations of the workspace 101, the sensors 102 capture 2D images of physical structures in the workspace 101 from a variety of angles. For example, although a single 2D image of a fixture 116 or a part 114 may be inadequate to generate a 3D representation of that component, and, similarly, a set of multiple 2D images of the fixture 116 or the part 114 from a single angle, view, or plane may be inadequate to generate a 3D representation of that component, multiple 2D images captured from multiple angles in a variety of positions within the workspace 101 may be adequate to generate a 3D representation of a component, such as a fixture 116 or part 114.”; Paragraph 0038, “To summarize, and without limitation, using the techniques described above, the controller 108 receives image data captured by the sensors 102 from various locations and vantage points within the workspace 101. The controller 108 performs a pixel-wise and/or point-wise classification technique using a neural network to classify and identify each pixel and/or point as a part 114, a candidate seam on a part 114 or at an interface between multiple parts 114, a fixture 116, etc. Structures identified as being non-part 114 structures and non-candidate seam structures are segmented out, and the controller 108 may perform additional processing on the remaining points (e.g., to mitigate noise). By performing these actions, the controller 108 may produce a set of candidate seams on parts 114 that indicate locations and orientations of those seams.”; Examiner states that since the sensors 102 capture images of the parts 114 from multiple angles in the workspace 101 and that the previously mentioned set of updated points are part of the candidate seams, the sensors 102 would obviously capture images of the set of updated points from multiple angles since their candidates seams are stated to be on parts 114 because it would provide the user with visual feedback of the seams in the visual representation)
Re Claim 5, Lonsberry discloses wherein the set of waypoints comprises a sequence of locations in the robot workspace that can be reached by a robot. (Paragraph 0065, “In evaluating the feasibility of welding at each of the divided nodes or node-waypoint pairs, the controller 108 may perform multiple computations. In some examples, each of the multiple computations may be mutually exclusive from one another. In some examples, the first computation may include kinematic feasibility computation, which computes for whether the arm of the robot 110 of the welding robot being employed can mechanically reach (or exist) at the state defined by the node or node-waypoint pair. In some examples, in addition to the first computation, a second computation—which may be mutually exclusive to the first computation—may also be performed by the controller 108. The second computation may include determining whether the arm of the robot 110 will encounter a collision (e.g., collide with the workspace 101 or a structure in the workspace 101) when accessing the portion of the seam (e.g., the node or node-waypoint pair in question).”)
Re Claim 6, Lonsberry discloses wherein the graphical annotations specify one or more regions of interest in the visual representation. (Paragraph 0078, “At step 1002, the method 1000 includes identifying an expected orientation and an expected position of a candidate seam on a part to be welded based on a CAD model of the part. The expected orientation and expected position may be determined using the annotations provided by a user/operator to the CAD model.”; Examiner is treating candidate seam as region of interest.)
Re Claim 7, Lonsberry discloses wherein the one or more regions of interest are used to generate one or more waypoints at which the robot can reach the one or more regions of interest. (Paragraph 0062, “The controller 108 can generate one or more feasible simulate (or evaluate, both terms used interchangeably herein) weld paths should they physically be feasible. A weld path can be a path that the welding robot takes to weld the candidate seam. In some examples, the weld path may include all the waypoints of a seam.”)
Re Claim 8, Lonsberry discloses wherein the natural language annotations specify instructions associated with the robot task. (Paragraph 0053, “In some examples, the user can be provided with an option to update welding parameters. For example, the user interface 106 can provide the user with a list of different welding parameters. The user can select a specific parameter to be updated. Changes to the selected parameter can be made using a drop-down menu, via text input, etc. This update can be transmitted to the controller 108 so that the controller 108 can update the instructions for the robot 110.”)
Re Claim 9, Lonsberry discloses wherein the one or more annotations comprise an annotation associated with a prior robot task. (Paragraph 0038, “As is now described, the controller 108 may then determine whether the candidate seams are actually seams and may optionally perform additional processing using a priori information, such as CAD models of the parts and seams.”)
Re Claim 11, Lonsberry discloses wherein the one or more annotations specify one or more objects that can be manipulated by a robot. (Paragraph 0054, “The user can annotate seams that are to be welded in the representation via the user interface 106.”; Paragraph 0033, “The controller 108 may then use the point clouds 400, 500 (or, in some examples, image data useful to generate the point clouds 400, 500) to identify and locate seams, such as the seams 406, 506, to plan a welding path along the seams 406, 506, and to lay welds along the seams 406, 506 according to the path plan and using the robot 110 (FIG. 1)”)
Re Claim 13, Lonsberry discloses wherein the optimization process comprises validating the one or more annotations by algorithmically checking the one or more annotations against one or more preconditions. (Paragraph 0054, “The user can annotate seams that are to be welded in the representation via the user interface 106.”; Paragraph 0058, “First, the controller 108 may discretize an identified seam into a sequence of waypoints.”; Paragraph 0059, “The controller 108 may divide each waypoint into multiple nodes. Each node can represent a possible orientation of the weld head at that waypoint.”; Paragraph 0065, “In evaluating the feasibility of welding at each of the divided nodes or node-waypoint pairs, the controller 108 may perform multiple computations. In some examples, each of the multiple computations may be mutually exclusive from one another. In some examples, the first computation may include kinematic feasibility computation, which computes for whether the arm of the robot 110 of the welding robot being employed can mechanically reach (or exist) at the state defined by the node or node-waypoint pair.”; Examiner is treating the reachability of the nodes as the precondition)
Re Claim 14, Lonsberry discloses wherein the one or more preconditions comprise one or more of reachability of an annotated location, distance to a singularity, and travel distance. (Paragraph 0054, “The user can annotate seams that are to be welded in the representation via the user interface 106.”; Paragraph 0058, “First, the controller 108 may discretize an identified seam into a sequence of waypoints.”; Paragraph 0059, “The controller 108 may divide each waypoint into multiple nodes. Each node can represent a possible orientation of the weld head at that waypoint.”; Paragraph 0065, “In evaluating the feasibility of welding at each of the divided nodes or node-waypoint pairs, the controller 108 may perform multiple computations. In some examples, each of the multiple computations may be mutually exclusive from one another. In some examples, the first computation may include kinematic feasibility computation, which computes for whether the arm of the robot 110 of the welding robot being employed can mechanically reach (or exist) at the state defined by the node or node-waypoint pair.”; Examiner is treating the reachability of the nodes as the precondition)
Re Claim 15, Lonsberry discloses wherein the optimization process comprises precomputing a set of trajectories between two or more waypoints of the set of waypoints to optimize one or more of speed, safety, obstacle avoidance, and travel distance. (Paragraph 0062, “The controller 108 can generate one or more feasible simulate (or evaluate, both terms used interchangeably herein) weld paths should they physically be feasible. A weld path can be a path that the welding robot takes to weld the candidate seam. In some examples, the weld path may include all the waypoints of a seam. In some examples, the weld path may include some but not all the waypoints of the candidate seam. The weld path can include the motion of the robot and the weld head as the weld head moves between each waypoint-node pair. Once a feasible path between node-waypoint pairs is identified, a feasible node-waypoint pair for the next sequential waypoint can be identified should it exist. Those skilled in the art will recognize that many search trees or other strategies may be employed to evaluate the space of feasible node-waypoint pairs. As discussed in further detail herein, a cost parameter can be assigned or calculated for movement from each node-waypoint pair to a subsequent node-waypoint pair. The cost parameter can be associated with a time to move, an amount of movement (e.g., including rotation) between node-waypoint pairs, and/or a simulated/expected weld quality produced by the weld head during the movement.”)
Re Claim 19, Lonsberry discloses a system for specifying a robot task, the system comprising: (Paragraph 0020, “FIG. 1 is a block diagram of an autonomous robotic welding system 100, in accordance with various examples. The system 100 includes a manufacturing workspace 101, a user interface 106, a controller 108, and storage 109 storing a database 112. The system 100 may include other components or subsystems that are not expressly described herein.”)
a robot; a robot workspace associated with one or more regions in an environment of the robot that the robot can reach; (Paragraph 0065, “In evaluating the feasibility of welding at each of the divided nodes or node-waypoint pairs, the controller 108 may perform multiple computations. In some examples, each of the multiple computations may be mutually exclusive from one another. In some examples, the first computation may include kinematic feasibility computation, which computes for whether the arm of the robot 110 of the welding robot being employed can mechanically reach (or exist) at the state defined by the node or node-waypoint pair. In some examples, in addition to the first computation, a second computation—which may be mutually exclusive to the first computation—may also be performed by the controller 108. The second computation may include determining whether the arm of the robot 110 will encounter a collision (e.g., collide with the workspace 101 or a structure in the workspace 101) when accessing the portion of the seam (e.g., the node or node-waypoint pair in question).”)
one or more cameras mounted in the environment of the robot; (Paragraph 0030, “FIG. 3 is a schematic diagram of an autonomous robotic welding system 300, in accordance with various examples. The system 300 is an example of the system 100 of FIG. 1 and the system 200 of FIG. 2, with like numerals referring to like components… In operation, the sensors 302 collect 2D images of the workspace 301 and provide the 2D images to a controller (not expressly shown in FIG. 3).”; Paragraph 0021, “The sensors 102 are configured to capture information about the workspace 101. In examples, the sensors 102 are image sensors that are configured to capture visual information (e.g., two-dimensional (2D) images) about the workspace 101. For instance, the sensors 102 may include cameras (e.g., cameras with built-in laser)…”)
and an electronic device comprising one or more processors configured to perform a method comprising: (Paragraph 0028, “In some examples, one or more functions attributed to execution of the executable code 111 may be implemented by hardware. For instance, multiple processors may be useful to perform one or more discrete tasks of the executable code 111.”)
capturing, via the one or more cameras, one or more images of the robot workspace; (Paragraph 0030, “FIG. 3 is a schematic diagram of an autonomous robotic welding system 300, in accordance with various examples. The system 300 is an example of the system 100 of FIG. 1 and the system 200 of FIG. 2, with like numerals referring to like components… In operation, the sensors 302 collect 2D images of the workspace 301 and provide the 2D images to a controller (not expressly shown in FIG. 3).”; Paragraph 0021, “The sensors 102 are configured to capture information about the workspace 101. In examples, the sensors 102 are image sensors that are configured to capture visual information (e.g., two-dimensional (2D) images) about the workspace 101. For instance, the sensors 102 may include cameras (e.g., cameras with built-in laser)…”)
displaying a visual representation of the robot workspace to a user based on the one or more captured images; (Paragraph 0030, “The system 300 further includes a UI 306 coupled to the workspace 301. In operation, the sensors 302 collect 2D images of the workspace 301 and provide the 2D images to a controller (not expressly shown in FIG. 3). The controller generates 3D representations (e.g., point clouds) of the workspace 301, such as the fixtures 316, a part supported by the fixtures 316, and/or other structures within the workspace 301.”; Paragraph 0051, “In examples, the user interface 106 can provide the user with an option to view candidate seams. For example, the user interface 106 may provide a graphical representation of a part 114 and/or candidate seams on a part 114.”)
receiving, from the user, one or more annotations associated with the visual representation… (Paragraph 054, “The user can annotate seams that are to be welded in the representation via the user interface 106.”)
determining a set of waypoints based on the one or more annotations via an optimization process; (Paragraph 40, “Accordingly, the controller 108 compares a first seam (e.g., a candidate seam on a part 114 that has been verified as an actual seam) to a second seam (e.g., a seam annotated (e.g., by an operator/user) on the CAD model corresponding to the first seam) to determine differences between the first and second seams. Seams on the CAD model may be annotated as described above. The first seam and the second seam can be in nearly the same location, in instances in which the CAD model and/or controller 108 accurately predicts the location of the candidate seam. Alternatively, the first seam and the second seam can partially overlap, in instances in which the CAD model and/or controller 108 is partially accurate. The controller 108 may perform a comparison of the first seam and the second seam. This comparison of first seam and the second seam can be based in part on shape and relative location in space of both the seams. Should the first seam and the second seam be relatively similar in shape and be proximal to each other, the second seam can be identified as being the same as the first seam. In this way, the controller 108 can account for the topography of the surfaces on the part that are not accurately represented in the CAD models. In this manner, the controller 108 can identify candidate seams and can sub-select or refine or update candidate seams relative to the part using a CAD model of the part. Each candidate seam can be a set of updated points that represents the position and orientation of the candidate seam relative to the part.”)
and obtaining the robot task based on the set of waypoints and the one or more annotations. (Paragraph 76, “In some examples, the welding instructions can be based on the type of seam (e.g., butt joint, corner joint, edge joint, lap joint, tee joint, and/or the like). In some examples, the welding instructions can be updated based on input from a user via a user interface (e.g., user interface 106 in FIG. 1). The user can select a candidate seam to be welded from all the available candidate seams via the user interface. Path planning can be performed for the selected candidate seam and welding instructions can be generated for the selected candidate seam.”) but does not explicitly disclose …wherein the one or more annotations include at least one of graphical annotations and natural language annotations;
However, Bell teaches a system/method for generating a three-dimensional composite scene of an environment using a robotic arm, and discloses …wherein the one or more annotations include at least one of graphical annotations and natural language annotations; (Paragraph 113, “A reference dataset 401 containing auxiliary 3D data that is spatially aligned 403 (using the 3D alignment techniques described herein or other techniques) to known captured 3D data of the object or environment being viewed 402 may be used as a source of information to display in an overlay to, in combination with, or in replacement of, the current captured scene 402 as seen by the outward-facing camera or 3D capture device 304. Types of auxiliary datasets include, but are not limited to: Marks or labels identifying captured objects or locations; these marks may be manually or automatically (via image or object recognition algorithms) made during a current or prior capture of the environment or via a user of an external tool manipulating the captured 3D data, for example using the annotation interface described herein.”; Examiner is treating the marks as graphical annotations and the labels as natural language annotations)
Thus, it would be obvious to a person of ordinary skill in the art at the time of the effective filing of the application to modify Lonsberry’s autonomous welding robot system with Bell’s manual auxiliary dataset because it would allow the user to provide the robot with both structural and contextual feedback regarding its welding operations.
Re Claim 20, Lonsberry discloses a non-transitory computer-readable storage medium storing one or more programs, (Paragraph 0028, “Furthermore, the controller 108 may interact with the database 112, for example, by storing data to the database 112 and/or retrieving data from the database 112. The database 112 may more generally be stored in any suitable type of storage 109 that is configured to store any and all types of information. In some examples, the database 112 can be stored in storage 109 such as a random access memory (RAM), a memory buffer, a hard drive, an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), a read-only memory (ROM), Flash memory, and the like.”) the one or more programs comprising instructions, which when executed by one or more processors of an electronic device of a system for specifying a robot task, cause the electronic device to perform: (Paragraph 0028, “In examples, the storage 109 stores executable code 111, which, when executed, causes the controller 108 to perform one or more actions attributed herein to the controller 108, or, more generally, to the system 100.”)
capturing, via one or more cameras, one or more images of a robot workspace, where the one or more cameras are mounted in an environment of the robot workspace; (Paragraph 0030, “FIG. 3 is a schematic diagram of an autonomous robotic welding system 300, in accordance with various examples. The system 300 is an example of the system 100 of FIG. 1 and the system 200 of FIG. 2, with like numerals referring to like components… In operation, the sensors 302 collect 2D images of the workspace 301 and provide the 2D images to a controller (not expressly shown in FIG. 3).”; Paragraph 21, “The sensors 102 are configured to capture information about the workspace 101. In examples, the sensors 102 are image sensors that are configured to capture visual information (e.g., two-dimensional (2D) images) about the workspace 101. For instance, the sensors 102 may include cameras (e.g., cameras with built-in laser)…”)
displaying a visual representation of the robot workspace to a user based on the one or more captured images; (Paragraph 0030, “The system 300 further includes a UI 306 coupled to the workspace 301. In operation, the sensors 302 collect 2D images of the workspace 301 and provide the 2D images to a controller (not expressly shown in FIG. 3). The controller generates 3D representations (e.g., point clouds) of the workspace 301, such as the fixtures 316, a part supported by the fixtures 316, and/or other structures within the workspace 301.”; Paragraph 0051, “In examples, the user interface 106 can provide the user with an option to view candidate seams. For example, the user interface 106 may provide a graphical representation of a part 114 and/or candidate seams on a part 114.”)
receiving, from the user, one or more annotations associated with the visual representation… (Paragraph 054, “The user can annotate seams that are to be welded in the representation via the user interface 106.”)
determining a set of waypoints based on the one or more annotations via an optimization process; (Paragraph 40, “Accordingly, the controller 108 compares a first seam (e.g., a candidate seam on a part 114 that has been verified as an actual seam) to a second seam (e.g., a seam annotated (e.g., by an operator/user) on the CAD model corresponding to the first seam) to determine differences between the first and second seams. Seams on the CAD model may be annotated as described above. The first seam and the second seam can be in nearly the same location, in instances in which the CAD model and/or controller 108 accurately predicts the location of the candidate seam. Alternatively, the first seam and the second seam can partially overlap, in instances in which the CAD model and/or controller 108 is partially accurate. The controller 108 may perform a comparison of the first seam and the second seam. This comparison of first seam and the second seam can be based in part on shape and relative location in space of both the seams. Should the first seam and the second seam be relatively similar in shape and be proximal to each other, the second seam can be identified as being the same as the first seam. In this way, the controller 108 can account for the topography of the surfaces on the part that are not accurately represented in the CAD models. In this manner, the controller 108 can identify candidate seams and can sub-select or refine or update candidate seams relative to the part using a CAD model of the part. Each candidate seam can be a set of updated points that represents the position and orientation of the candidate seam relative to the part.”)
and obtaining the robot task based on the set of waypoints and the one or more annotations. (Paragraph 76, “In some examples, the welding instructions can be based on the type of seam (e.g., butt joint, corner joint, edge joint, lap joint, tee joint, and/or the like). In some examples, the welding instructions can be updated based on input from a user via a user interface (e.g., user interface 106 in FIG. 1). The user can select a candidate seam to be welded from all the available candidate seams via the user interface. Path planning can be performed for the selected candidate seam and welding instructions can be generated for the selected candidate seam.”) but does not explicitly disclose …wherein the one or more annotations include at least one of graphical annotations and natural language annotations;
However, Bell teaches a system/method for generating a three-dimensional composite scene of an environment using a robotic arm, and discloses …wherein the one or more annotations include at least one of graphical annotations and natural language annotations; (Paragraph 113, “A reference dataset 401 containing auxiliary 3D data that is spatially aligned 403 (using the 3D alignment techniques described herein or other techniques) to known captured 3D data of the object or environment being viewed 402 may be used as a source of information to display in an overlay to, in combination with, or in replacement of, the current captured scene 402 as seen by the outward-facing camera or 3D capture device 304. Types of auxiliary datasets include, but are not limited to: Marks or labels identifying captured objects or locations; these marks may be manually or automatically (via image or object recognition algorithms) made during a current or prior capture of the environment or via a user of an external tool manipulating the captured 3D data, for example using the annotation interface described herein.”; Examiner is treating the marks as graphical annotations and the labels as natural language annotations)
Thus, it would be obvious to a person of ordinary skill in the art at the time of the effective filing of the application to modify Lonsberry’s autonomous welding robot medium with Bell’s manual auxiliary dataset because it would allow the user to provide the robot with both structural and contextual feedback regarding its welding operations.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Lonsberry (US20220266453A1) in view of Bell (US10529143B2) and Caron L'ecuyer (US 20200086493 A1).
Re Claim 10, Modified Lonsberry does not explicitly disclose wherein the one or more annotations specify one or more landmark objects that can be used to localize a robot in the robot workspace, however Caron L'ecuyer teaches a vision guiding method for a robot arm and discloses wherein the one or more annotations specify one or more landmark objects that can be used to localize a robot in the robot workspace (Paragraph 0107, “In an embodiment in which a plurality of objects have been identified in the displayed image, the user selects one of the objects as being the target object and confirms that the associated action would be performed. In this case, the identification of the target object and the confirmation of its associated action are received at step 60.”; Paragraph 0109, “Once the confirmations have been received, the robot arm 12 is automatically moved at step 62 so as to position the end effector of the robot arm 12 at a predefined position or distance from the target object.”; Examiner is treating the target object selected by the user as one of the marks or labels disclosed in Bell)
Thus, it would be obvious to a person of ordinary skill in the art at the time of the effective filing of the application to modify Modified Lonsberry’s autonomous welding robot method with Caron L'ecuyer’s relative object positioning because it would allow the user’s selected annotations to serve as references for the positioning of the robot during its welding operations.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Lonsberry (US20220266453A1) in view of Bell (US10529143B2) and Declerck (US20200276714A1).
Re Claim 12, Modified Lonsberry does not explicitly disclose wherein the optimization process comprises generating metadata associated with the robot task, however Declerck teaches a robotic manipulation system featuring visual object metadata and discloses wherein the optimization process comprises generating metadata associated with the robot task. (Paragraph 0066, “In a particular example, the task module 64 causes the robot 14 (via the robot control module 62 and the controller 52) to pick and place the object 18 before picking and placing the object 16. In some examples, the objects 16, 18 can include unique visual object identifiers 86, 88 (e.g., scannable patterns or codes). Visual information for the unique object identifiers 86, 88, as well as any metadata (e.g., weight, dimensions, materials, etc.) for the object that can be linked to the object identifiers 86, 88, can be stored in visual reference data 67 and referred to by the optics system 80 and/or the task module 64, such that it can be determined if the appropriate physical object has been visually located for the next gripping operation by the robot 14 and what certain physical parameters of that object are.”; Examiner is treating the newly collected object metadata from object identifiers 86 and 88 to be used for robot 14’s gripping operation as generating metadata associated with the robot task.)
Thus, it would be obvious to a person of ordinary skill in the art at the time of the effective filing of the application to modify Modified Lonsberry’s autonomous welding robot method with Declerck’s visual object metadata because it would increase the consistency of the robot’s welds across production runs and reduce defects.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Lonsberry (US20220266453A1) in view of Bell (US10529143B2) and Kjellsson (US20070276538A1).
Re Claim 16, Modified Lonsberry does not explicitly disclose wherein the robot task comprises one or more of performing pick and/or place operations, operating a machine, and loading and/or unloading a machine, however Kjellsson teaches an industrial robot multi tool and discloses (Paragraph 0038, “The robot and/or automation application with a tool according to the present invention may applied to operations such automobile assembly and to manufacturing processes used in automobile manufacturing… The robot application may comprise a plurality of tools, both specialised tools for welding, painting etc as well as other more general devices, grippers, claws, manipulators and so on that carry out manipulation-type tasks such as holding, placing, pick and place, and even packing of components or subcomponents in a container.”; Paragraph 0039, “Automatic tool changes in particular are also facilitated by this invention, enabling automatic tool changes without interrupting production.”)
Thus, it would be obvious to a person of ordinary skill in the art at the time of the effective filing of the application to modify Modified Lonsberry’s autonomous welding robot method with Kjellsson’s industrial robot tool swapping and object manipulation tool because they would expand the possible tasks the user could accomplish through Modified Lonsberry’s annotation guided robot. Kjellsson’s robot being able to perform machining tasks sequentially using multiple tool types such as spot welding and object manipulation would be highly beneficial for Modified Lonsberry’s robot which already performs welding. Allowing Modified Lonsberry’s robot to swap from its welding tool to an object manipulation tool to place the to be welded part in the workspace and/or pick the welded part up from the workspace following the welding operation would further remove the need for human involvement in the manufacturing of the particular part.
Claim 17-18 is rejected under 35 U.S.C. 103 as being unpatentable over Lonsberry (US20220266453A1) in view of Bell (US10529143B2) and Barajas (US9387589B2).
Re Claim 17, Modified Lonsberry does not explicitly disclose further comprising providing visual feedback corresponding to an appearance of the robot workspace after the robot task is completed, however Barajas teaches a visual debugging of robotic tasks, and discloses further comprising providing visual feedback corresponding to an appearance of the robot workspace after the robot task is completed. (Col. 6, Line 24-36, “The Marker Generator Module 60 of FIG. 1A is operable to produce and output marker models or visual markers (arrow 62) associated with the actions of the robot 12 shown in FIG. 1. When inserted into the Simulator Module (SIM) 70, the visual markers (arrow 62) provide a graphical indicator, e.g., an icon, picture, or cartoon representation of, current and future actions of the robot 12. The visual markers (arrow 62) may be associated with specific objects that the robot 12 is or will be attending to, and indicate attributes of the object related to a planned action of the robot 12. Marker attributes can indicate, for example, the position of the robot 12 or a position, orientation, or trajectory of an object such as object 21 or 23 of FIG. 1 in a current or future state.”; Examiner is treating future state markers of object 21/23 as visual workspace feedback after the robot task is completed)
Thus, it would be obvious to a person of ordinary skill in the art at the time of the effective filing of the application to modify Modified Lonsberry’s autonomous welding robot method with Barajas’ future visual markers because: (Barajas, Col 1, Line 57-67, “For instance, all possible action trajectories and future robot and object positions and orientations may be depicted via the visual markers in a simulated environment viewable via a display screen of the GUI. By integrating the action planning and simulation modules, the controller allows for visualization of all currently planned actions and also facilitates necessary control adjustments via the GUI. This in turn enables a user to change the robots behavior in real time. That is, a user can quickly discern all possible future actions of the robot and quickly ascertain whether the action planning module has chosen a desirable solution.”)
Re Claim 18, Modified Lonsberry does not explicitly disclose wherein the visual feedback comprises a graphical display overlaid on the visual representation, however Barajas teaches a visual debugging of robotic tasks, and discloses wherein the visual feedback comprises a graphical display overlaid on the visual representation. (Col. 6, Line 24-36, “The Marker Generator Module 60 of FIG. 1A is operable to produce and output marker models or visual markers (arrow 62) associated with the actions of the robot 12 shown in FIG. 1. When inserted into the Simulator Module (SIM) 70, the visual markers (arrow 62) provide a graphical indicator, e.g., an icon, picture, or cartoon representation of, current and future actions of the robot 12. The visual markers (arrow 62) may be associated with specific objects that the robot 12 is or will be attending to, and indicate attributes of the object related to a planned action of the robot 12. Marker attributes can indicate, for example, the position of the robot 12 or a position, orientation, or trajectory of an object such as object 21 or 23 of FIG. 1 in a current or future state.”; Col. 6, Line 62-67, “In FIG. 2, the region surrounding the object 123 to be grasped provides a visualization of the Target Marker (G) for the next pickup action of the robot 12. Arrows A and D are Trajectory Markers indicating an approach trajectory (A) and a departure trajectory (D) for a grasp action related to an action of the robot 12 with respect to the Target Marker (G).”; Col. 7, Line 8-13, “As shown in FIG. 3, the Objective Markers (O1, O2) indicate where an object will be in future steps. When actions are planned by the Action Planner Module 80 of FIG. 1A, one or more Objective Markers (O1, O2) can be used to show all objects in the final position they are expected to be in after being manipulated by the robot 12.”; Examiner is treating future state markers of object 21/23 as visual workspace feedback after the robot task is completed)
PNG
media_image1.png
949
794
media_image1.png
Greyscale
Thus, it would be obvious to a person of ordinary skill in the art at the time of the effective filing of the application to modify Modified Lonsberry’s autonomous welding robot method with Barajas’ future visual markers because: (Barajas, Col 1, Line 57-67, “For instance, all possible action trajectories and future robot and object positions and orientations may be depicted via the visual markers in a simulated environment viewable via a display screen of the GUI. By integrating the action planning and simulation modules, the controller allows for visualization of all currently planned actions and also facilitates necessary control adjustments via the GUI. This in turn enables a user to change the robots behavior in real time. That is, a user can quickly discern all possible future actions of the robot and quickly ascertain whether the action planning module has chosen a desirable solution.”)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW KILLIAN PEPPER whose telephone number is (571)272-6815. The examiner can normally be reached Monday - Friday 10:00-6:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abby Lin can be reached at (571) 270-3976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/A.K.P./Examiner, Art Unit 3657 /ABBY LIN/Supervisory Patent Examiner, Art Unit 3657
Prosecution Insights