DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This is a Final Office Action on the merits.
Claims 1-3, 5-8, 10, 14-15, and 18-20 were amended; and claims 21-22 are new. Therefore, claims 1-22 are pending in the current application.
Response to Amendment/Arguments
Applicant’s arguments filed on 12/22/2025 have been fully considered as below.
Regarding the rejections made under 35 USC 101, the Applicant’s amendments and arguments, see page 8 of Remarks, have been fully considered but are not persuasive in view of the amendments. Therefore, the rejections made under 35 USC 101 are maintained as discussed below.
Regarding the rejections made under 35 USC 112, the Applicant’s amendments and arguments, see page 8 of Remarks, have been fully considered and are partially persuasive in view of the amendments. Therefore, the rejection made under 35 USC 112 to claim 17 is maintained. New issue under this section is also discussed below.
Regarding the objection to drawings, the Applicant’s amendments and arguments, see pages 6-7 of Remarks, have been fully considered and are persuasive in view of the amendments. Therefore, the objection to drawings is withdrawn.
Regarding the rejections under 35 USC 102 and/or 103 to the claims, the Applicant’s amendments and arguments, see pages 8-11 of Remarks, have been fully considered but are moot in view of the new grounds of rejection provided below, in light of newly found prior art, which was necessitated based on Applicant's amendments which changed the scope of the claims.
Specification
The disclosure is objected to because of the following informalities: In published par. [0032] of the specification, “In some cases, sensor data being used to reconstruct estimated state (i.e., “cold start”) my include insufficient information” should read “In some cases, sensor data being used to reconstruct estimated state (i.e., “cold start”) may include insufficient information”.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2, 7, 17 and 19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 2, Applicant provides claim limitation, “wherein the processor is further configured to generate said indication that the estimated state information is not suitable to the next placement decision with respect to the next object to be stacked on or in the receptacle”, however, based on currently provided claim language, it is unclear whether “the estimated state information” is referring to “estimated state information” in line 1 of claim 1 or “current estimated state information” in line 5 of claim 1. Therefore, this renders the claim indefinite. The examiner assumes “the current estimated state information” for further examination. According, correction and/or clarification is/are required.
Regarding claim 7, Applicant provides claim limitation, “wherein said indication that the estimated state information is not suitable to the next placement decision comprises an indication that a proposed the next placement would result in instability”, however, based on currently provided claim language, it is unclear whether “the estimated state information” is referring to “estimated state information” in line 1 of claim 1 or “current estimated state information” in line 5 of claim 1. Therefore, this renders the claim indefinite. The examiner assumes “the current estimated state information” for further examination. According, correction and/or clarification is/are required.
Regarding claim 17, Applicant provides claim limitation, “wherein the processor is configured to generate the constructed estimated state at least in part by determining based on a simulation whether a candidate estimated state satisfies a stability criterion”, however, based on currently provided claim language, it is unclear what the metes and bounds regarding the term “a candidate estimated state satisfies a stability criterion” and how the claimed limitation is applied to determine whether the candidate estimated state satisfies the stability criterion. Therefore, this renders the claim indefinite. According, correction and/or clarification is/are required.
Regarding claim 19, Applicant provides claim limitation, “wherein said indication is generated based on the sensor information from said one or more sensors and comprises an indication that a perceived real-world state is inconsistent with the stored estimated state information”, however, based on currently provided claim language, it is unclear whether “the estimated state information” is referring to “estimated state information” in line 1 of claim 18 or “current estimated state information” in line 4 of claim 18. Therefore, this renders the claim indefinite. The examiner assumes “the current estimated state information” for further examination. According, correction and/or clarification is/are required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 12, 13 and 17 are rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract idea without significantly more.
Claim 12 recites:
A robotic system, comprising:
a memory configured to store estimated state information representing an estimated state of one or both of a receptacle and one or more objects stacked on or in the receptacle; and
a processor coupled to the memory and configured to:
receive an indication that a current estimated state information stored in the memory is not suitable for use in making a next placement decision with respect to a next object to be stacked on or in the receptacle; and
store in the memory a constructed estimated state information replacing the current estimated state information, wherein the constructed estimated state information is generated at least in part by processing sensor information generated by one or more sensors positioned and configured to generate the sensor information providing an at least partial view of one or both of the receptacle and the one or more objects stacked on or in the receptacle.
wherein the processor is further configured to determine and implement the next placement decision.
The limitation of determining and implementing the next placement decision cover performance of the limitations in the mind but for the recitation of generic computer component (a processor). That is, other than reciting a processor/robotic device and other generic computer components such as memory, nothing in the claim element precludes the step from being performed in the mind. For example, one can mentally determine object placement decision on a pallet based on information provided by a vision system. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the "2106.04(a)(2) Abstract Idea Groupings. Accordingly, the claim recites an abstract idea and therefore, not patent eligible.
Step 2A Prong two evaluation: Practical Application - No
In Step 2A, Prong two of the 2019 PEG, a claim is to be evaluated whether, as a whole, it integrates the recited judicial exception into a practical application. As noted in MPEP 2106.04(d), it must be determined whether any additional elements in the claim beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the judicial exception. The courts have indicated that additional elements such as: merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application.”
In the present case, the additional limitations beyond the above-noted abstract idea are as follows (where the underlined portions are the “additional limitations” while the bolded portions continue to represent the “abstract idea”)
The claim recites additional element of using a processor/robotic device to perform mental determine and implement steps. The processor/robotic device in the steps are recited at a high-level of generality (i.e., as a generic processor performing a generic computer function of determine and implement) such that it amounts no more than mere instructions to apply the exception using a generic computer component.
The claim recites additional steps of “… store estimated state information …”, “… receive an indication …”, “… store in the memory a constructed estimated state information …” and “… providing an at least partial view …”. These steps are recited at high level of generality (i.e. as general means of gathering/storing/outputting data for use in the determine and implement steps). This amounts to mere data gathering and outputting of result which are forms of insignificant extra-solution activities (MPEP 2106.05(g)). The claim is therefore not eligible.
Step 2B evaluation: Inventive Concept - No
As discussed with respect to Step 2A Prong Two, the additional elements in the claim amount to no more than mere instructions to apply the exception using a generic computer component(s). The same analysis applies here in 2B, i.e., mere instructions to apply an exception on a generic computer cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Under the 2019 PEG, a conclusion that an additional element is insignificant extra-solution activity in Step 2A should be reevaluated in Step 2B. Here, the storing step was considered to be extra-solution activities in Step 2A, and thus it is reevaluated in Step 2B to determine if it is more than what is well-understood, routine, conventional activity in the field. The specification and background herein do not provide any indication that the storage apparatus is anything other than possible generic, off the-shelf computer components and the Symantec, TLI, and OIP Techs, court decisions cited in MPEP 2106.05(d)(II) indicate that mere collection or receipt of data over a network is a well-understood, routine, and conventional function when it is claimed in a merely generic manner (as it is here). Accordingly, a conclusion that the storing step is well-understood, routine, conventional activities are supported under Berkheimer Option 2. For these reasons, there is no inventive concept in the claim, and thus it is ineligible.
Regarding claim 13, the additional element in the claim does not integrate the identified abstraction in practical application. The claim is ineligible.
Regarding claim 17, the claim recites, “wherein the processor is configured to generate the constructed estimated state at least in part by determining based on a simulation whether a candidate estimated state satisfies a stability criterion.” The limitation of generate estimated state by determining covers performance in the mind but for the recitation of generic computer component (a processor). That is, other than reciting a processor/robotic device and other generic computer components such as memory, storage medium nothing in the claim element precludes the step from being performed in the mind. For example, one can mentally generate estimated state from observed simulation results being displayed. The claim is therefore ineligible.
Claims 12, 13 and 17 are rejected under 35 U.S.C. 101 as being drawn to an abstract idea without significant more, and thus are ineligible.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-8, 11-16, and 18-20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Diankov et al. (US 11794346 B2, hereinafter “Diankov”).
Regarding claim 1, Diankov discloses a robotic system (Diankov, see at least Figs. 1, 2, robotic system 100), comprising:
a memory configured to store estimated state information representing an estimated state of one or both of a receptacle and one or more objects stacked on or in the receptacle (Diankov, see at least Figs. 2, 3A, 3B, 5, col. 6-7, col. 19-20, storage device 204 configured to store information related to robot system, e.g., object tracking data includes locations and/or orientations of the objects at drop locations/pallet/receptacle, discretized object models 302, the discretized platform models 304, predetermined discretized models that represents the available packages and/or the task location 116 of Figs. 1, 5, 3D packing plan); and
a processor coupled to the memory (Diankov, see at least Fig. 2, processor 202 coupled to the storage device 204) and configured to:
receive an indication that a current estimated state information stored in the memory is not suitable for use in making a next placement decision with respect to a next object to be stacked on or in the receptacle (Diankov, see at least Fig. 8, cols. 28-31, determine whether one or more errors by identifying deviations/disparities/collision between expected object in the packing sequence and computer model is configured to represent an expected shaped and/or an expected surface contour (e.g., a set of height estimates corresponding to the expected placement surface) that correspond to the current progress of the 3D packing plan/discretized model based on identifying the objects that have been placed, e.g., palletized case heights in the context of a placement area error pose a risk of collision (e.g., due to a disparity between expected and actual height measurements of the placement area 340 as described above), previously placed objects 508 that were placed at a wrong location and/or orientation, one or more previously placed objects 508 that have moved, shifted, and/or fallen); and
store in the memory a constructed estimated state information replacing the current estimated state information (Diankov, see at least Fig. 8, cols. 31-33, it would be obvious for the storage device 204 to store the replaced/updated 3D packing plan/discretized model that replacing the current 3D packing plan/discretized model since storage device 204 configured to store information related to robot system), wherein the constructed estimated state information is generated at least in part by processing sensor information (Diankov, see at least Fig. 8, cols. 31-33, generating the replaced/updated 3D packing plan/discretized model based on real-time data from sensor) generated by one or more sensors positioned and configured to generate the sensor information providing an at least partial view of one or both of the receptacle and the one or more objects stacked on or in the receptacle (Diankov, see at least Fig. 2, cols. 8-9, sensors 216 configured to capture and analyze image data the object, the start location, and task location, e.g., the associated poses, a packing/placement location, and/or other processing results).
Regarding claim 2, Diankov teaches all the limitation of claim 1 as discussed above. Diankov further teaches wherein the processor is further configured to generate said indication that the estimated state information is not suitable to the next placement decision with respect to the next object to be stacked on or in the receptacle (Diankov, see at least Fig. 2, cols. 30-31, “In some embodiments, the robotic system 100 can identify whether packaging conditions pose a risk of collision independently from or in response to determining one or more errors have occurred (at blocks 832-836). The robotic system 100 can analyze the sensor data to identify a risk of collision between robotic units and/or objects should the robotic system 100 continue to package/palletize the target object 112. If the target object 112 is the expected object according to the packing sequence, the robotic system 100 can recalculate and/or access the approach path 510 of FIG. 5 for placing the target object 112 according to the 3D packing plan […] As such, in response to determining one or more errors and/or identifying one or more packaging conditions posing a risk of collision, the robotic system 100 (at block 840) can determine a response to the errors and/or potential collisions identified at blocks 832-838”).
Regarding claim 3, Diankov teaches all the limitation of claims 1-2 as discussed above. Diankov further teaches wherein the processor is configured to generate said indication at least in part by processing the sensor information from said one or more sensors (Diankov, see at least Fig. 2, cols. 8-9, sensors 216 configured to capture and analyze image data the object, the start location, and task location, e.g., the associated poses, a packing/placement location, and/or other processing results).
Regarding claim 4, Diankov teaches all the limitation of claims 1-3 as discussed above. Diankov further teaches wherein the sensor information comprises one or both of image data and depth information (Diankov, see at least Fig. 2, col. 8, “the sensors 216 can include one or more imaging devices 222 (e.g., visual and/or infrared cameras, two-dimensional (2D) and/or 3D imaging cameras, distance measuring devices such as lidars or radars, etc.)”).
Regarding claim 5, Diankov teaches all the limitation of claims 1-3 as discussed above. Diankov further teaches wherein the sensor information indicates a perceived real-world state that is inconsistent with the current estimated state information stored in the memory (Diankov, see at least col. 11, lines 36-48, col. 27, lines 23-60, the sensor information indicates a perceived real-world state that is inconsistent with the stored estimated state information, such as error in arriving objects, unrecognizable and/or unexpected packages on the pallet and/or a shift in one or more of the previously placed objects).
Regarding claim 6, Diankov teaches all the limitation of claims 1-2 as discussed above. Diankov further teaches wherein the processor is configured to generate said indication at least in part by simulating performance of a propose placement of the next object to be stacked on or in the receptacle (Diankov, see at least Figs. 3A-B, 4A-B, 5, 6A-B, 8, cols. 12, “the robotic system 100 of FIG. 1 can generate the candidate positions 360 of FIG. 3B based on overlapping the discretized object model 302 of FIG. 3A of the target object 112 of FIG. 1 over the discretized platform model 304 of the task location 116 of FIG. 1”; col. 34-36, the processor configured to evaluate the placement score for each of the candidate positions 360).
Regarding claim 7, Diankov teaches all the limitation of claim 1 as discussed above. Diankov further teaches wherein said indication that the estimated state information is not suitable to the next placement decision comprises an indication that a proposed the next placement would result in instability (Diankov, see at least Fig. 8, cols. 35-36, the processor configured to calculate the placement score for each of the candidate positions 360 according to one or more of the placement conditions, such as the collision probabilities, separation distances between packages, differences in package dimensions/fragility ratings/package weights for horizontally adjacent packages, in order to determine that the candidate positions 360 would result in instability).
Regarding claim 8, Diankov teaches all the limitation of claim 1 as discussed above. Diankov further teaches wherein said indication that the estimated state information is not suitable to the next placement decision comprises an indication that a proposed next placement would result in damage to one or both of the next object to be placed and one or more objects stacked on or in the receptacle (Diankov, see at least Fig. 8, cols. 35-36, the processor configured to calculate the placement score for each of the candidate positions 360 according to one or more of the placement conditions, such as the collision probabilities, separation distances between packages, differences in package dimensions/fragility ratings/package weights for horizontally adjacent packages, in order to determine that the candidate positions 360 would result in damage to next object to be placed and one or more objects in the stack).
Regarding claim 11, Diankov teaches all the limitation of claim 1 as discussed above. Diankov further teaches wherein the receptacle comprises a pallet (Diankov, see at least Figs. 1, 3B, 6A-6B, col. 5, “the task can include placing the objects on a target location (e.g., on top of a pallet and/or inside a bin/cage/box/case)”).
Regarding claim 12, Diankov teaches all the limitation of claim 1 as discussed above. Diankov further teaches wherein the processor is further configured to determine and implement the next placement decision (Diankov, see at least Fig. 7, col. 26, lines 15-34, col. 31, lines 26-39, the processor configured to determine and implement the next placement decision based on the stacking plan for placing the available packages on the platform).
Regarding claim 13, Diankov teaches all the limitation of claims 1 and 12 as discussed above. Diankov further wherein the processor is further configured to update the estimated state information based at least in part on a result of implementing said next placement decision (Diankov, see at least col. 32, line 17-29, col. 37, lines 35-55, “In some embodiments, the robotic system 100 can update or re-identify real-time packaging conditions after placing the target object 112. In other words, following block 818, the control flow can move to block 801 and/or 802. Accordingly, the robotic system 100 can update/identify the next incoming object as the target object 112. The robotic system 100 can also update information for placement area 340 and/or the previously placed objects 508 thereon to include the recently placed object”).
Regarding claim 14, Diankov teaches all the limitation of claim 1 as discussed above. Diankov further teaches wherein the sensor information comprises a point cloud defining a partial image of an object stacked on or in the receptacle (Diankov, see at least col. 7, lines 32-62, col. 8, lines 52-67, “The imaging devices 222 can generate representations of the detected environment, such as digital images and/or point clouds, that may be processed via machine/computer vision (e.g., for automatic inspection, robot guidance, or other robotic applications)”) and the processor is configured to process the sensor information at least in part by determining one or more dimensions of the object (Diankov, see at least Figs. 4A-B, cols. 9-10, the processor configured to discretize the image data based on one or more dimensions of the object) and including in the constructed estimated state information representing the object as a cuboid (Diankov, see at least col. 23, col. 33, lines 43-53, the processor configured to generate discretized object models 302 and convert each of the discretized object models 302 into 3D state based on adding the object height, such as height of a package/box, for placing packages (e.g., cases and/or boxes) onto a platform (e.g., a pallet) and/or for placing the packages accordingly).
Regarding claim 15, Diankov teaches all the limitation of claim 1 as discussed above. Diankov further teaches wherein the processor is configured to process the sensor information based at least in part on one or more of a position of the one or more sensors (Diankov, see at least Fig. 2, cols. 16-17, camera 122/216 located over the task location 116, i.e. pallet), a feature of the one or more sensors (Diankov, see at least Figs. 1, 2, col. 6, the camera 122/216 is a three-dimensional (3D) vision cameras), and a configuration of the one or more sensors (Diankov, see at least Fig. 2, col. 8, lines 41-67, col. 9, lines 20-27, the sensor 216 configured to detect or measure one or more physical properties of the robotic system 100, the surround environment, and generate representations of the detected environment, such as digital images and/or point clouds, that may be processed via machine/computer vision).
Regarding claim 16, Diankov teaches all the limitation of claim 1 as discussed above. Diankov further teaches further comprising a communication interface couple to the processor and configured to receive the sensor information (Diankov, see at last Fig. 2, cols. 7-8, “The communication devices 206 can include circuits configured to communicate with external or remote devices via a network. For example, the communication devices 206 can include receivers, transmitters, modulators/demodulators (modems), signal detectors, signal encoders/decoders, connector ports, network cards, etc. The communication devices 206 can be configured to send, receive, and/or process electrical signals according to one or more communication protocols (e.g., the Internet Protocol (IP), wireless communication protocols, etc.)”).
Regarding claim 18, Diankov discloses a method (Diankov, see at least Fig. 8 of operating the robotic system 100 of FIG. 1), comprising:
storing, by one or more processors, estimated state information representing an estimated state of one or both of a receptacle and one or more objects stacked on or in the receptacle (Diankov, see at least Figs. 2, 3A, 3B, 5, col. 6-7, col. 19-20, storage device 204 configured to store information related to robot system, e.g., object tracking data includes locations and/or orientations of the objects at drop locations/pallet/receptacle, discretized object models 302, the discretized platform models 304, predetermined discretized models that represents the available packages and/or the task location 116 of Figs. 1, 5, 3D packing plan);
receiving an indication that a current estimated state information stored in the memory is not suitable for use in making a next placement decision with respect to a next object to be stacked on or in the receptacle (Diankov, see at least Fig. 8, cols. 28-31, determine whether one or more errors by identifying deviations/disparities/collision between expected object in the packing sequence and computer model is configured to represent an expected shaped and/or an expected surface contour (e.g., a set of height estimates corresponding to the expected placement surface) that correspond to the current progress of the 3D packing plan/discretized model based on identifying the objects that have been placed, e.g., palletized case heights in the context of a placement area error pose a risk of collision (e.g., due to a disparity between expected and actual height measurements of the placement area 340 as described above), previously placed objects 508 that were placed at a wrong location and/or orientation, one or more previously placed objects 508 that have moved, shifted, and/or fallen);
storing in the memory a constructed estimated state information replacing the current estimated state information (Diankov, see at least Fig. 8, cols. 31-33, it would be obvious for the storage device 204 to store the replaced/updated 3D packing plan/discretized model that replacing the current 3D packing plan/discretized model since storage device 204 configured to store information related to robot system), wherein the constructed estimated state information is generated at least in part by processing sensor information (Diankov, see at least Fig. 8, cols. 31-33, generating the replaced/updated 3D packing plan/discretized model based on real-time data from sensor) generated by one or more sensors positioned and configured to generate the sensor information providing an at least partial view of one or both of the receptacle and the one or more objects stacked on or in the receptacle (Diankov, see at least Fig. 2, cols. 8-9, sensors 216 configured to capture and analyze image data the object, the start location, and task location, e.g., the associated poses, a packing/placement location, and/or other processing results).
Regarding claim 19, Diankov teaches all the limitation of claim 18 as discussed above. Diankov further teaches wherein said indication is generated based on the sensor information from said one or more sensors and comprises an indication that a perceived real-world state is inconsistent with the stored estimated state information (Diankov, see at least col. 11, lines 36-48, col. 27, lines 23-60, the sensor information indicates a perceived real-world state that is inconsistent with the stored estimated state information, such as computer vision errors, error in arriving objects, unrecognizable and/or unexpected packages on the pallet and/or a shift in one or more of the previously placed objects).
Regarding claim 20, Diankov discloses a computer program product embodied in a non-transitory computer readable medium and comprising computer instructions (Diankov, see at least Fig. 1, cols. 5, 7, “The processors 202 can include data processors (e.g., central processing units (CPUs), special-purpose computers, and/or onboard servers) configured to execute instructions (e.g., software instructions) stored on the storage devices 204 (e.g., computer memory)”) for:
storing, by one or more processors, estimated state information representing an estimated state of one or both of a receptacle and one or more objects stacked on or in the receptacle (Diankov, see at least Figs. 2, 3A, 3B, 5, col. 6-7, col. 19-20, storage device 204 configured to store information related to robot system, e.g., object tracking data includes locations and/or orientations of the objects at drop locations/pallet/receptacle, discretized object models 302, the discretized platform models 304, predetermined discretized models that represents the available packages and/or the task location 116 of Figs. 1, 5, 3D packing plan);
receiving an indication that a current estimated state information stored in the memory is not suitable for use in making a next placement decision with respect to a next object to be stacked on or in the receptacle (Diankov, see at least Fig. 8, cols. 28-31, determine whether one or more errors by identifying deviations/disparities/collision between expected object in the packing sequence and computer model is configured to represent an expected shaped and/or an expected surface contour (e.g., a set of height estimates corresponding to the expected placement surface) that correspond to the current progress of the 3D packing plan/discretized model based on identifying the objects that have been placed, e.g., palletized case heights in the context of a placement area error pose a risk of collision (e.g., due to a disparity between expected and actual height measurements of the placement area 340 as described above), previously placed objects 508 that were placed at a wrong location and/or orientation, one or more previously placed objects 508 that have moved, shifted, and/or fallen);
storing in the memory a constructed estimated state information replacing the current estimated state information (Diankov, see at least Fig. 8, cols. 31-33, it would be obvious for the storage device 204 to store the replaced/updated 3D packing plan/discretized model that replacing the current 3D packing plan/discretized model since storage device 204 configured to store information related to robot system), wherein the constructed estimated state information is generated at least in part by processing sensor information (Diankov, see at least Fig. 8, cols. 31-33, generating the replaced/updated 3D packing plan/discretized model based on real-time data from sensor) generated by one or more sensors positioned and configured to generate the sensor information providing an at least partial view of one or both of the receptacle and the one or more objects stacked on or in the receptacle (Diankov, see at least Fig. 2, cols. 8-9, sensors 216 configured to capture and analyze image data the object, the start location, and task location, e.g., the associated poses, a packing/placement location, and/or other processing results).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Diankov et al. (US 11794346 B2, hereinafter “Diankov”) as applied to claim 1 above, and further in view of Yu et al. (US 20230128352 A1, hereinafter “Yu”).
Regarding claim 9, Diankov teaches all the limitations of claim 1 as discussed above. Diankov fails to explicitly teach wherein the processor is configured to process the sensor information at least in part by filtering the sensor information to remove noise.
Yu, a prior art reference in the field of robot control, teaches the processor configured to filter and remove noise from the image captured by the camera (see at least Fig. 4A, par. [0050]).
It would have been obvious to a person of ordinary skill in the art at the time of invention to modify the apparatus of Diankov to have the processor is configured to process the sensor information at least in part by filtering the sensor information to remove noise as taught by Yu for performing a smoothing or smoothening operation on the image.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Diankov et al. (US 11794346 B2, hereinafter “Diankov”) as applied to claim 1 above, and further in view of Chavez et al. (US 10549928 B1, hereinafter “Chavez”).
Regarding claim 10, Diankov teaches all the limitations of claim 1 as discussed above. Diankov fails to explicitly teach wherein the processor is configured to process the sensor information at least in part by augmenting the sensor information to fill one or more gaps in the view of the sensor information of one or both of the receptacle and the one or more objects stacked on or in the receptacle.
Chavez, a prior art reference in the field of robot control, teaches the processor configured to fill gaps in the image data by using data from a best fit model (e.g., a bounding “box” for an item that is rectangular or nearly so in all dimensions) and superimpose a graphical depiction of the bounding container on the composite and/or raw image of the object stacked on the receptacle (e.g., raw video) to provide a composite display, e.g., to a human user monitoring the operation (see at least Figs. 7, 4A-4C, cols. 7-8, col. 9, lines 30-62).
It would have been obvious to a person of ordinary skill in the art at the time of invention to modify the apparatus of Diankov to have the processor is configured to process the sensor information at least in part by augmenting the sensor information to fill one or more gaps in the view of sensor information of one or both of a receptacle and one or more objects stacked on or in the receptacle as taught by Chavez for visualizing the operation to pick and place an object on a pallet to a human user monitoring the operation.
Claim 17 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Diankov et al. (US 11794346 B2, hereinafter “Diankov”) as applied to claim 1 above, and further in view of Kimoto (US 20190143504 A1).
Regarding claim 17, Diankov teaches all the limitations of claim 1 as discussed above. Diankov fails to explicitly teach wherein the processor is configured to generate the constructed estimated state at least in part by determining based on a simulation whether a candidate estimated state satisfies a stability criterion.
Kimoto teaches, see at least Figs. 2, 3, 4a, 4b, 6, 7, par. [0043, 0048, 0049], a palletizing system 30 comprises a processor 10 configured to generate a physical model 62 including physical model of objects on a receptacle by repeating steps S3 to S5 to modify the physical model 62 using a simulation in which the vibration or the shock and the gravity force are considered until the physical model 62 satisfies a stability condition, e.g., physical model objects are aligned and positioned at the generally center of the physical model 62 or calculating whether an object is stably stacked based on a distance, from the position of the gravity center of a object to a peripheral edge of a second object under the first object, is larger than the margin.
It would have been obvious to a person of ordinary skill in the art at the time of invention to modify the apparatus of Diankov to include wherein the processor is configured to generate the constructed estimated state at least in part by determining based on a simulation whether a candidate estimated state satisfies a stability criterion as taught by Kimoto. This modification would allow to calculating a change in position and/or orientation of model represents objects on a receptacle due to vibration and/or shock (Kimoto, par. [0005-0006]).
Regarding claim 22, Diankov teaches all the limitations of claim 1 as discussed above. Diankov fails to explicitly teach wherein the processor is configured to generate the constructed estimated state information at least in part by iteratively modifying a candidate estimated state using a physics-based simulation until the candidate estimated state satisfies a predefined stability criterion, and wherein the physics-based simulation provides one or more of derivatives and gradients indicating a direction and extent of modification needed to achieve stability.
Kimoto teaches, see at least Figs. 2, 3, 4a, 4b, 6, 7, par. [0043, 0048, 0049], a palletizing system 30 comprises a processor 10 configured to generate a physical model 62 including physical model of objects on a receptacle by repeating steps S3 to S5 to modify the physical model 62 using a simulation in which the vibration or the shock and the gravity force are considered until the physical model 62 satisfies a predefined stability condition, e.g., physical model objects are aligned and positioned at the generally center of the physical model 62 or calculating whether an object is stably stacked based on a distance, from the position of the gravity center of a object to a peripheral edge of a second object under the first object, is larger than the margin; and wherein the simulation is configured to adjust the position and orientation of the physical model of objects located in the physical model 62 by applying equal force to the models of the three workpieces in the front-back and left-right directions.
It would have been obvious to a person of ordinary skill in the art at the time of invention to modify the apparatus of Diankov to include wherein the processor is configured to generate the constructed estimated state information at least in part by iteratively modifying a candidate estimated state using a physics-based simulation until the candidate estimated state satisfies a predefined stability criterion, and wherein the physics-based simulation provides one or more of derivatives and gradients indicating a direction and extent of modification needed to achieve stability as taught by Kimoto. This modification would allow to calculating a change in position and/or orientation of model represents objects on a receptacle due to vibration and/or shock (Kimoto, par. [0005-0006]).
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Diankov et al. (US 11794346 B2, hereinafter “Diankov”) as applied to claim 1 above, and further in view of Whitman et al. (US 20210041887 A1, hereinafter “Whitman”).
Regarding claim 21, Diankov teaches all the limitations of claim 1 as discussed above. Diankov fails to explicitly teach wherein the processor is configured to process the sensor information at least in part by merging a plurality of adjacent voxels derived from the sensor information into one or more larger cuboids based on one or more of image segmentation, known object sizes, and similarity in voxel heights along a vertical axis, and including the one or more larger cuboids in the constructed estimated state information.
Whitman teaches, see at least Fig. 1A, 2A-2L, par. [0049-0062, 0067-0068], a system and method for processing sensor data to generate a voxel map to represent a three-dimensional space about the robot 100 by combining voxels 212 based on one or more of image segmentation, known object sizes, and similarity in voxel heights along a vertical axis, e.g., combining voxels with same height 212h corresponds to the same object, classification and in the same vertical column of the 3D grid to form one or more segments 214, and including the one or more segments 214 in the voxel map.
It would have been obvious to a person of ordinary skill in the art at the time of invention to modify the apparatus of Diankov to include wherein the processor is configured to process the sensor information at least in part by merging a plurality of adjacent voxels derived from the sensor information into one or more larger cuboids based on one or more of image segmentation, known object sizes, and similarity in voxel heights along a vertical axis, and including the one or more larger cuboids in the constructed estimated state information as taught by Whitman. This modification would allow to simplify classification of various objects within the environment of the robot to generate a depth map based on the sensor data (Whitman, par. [0004]).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TRANG DANG whose telephone number is (703)756-1049. The examiner can normally be reached Monday-Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Khoi Tran can be reached at (571)272-6919. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TRANG DANG/ Examiner, Art Unit 3656 /KHOI H TRAN/Supervisory Patent Examiner, Art Unit 3656
Prosecution Insights