DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Remarks
The claims being considered in this application are those submitted on 12/30/2024. Claims 1-21 are pending.
Priority
The applicant’s claim to priority of PRO 63/675,066 on 07/24/2024 is acknowledged.
Information Disclosure Statement
The information disclosure statements filed on 05/16/2025 and 09/16/2025 have been annotated and considered.
Double Patenting
Claims 1, 6-7, 11-12, and 17-18 of this application is patentably indistinct from Claims 1, 5-6, 15-16, and 20-21 of Application No. 19/004,885. Pursuant to 37 CFR 1.78(f), when two or more applications filed by the same applicant or assignee contain patentably indistinct claims, elimination of such claims from all but one application may be required in the absence of good and sufficient reason for their retention during pendency in more than one application. Applicant is required to either cancel the patentably indistinct claims from all but one application or maintain a clear line of demarcation between the applications. See MPEP § 822.
Claims 1, 6-7, 11-12, and 17-18 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over Claims 1, 5-6, 15-16, and 20-21 of copending Application No. 19/004,885 (reference application) which is also published as US 20260027729 A1.
Although the claims at issue are not identical, they are not patentably distinct from each other because of the following reasons and table reproduced below:
Instant App 19/004,849
Reference Application 19/004,885
Claim #
Claim Language
Claim #
Claim Language
1
A computer-implemented method for performing object detection in a robotic pick-and- place system, comprising:
capturing an image of a field of view of a sensor associated with a robotic arm;
identifying, in the image and using object detection logic, a target object in the image;
transmitting information about the target object from the object detection logic to object tracking logic that operates separately from the object detection logic; and
instructing the robotic arm to pick up the target object.
1
A computer-implemented method for performing object tracking in a robotic pick-and-place system, comprising:
capturing an image of a field of view of a sensor associated with a robotic arm;
receiving, from object detection logic, information about a target object in the field of view;
updating, using object tracking logic that operates separately from the object detection logic, a location of the target object in the image;
and using the updated location to instruct the robotic arm to pick up the target object.
6
wherein the target object is identified in the image using a machine learning construct.
5
wherein the target object's location is determined using a machine learning construct.
7
wherein the machine learning construct is one head of a multi-headed machine learning model.
6
wherein the machine learning construct comprises one or more heads of a multi-headed model.
11
A system comprising:
a robotic arm;
a conveyor for conveying objects to the robotic arm;
a sensor; and
a processor configured to perform the method of claim 1
15
A system comprising:
a robotic arm;
a conveyor for conveying objects to the robotic arm;
a sensor; and
a processor configured to perform the method of claim 1
12
A computer-readable medium storing instructions configured to cause one or more processors to:
capture an image of a field of view of a sensor associated with a robotic arm;
identify, in the image and using object detection logic, a target object in the image;
transmit information about the target object from the object detection logic to object tracking logic that operates separately from the object detection logic; and
instruct the robotic arm to pick up the target object.
16
A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to:
capture an image of a field of view of a sensor associated with a robotic arm;
receive, from object detection logic, information about a target object in the field of view;
update, using object tracking logic that operates separately from the object detection logic, a location of the target object in the image; and
using the updated location to instruct the robotic arm to pick up the target object.
17
wherein the target object is identified in the image using a machine learning construct
20
wherein the target object's location is determined using a machine learn construct.
18
wherein the machine learning construct is one head of a multi-headed machine learning model
21
wherein the machine learn construct comprises one or more heads of a multi-headed model
Regarding the Claim 1s (and similarly Claim 12 of ‘849 and Claim 16 of ‘885), the instant application 19/004,849 (‘849) recites: “identifying, in the image and using object detection logic, a target object in the image;” whereas the reference application 19/004,885 (‘885) recites: “receiving, from object detection logic, information about a target object in the field of view;”. These limitations correspond as the receiving information regarding a target object and identifying a target object have the same functionality.
Furthermore, the instant application 19/004,849 recites: “transmitting information about the target object from the object detection logic to object tracking logic that operates separately from the object detection logic;” and the reference application 19/004,885 recites: “updating, using object tracking logic that operates separately from the object detection logic, a location of the target object in the image;”. The ‘updating’ of ‘885 is a type of transmission. Additionally, ‘885 recites updating a location of an object, which is a type of information being transmitted about the object (as in ‘849).
Thus, although the language is slightly different, these methods are performing the same functions.
Regarding the dependent claims, the language is essentially the same with only slight variations (for example, Claim 6 of ‘849 recites “…the target object is identified…” versus Claim 5 of ‘885 recites “…the target object's location is determined…”) However, in addition to the reasons stated above, the claims have the same functions and are thus also rejected under provisional nonstatutory double patenting.
This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented.
Claim Objections
Claims 12-21 are objected to because of the following informalities:
Claims 12-21 should recite “non- transitory computer-readable medium” rather than merely “computer readable medium” to avoid potential 101 issues regarding subject matter eligibility. Support in applicant’s specification for “non- transitory computer-readable medium” was found in at least ¶0013.
Additionally, Claims 19-21 appear to be missing text as they recite “The computer-readable of Claim 12…” or “The computer-readable of Claim 19”. This should be corrected in addition to reciting “non- transitory” as stated above.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4, 6, 8, 10-12, 15, 17, 19, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Balzer et. al. (US 20250028300 A1, corresponding to EP4177013A1 in IDS dated 05/16/2025) in view of Horowitz et. al. (US 20230191608 A1, IDS 05/16/2025).
Regarding Claim 1, Balzer discloses:
A computer-implemented method for performing object detection in a robotic pick-and-place system, comprising: (See at least ¶0001 via "The present invention comprises a distributed at least partially computer-implemented system for controlling at least one robot for gripping objects of different types")
capturing an image of a field of view of a sensor associated with a robotic arm; (See at least ¶0014 via "an optical device used to capture image data of objects in the robot's working area;" and ¶0131 via "The camera C is preferably set up so that it can capture the working area CB, B, T of the robot R. The camera C can be designed to capture depth images and, if necessary, intensity images. The real image data captured of the object O is transmitted by the camera C to the local processing unit LCU in order to be evaluated there, i.e., on the local processing unit LCU. This is done using the previously trained neural network ANN")
identifying, in the image and using object detection logic, a target object in the image; (See at least ¶0020 via " The neural network is trained to understand the spatial arrangement (the “state”) of some objects to be gripped (next to each other or partially on top of each other, superimposed, in a box or on a conveyor belt, etc.) so that the system can react by calculating gripping instructions that are specific to the detected state. The state is characterized by the class/type of the respective objects (object identification), their position (position estimation), their orientation (orientation estimation) relative to coordinate system inside the working area of the robot" and ¶0016 via "wherein the pre-trained or post-trained ANN is evaluated in an inference phase on the local computing unit determining a result data set from the image data captured by the optical capture device, which is then used to calculate the instructions for the end effector unit to grasp the object and transmit them to the robot controller for execution")
transmitting information about the target object from the object detection logic to object (See at least ¶0016 via "wherein the pre-trained or post-trained ANN is evaluated in an inference phase on the local computing unit determining a result data set from the image data captured by the optical capture device, which is then used to calculate the instructions for the end effector unit to grasp the object and transmit them to the robot controller for execution;" as well as ¶0017. Additionally, see ¶0036-¶0037 via "During inference, a result data set with the labels is determined from the captured image data in which the object to be gripped is depicted, in particular the position and orientation of the object in the working area and optionally the class. The result data set is an intermediate result…The modified ICP algorithm is applied to the result data set in order to calculate a refined result data set that serves as the final result." ***Wherein, the ANN corresponds to the object detection logic, and the ICP algorithm corresponds to the subsequent/separate processing that uses the result data outputted from the ANN for pose refinement. Additionally see ¶0131-¶0132.)
instructing the robotic arm to pick up the target object (See at least ¶0132 via " The gripping instructions with the specific gripping positions can then be calculated from this refined result data set 200. The gripping instructions can be transferred to the robot controller RC for execution so that it can calculate the movement planning of an end effector unit EE. The robot controller RC can then control the robot R to execute the movement.").
However, although Balzer discloses the subsequent and separate processing via the ICP algorithm (See at least ¶0036-¶0037 and ¶0131-¶0132); Balzer does not explicitly disclose the tracking.
Nevertheless, Horowitz--who is directed towards using machine learning to recognize variant objects--discloses: object tracking logic (See at least ¶0122 via "For example, object tracking logic 708 is configured to apply one or more machine learning models to visual sensor signals (e.g., images) to identify object regions (e.g., masks, bounding polygons, etc.) that define the shape and location of the objects…In some embodiments, object tracking logic 708 is configured to compare the determined classification(s) associated with each object against a dynamically configurable set of target object criteria. For example, the set of target object criteria may describe one or more classifications associated with objects that should be harvested by a sorting device." and ¶0124 via "as object tracking logic 708 obtains additional vision sensor signals associated with an object, object tracking logic 708 is configured to apply machine learning models on the sensed signals to update the confidence values associated with different portions of the object's dynamically variable bounding polygon." and ¶0131 via "Object tracking logic 708 is configured to overlay dynamic model estimates of object locations (which could appear as a bounding polygon around each detected object) on each panorama, resulting in a large image that captures object trajectory over time from a pixel-based viewpoint". As well as ¶0167 via " In some embodiments, whether the target object has been previously detected before is determined by comparing the determined attributes associated with the target object with the stored attributes of previously detected target objects. For example, the attributes of a previously detected target object can be stored in a data structure that has been maintained for that previously detected target object. If the determined attributes associated with the new target object match those of a previously detected target object, then the target object is not new." as well as Figures 17-18).
Therefore, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the given invention to modify Balzer in view of supplementing Balzer's ICP algorithm/pose refinement processing with Horowitz' object tracking logic to account for a target object over time by continuing to track/update the object's state and predict trajectory/motion through multiple successive images/detections in order to improve the positioning accuracy/pose refinement in a dynamic environment (such as on a conveying surface) : "…one or more portions of a bounding polygon associated with the target object are updated based at least in part on the new sensed data. As more portions of the target object can be depicted and more clearly observed in the sensed data, the more accurate the bounding polygon estimate for the target object can be." [Horowitz ¶0184] and "Due to the vision sensors of the object recognition device capturing images of objects as they move across the conveyance, a generated panorama can show a stream of objects laid out on a portion of the surface of the conveyor device…the objects' trajectories can be calculated and evolved using inter-image frame differences as detected by one or more machine learning models, and thus become highly accurate paths within the range of the vision sensors" [Horowitz ¶0131].
Regarding Claim 12, Balzer discloses:
A computer-readable medium storing instructions configured to cause one or more processors to: (See at least ¶0107 via "The computer program product may be stored on a data carrier or a computer-readable storage medium.")
capture an image of a field of view of a sensor associated with a robotic arm; (See at least ¶0014 via "an optical device used to capture image data of objects in the robot's working area;" and ¶0131 via "The camera C is preferably set up so that it can capture the working area CB, B, T of the robot R. The camera C can be designed to capture depth images and, if necessary, intensity images. The real image data captured of the object O is transmitted by the camera C to the local processing unit LCU in order to be evaluated there, i.e., on the local processing unit LCU. This is done using the previously trained neural network ANN")
identify, in the image and using object detection logic, a target object in the image; (See at least ¶0020 via " The neural network is trained to understand the spatial arrangement (the “state”) of some objects to be gripped (next to each other or partially on top of each other, superimposed, in a box or on a conveyor belt, etc.) so that the system can react by calculating gripping instructions that are specific to the detected state. The state is characterized by the class/type of the respective objects (object identification), their position (position estimation), their orientation (orientation estimation) relative to coordinate system inside the working area of the robot" and ¶0016 via "wherein the pre-trained or post-trained ANN is evaluated in an inference phase on the local computing unit determining a result data set from the image data captured by the optical capture device, which is then used to calculate the instructions for the end effector unit to grasp the object and transmit them to the robot controller for execution")
transmit information about the target object from the object detection logic to object (See at least ¶0016 via "wherein the pre-trained or post-trained ANN is evaluated in an inference phase on the local computing unit determining a result data set from the image data captured by the optical capture device, which is then used to calculate the instructions for the end effector unit to grasp the object and transmit them to the robot controller for execution;" as well as ¶0017. Additionally, see ¶0036-¶0037 via "During inference, a result data set with the labels is determined from the captured image data in which the object to be gripped is depicted, in particular the position and orientation of the object in the working area and optionally the class. The result data set is an intermediate result…The modified ICP algorithm is applied to the result data set in order to calculate a refined result data set that serves as the final result." ***Wherein, the ANN corresponds to the object detection logic, and the ICP algorithm corresponds to the subsequent/separate processing that uses the result data outputted from the ANN for pose refinement. Additionally see ¶0131-¶0132.)
instruct the robotic arm to pick up the target object (See at least ¶0132 via " The gripping instructions with the specific gripping positions can then be calculated from this refined result data set 200. The gripping instructions can be transferred to the robot controller RC for execution so that it can calculate the movement planning of an end effector unit EE. The robot controller RC can then control the robot R to execute the movement.")
However, although Balzer discloses the subsequent and separate processing via the ICP algorithm (See at least ¶0036-¶0037 and ¶0131-¶0132); Balzer does not explicitly disclose the tracking.
Nevertheless, Horowitz--who is directed towards using machine learning to recognize variant objects--discloses: object tracking logic (See at least ¶0122 via "For example, object tracking logic 708 is configured to apply one or more machine learning models to visual sensor signals (e.g., images) to identify object regions (e.g., masks, bounding polygons, etc.) that define the shape and location of the objects…In some embodiments, object tracking logic 708 is configured to compare the determined classification(s) associated with each object against a dynamically configurable set of target object criteria. For example, the set of target object criteria may describe one or more classifications associated with objects that should be harvested by a sorting device." and ¶0124 via "as object tracking logic 708 obtains additional vision sensor signals associated with an object, object tracking logic 708 is configured to apply machine learning models on the sensed signals to update the confidence values associated with different portions of the object's dynamically variable bounding polygon." and ¶0131 via "Object tracking logic 708 is configured to overlay dynamic model estimates of object locations (which could appear as a bounding polygon around each detected object) on each panorama, resulting in a large image that captures object trajectory over time from a pixel-based viewpoint". As well as ¶0167 via " In some embodiments, whether the target object has been previously detected before is determined by comparing the determined attributes associated with the target object with the stored attributes of previously detected target objects. For example, the attributes of a previously detected target object can be stored in a data structure that has been maintained for that previously detected target object. If the determined attributes associated with the new target object match those of a previously detected target object, then the target object is not new." as well as Figures 17-18)
Therefore, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the given invention to modify Balzer in view of supplementing Balzer's ICP algorithm/pose refinement processing with Horowitz' object tracking logic to account for a target object over time by continuing to track/update the object's state and predict trajectory/motion through multiple successive images/detections in order to improve the positioning accuracy/pose refinement in a dynamic environment (such as on a conveying surface) : "…one or more portions of a bounding polygon associated with the target object are updated based at least in part on the new sensed data. As more portions of the target object can be depicted and more clearly observed in the sensed data, the more accurate the bounding polygon estimate for the target object can be." [Horowitz ¶0184] and "Due to the vision sensors of the object recognition device capturing images of objects as they move across the conveyance, a generated panorama can show a stream of objects laid out on a portion of the surface of the conveyor device…the objects' trajectories can be calculated and evolved using inter-image frame differences as detected by one or more machine learning models, and thus become highly accurate paths within the range of the vision sensors" [Horowitz ¶0131].
Regarding Claims 4 and 15 respectively, Modified Balzer discloses the computer-implemented method of Claim 1 and the computer-readable medium storing instructions of Claim 12.
Furthermore, Balzer discloses: wherein the sensor is uniquely associated with a single robotic arm (See at least ¶0004 via "A precise grip requires an equally precise estimate of the position and orientation (possibly also “recognition” of the type) of the objects on the basis of images from a 3D camera, which can be mounted at the end of the robot arm or above the box")
Regarding Claim 6 and 17 respectively, Modified Balzer discloses the computer-implemented method of Claim 1 and the computer-readable medium storing instructions of Claim 12.
Furthermore, Balzer discloses: wherein the target object is identified in the image using a machine learning construct (See at least ¶0020 via "The neural network is trained to understand the spatial arrangement (the “state”) of some objects to be gripped (next to each other or partially on top of each other, superimposed, in a box or on a conveyor belt, etc.) so that the system can react by calculating gripping instructions that are specific to the detected state. The state is characterized by the class/type of the respective objects (object identification), their position (position estimation), their orientation (orientation estimation) relative to coordinate system inside the working area of the robot.")
Regarding Claim 8 and 19 respectively, Modified Balzer discloses the computer-implemented method of Claim 1 and the computer-readable of Claim 12.
Furthermore, Horowitz discloses: further comprising: capturing a further image of the field of view of the sensor; and (See at least ¶0128 via "Once created, the dynamic model evolves its state over time based on the defined dynamics, and further updates based on new measurements, which is newly sensed data (e.g., new image frames) that is obtained for that object" and ¶0042 as well as Figures 16-17)
tracking a location of the target object in the further image using the object tracking logic (See at least ¶0036 via " In some embodiments, the trajectory associated with the target object is determined based on a plurality of sets of sensed data of the target (e.g., a series of images of the target object as taken from above the conveyor device(s))").
Therefore, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the given invention to modify Modified Balzer in view of Horowitz's tracking of the target objects location based on further imaged/sensed data in order to account for the target object 's motion over time by continuing to track/update the object's state through multiple successive images/detections in order to improve the positioning accuracy/pose refinement in a dynamic environment (such as on a conveying surface) : "…one or more portions of a bounding polygon associated with the target object are updated based at least in part on the new sensed data. As more portions of the target object can be depicted and more clearly observed in the sensed data, the more accurate the bounding polygon estimate for the target object can be." [Horowitz ¶0184] and "Due to the vision sensors of the object recognition device capturing images of objects as they move across the conveyance, a generated panorama can show a stream of objects laid out on a portion of the surface of the conveyor device…the objects' trajectories can be calculated and evolved using inter-image frame differences as detected by one or more machine learning models, and thus become highly accurate paths within the range of the vision sensors" [Horowitz ¶0131].
Regarding Claim 10 and 21 respectively, Modified Balzer discloses the computer-implemented method of Claim 8 and the computer-readable of claim 19.
Furthermore, Horowitz discloses: wherein identifying the target object in the image comprises identifying a first location of the target object using the object detection logic, and (See at least ¶0122 via "object tracking logic 708 is configured to apply one or more machine learning models to visual sensor signals (e.g., images) to identify object regions (e.g., masks, bounding polygons, etc.) that define the shape and location of the objects")
wherein tracking the location of the target object comprises updating the first location using the object tracking logic (See at least ¶0124 via "object tracking logic 708 obtains additional vision sensor signals associated with an object, object tracking logic 708 is configured to apply machine learning models on the sensed signals to update the confidence values associated with different portions of the object's dynamically variable bounding polygon.").
Therefore, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the given invention to modify Modified Balzer in view of Horowitz's identification of the target objects location/updating based on further imaged/sensed data in order to account for the target object 's motion over time by continuing to track/update the object's state through multiple successive images/detections in order to improve the positioning accuracy/pose refinement in a dynamic environment (such as on a conveying surface) : "…one or more portions of a bounding polygon associated with the target object are updated based at least in part on the new sensed data. As more portions of the target object can be depicted and more clearly observed in the sensed data, the more accurate the bounding polygon estimate for the target object can be." [Horowitz ¶0184] and "Due to the vision sensors of the object recognition device capturing images of objects as they move across the conveyance, a generated panorama can show a stream of objects laid out on a portion of the surface of the conveyor device…the objects' trajectories can be calculated and evolved using inter-image frame differences as detected by one or more machine learning models, and thus become highly accurate paths within the range of the vision sensors" [Horowitz ¶0131].
Regarding Claim 11, Balzer discloses:
A system comprising: a robotic arm; (See at least Figure 6)
a conveyor for conveying objects to the robotic arm; (See at least Figure 6 and ¶0142 via " FIG. 6 again shows a similar scenario to that illustrated in FIGS. 4 and 5, with the difference that the objects O can be arranged here in a container B on a schematically illustrated conveyor belt CB and the robot R is instructed to place the objects O in a transport container T")
a sensor; and (See at least Figure 6 via Camera "C")
a processor configured to perform the method of claim 1 (See at least Claim 1 rejection as well as ¶0038 via "In particular, the local operating method can be closely coupled with the robot controller. In particular, a processor may be a central processing unit (CPU), a microprocessor or a microcontroller, for example an application-specific integrated circuit or a digital signal processor, possibly in combination with a memory unit for storing program instructions, etc").
Claims 2 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Balzer et. al. (US 20250028300 A1, corresponding to EP4177013A1 in IDS dated 05/16/2025) and Horowitz et. al. (US 20230191608 A1, IDS 05/16/2025) in view of Sun et. al. (US 10906188 B1, IDS 05/16/2025).
Regarding Claims 2 and 13 respectively, Modified Balzer discloses the computer-implemented method of Claim 1 and the computer-readable medium storing instructions of Claim 12.
Furthermore, although Balzer discloses objects touching (See Figure 6 which depicts objects in container B that are touching), Balzer does not appear to explicitly disclose the target object contacting one or more objects.
Nevertheless, Sun discloses: wherein the target object is contacting one or more other objects in the image (See at least Col. 6 Lines 39-42 via "In some embodiments, a degree of overlap (i.e., occlusion by other items) is estimated for each item, and the degree of overlap is taken into consideration in selecting a next item to attempt to grasp.")
Therefore, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the given invention to modify Modified Balzer in view of Sun's consideration for item overlap in order to account for situations where there may be a plurality of objects in an area that are overlapping each other, and thus configure the picking in a way that has a higher probability for grasp success: "for each item a score may be computed to estimate the probability of grasp success, and in some embodiments the score is determined at least in part by the degree of overlap/occlusion by other items. Less occluded items may be more likely to be selected, for example, other considerations being equal" [Sun Col. 6 Lines 42-48].
Claims 3, 5, 14, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Balzer et. al. (US 20250028300 A1, corresponding to EP4177013A1 in IDS dated 05/16/2025) and Horowitz et. al. (US 20230191608 A1, IDS 05/16/2025) in view of Fujikawa et. al. (US 20140015956 A1).
Regarding Claims 3 and 14 respectively, Modified Balzer discloses the computer-implemented method of Claim 1 and the computer-readable medium storing instructions of Claim 12.
Furthermore, Modified Balzer does not explicitly disclose the field of view comprising an area upstream/out of reach of the robot arm.
Nevertheless, Fujikawa--who is directed towards image processing--discloses: wherein the image of the field of view of the sensor comprises an area upstream of the robotic arm that is not yet accessible to the robotic arm at a time that the image is captured (See at least Figure 24 as well as Figures 26-27 which illustrate the image field of view/range comprising an area upstream of the robotic arm that is not yet accessible to the robot arm/outside of the tracking/operating range for the robot)
PNG
media_image1.png
500
834
media_image1.png
Greyscale
Therefore, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the given invention to modify Modified Balzer in view of Fujikawa's imaging range upstream of the robot arm's range in order to ensure the workpieces can be accurately tracked by incorporating a calibration that considers the relationship between the visual sensor and the robot: "A relational expression is acquired that converts positional information (a coordinate value (xi, yi) [pixel] of the image coordinate system) on the workpiece measured by the visual sensor 100 into a coordinate value (X, Y) [mm] of the robot coordinate system." [Fujikawa ¶0205], and additionally, so the information can be transmitted prior to the time that the robot has to execute a holding action: "The positional information on each workpiece W is transmitted from the visual sensor 100 to the robot control device 200 through the network NW. Using the received positional information, the robot control device 200 provides to robot 300 the instruction necessary for the holding action" [Fujikawa ¶0068].
Regarding Claim 5 and 16 respectively, Modified Balzer discloses the computer-implemented method of Claim 1 and the computer-readable medium storing instructions of Claim 12.
However, although Balzer discloses that the 3D camera can be mounted above the box (¶0004: a 3D camera, which can be mounted at the end of the robot arm or above the box), Balzer does not explicitly disclose the fixed location. Nevertheless, Fujikawa discloses: wherein the sensor is mounted to a fixed location proximate to the robotic arm (See at least Figure 1 and ¶0054 via "A visual sensor 100 of the first embodiment is provided in a predetermined position above the line 1")
PNG
media_image2.png
490
768
media_image2.png
Greyscale
Therefore, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the given invention to modify Modified Balzer in view of Fujikawa's predetermined sensor position that is proximate to the robotic arm in order to ensure the desired region/range can be sensed: "In the visual sensor 100, an imaging visual field is set so to include the entire width direction (a direction orthogonal to a conveying direction) of the line 1. The imaging visual field can be determined by setting an angle of view (or a view angle) of the imaging unit (camera). In the description, the imaging visual field corresponds to a range where the image can be captured with the imaging unit, and sometimes the imaging visual field is called an 'imaging range'" [Fujikawa ¶0055].
Claims 7 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Balzer et. al. (US 20250028300 A1, corresponding to EP4177013A1 in IDS dated 05/16/2025) and Horowitz et. al. (US 20230191608 A1, IDS 05/16/2025) in view of Saranadi et. al. (US 20240212195 A1).
Regarding Claim 7 and 18 respectively, Modified Balzer discloses the computer-implemented method of Claim 6 and the computer-readable medium storing instructions of Claim 17.
However, Modified Balzer does not explicitly disclose the multi-headed machine learning model.
Nevertheless, Sarandi--who is directed towards a method for training a pose estimator--discloses: wherein the machine learning construct is one head of a multi-headed machine learning model (See at least ¶0150 via " FIG. 2.2 shows a shared backbone 300, and separate prediction heads: a prediction head 313 for training data set 210, a prediction head 323 for training data set 220, and a prediction head 333 for training data set 230.")
Therefore, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the given invention to modify Modified Balzer in view of Saranadi's shared backbone machine learning model in order to improve the accuracy and consistency in pose estimation, which further increases the accuracy of Balazer's pose refinement/ICP algorithm: "Typically, the shared backbone and multiple heads each comprise layers of a neural network. These may be fine-tuned using the autoencoder. The further pose estimator provides more accurate and consistent key point estimates." [Saranadi ¶0186].
Claims 9 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Balzer et. al. (US 20250028300 A1, corresponding to EP4177013A1 in IDS dated 05/16/2025) and Horowitz et. al. (US 20230191608 A1, IDS 05/16/2025) in view of Ryder et. al. (US 20210364629 A1).
Regarding Claim 9 and 20 respectively, Modified Balzer discloses the computer-implemented method of Claim 8 and the computer-readable of Claim 19.
Furthermore, Horowitz discloses: further comprising identifying a second target object in the further image, wherein identifying the second target object is performed (See at least ¶0128 via "Once created, the dynamic model evolves its state over time based on the defined dynamics, and further updates based on new measurements, which is newly sensed data (e.g., new image frames) that is obtained for that object" as well as Figure 16 which illustrates a determination of a target object being 'new' which corresponds to the second object. Additionally, see at least ¶0036 via " In some embodiments, the trajectory associated with the target object is determined based on a plurality of sets of sensed data of the target (e.g., a series of images of the target object as taken from above the conveyor device(s))").
However, Horowitz does not explicitly disclose the second object being identified in parallel with the [first] object's location being tracked.
Nevertheless, Ryder--who discloses object detection and tracking--discloses: identifying the second target object is performed in parallel with tracking the location of the target object (See at least ¶0100 via "In some embodiments, if the candidate object is determined to be a threat or an object of interest, the system may further be configured to use the unique ID assigned to the object during the identification stage to track and monitor the object of interest using the sensor component, while at the same time continuing to identify and scan new candidate objects as described above.")
Therefore, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the given invention to modify Modified Balzer in view of Ryder's simultaneous object identification and object tracking in order to account for newly detected objects as well as currently detected objects, especially in a dynamic or chaotic environment: "…Generating such a model allows the position of an object identified as an object of interest to be tracked before and after classification. This is especially important in a crowded or chaotic environment because occlusions of objects of interest can be overcome…" [Ryder ¶0024].
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KAYLA RENEE DOROS whose telephone number is (703)756-1415. The examiner can normally be reached Generally: M-F (8-5) EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abby Lin can be reached on (571) 270-3976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/K.R.D./Examiner, Art Unit 3657
/ABBY LIN/Supervisory Patent Examiner, Art Unit 3657
Prosecution Insights