Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This Office Action is in response to remarks filed on 8/18/2025.
Claims 1-20 are pending and presented for examination.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 7/24/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, this information disclosure statement is being considered by the examiner.
Response to Amendment
Claims 1-20 have been reviewed based on amendments to claims 1, 4, 7, 12, 14, 16 & 19 and are presented for examination.
Response to Arguments
Applicant’s arguments, see Remarks pages 8-9, filed 8/18/2025, with respect to 35 U.S.C. 101 rejection of claims 12-20 have been fully considered and are persuasive. Based on the amendment to claim 12 to include a host system comprising a set of processors and memory storing instructions that can be executed by the processors, there is sufficient structure to support the functions of claim 12 and dependent claims 13-20. Rejections of claims 12-20 based on 35 U.S.C. 101 have been withdrawn.
Applicant’s arguments, see Remarks pages 8-10, filed 8/18/2025, with respect to the rejections of claims 1-4, 9 & 10 under 35 U.S.C. 102 based on Chen have been fully considered and are persuasive in view of the amendments to claims 1 & 4. Therefore, rejection to these claims under 35 U.S.C. 102 based on Chen have been withdrawn. However, upon further consideration, new grounds of rejection based on 35 U.S.C. 103 are made in view of Mronga et al. (Dennis Mronga & Frank Kirchner, “Learning Context-Adaptive Task Constraints for Robotic Manipulation”, Journal of Robotics and Autonomous Systems, April 14, 2021)(herein after “Mronga”) and Nemallan et al. (US 10828790)(herein after “Nemallan”).
Regarding claims 1 & 4, applicant submits that amendments to these claims traverse the 35 U.S.C. 102 rejections of these claims based on Chen. Examiner agrees and withdraws 35 U.S.C. 102 rejection of claims 1 & 4 based on Chen.
Applicant argues that amendment to claim 1 to include the limitation “based on the component state change, modifying a constraint of a previous calibration specification” is not disclosed by Chen. Examiner agrees, but submits new reference Mronga which teaches “based on the component state change, modifying a constraint of a previous calibration specification” (see fig 2 and section 4.3 & 4.4). Thus, it would have been obvious to combine Chen with the teachings of Mronga to disclose all of the limitations of claim 1. Therefore, examiner withdraws rejection of claim 1 based on 35 U.S.C. 102 based on Chen, but introduces new grounds for rejection of claim 1 based on 35 U.S.C. 103 based on Chen in view of Mronga.
Applicant submits amendments to claim 4 to recite “wherein determining the component state change comprises detection of an added component”. Examiner notes that amendments to claim 4 are not disclosed by Chen, but submits new reference Nemallan which teaches “wherein determining the component state change comprises detection of an added component” (Col 4, lines 19-37). Thus, it would have been obvious to combine Chen in view of Mronga with the further teachings of Nemallan to disclose all of the limitations of claim 4. Therefore, examiner withdraws rejection of claim 4 based on 35 U.S.C. 102 based on Chen, but introduces new grounds for rejection of claim 4 based on 35 U.S.C. 103 based on Chen in view of Mronga, as applied to claim 1, and further in view of Nemallan.
Regarding claims 2, 3, 9 & 10, applicant submits that these claims are not disclosed by Chen given their dependency on claim 1 and amendments to claim 1. Examiner agrees, but has provided above new grounds for rejection of claim 1 based 35 U.S.C. 103 in view of Mronga. Applicant makes no further arguments regarding these claims. Therefore, for the same reasons as discussed above for claim 1, examiner withdraws rejection of these claims based on 35 U.S.C. 102 based on Chen, but introduces new grounds for rejection of these claims based on 35 U.S.C. 103 based on Chen in view of Mronga.
Applicant’s arguments, see Remarks, filed 8/18/2025, with respect to the rejections of claims 5-8 & 11 under 35 U.S.C. 103 have been fully considered and are persuasive in view of the amendments to claim 1. Therefore, rejection to these claims under 35 U.S.C. 103 based on Chen in view of their secondary prior art references have been withdrawn. However, upon further consideration, new grounds of rejection based on 35 U.S.C. 103 are made in view of Mronga et al. (Dennis Mronga & Frank Kirchner, “Learning Context-Adaptive Task Constraints for Robotic Manipulation”, Journal of Robotics and Autonomous Systems, April 14, 2021)(herein after “Mronga”).
Regarding claims 5-8 & 11, applicant submits that these claims are patentable given the amendments to claim 1 for which they depend. Examiner agrees, but has provided above new grounds for rejection of claim 1 based 35 U.S.C. 103 in view of Mronga. Applicant makes no further arguments regarding these claims. Therefore, examiner withdraws rejection of claims 5-6, 8 & 11 based on 35 U.S.C. 103 based on Chen in view of their respective secondary prior art references, but introduces new grounds for rejection of these claims based on 35 U.S.C. 103 based on Chen in view of Mronga and further in view of their respective secondary prior art references.
Applicant's arguments, see Remarks pages 10-11, filed 8/18/2025, with respect to rejection of claim 7 under 35 U.S.C. 103 based on Chen in view of Nandan have been fully considered but they are not persuasive.
Regarding claim 7, applicant submits that it would not be obvious to combine Chen with the teachings of Nandan to form the based for 35 U.S.C. 103 rejection. Examiner respectfully disagrees noting that, per 35 U.S.C. 103, a patent for a claimed invention may not be obtained if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains (see §MPEP 2141).
Applicant argues that the Nandan reference is non-analogous to the field of endeavor, does not read on any of claims 1-20 and in not materially relevant being based on a different technical field that robotic component calibration. Applicant argues that a person of ordinary skill in the art would have no motivation to combine Nandan with Chen and that Nandan contains no conceptual or technical overlap with the remainder of cited prior art or with elements such as a “robot”. Examiner respectfully disagrees noting that analogous art for a 35 U.S.C. 103 reference does not need to be in the same field of endeavor as the claimed invention so long as it is reasonably pertinent to the problem being faced by the inventor (see MPEP 2141.01(a)). Claim 7 pertains to merging a set of component streams into a unified stream and sending the unified stream to a host. Nandan specifically teaches of merging data streams and providing the merged data stream to a network. Therefore, it would have been obvious to someone having ordinary skill in the art to have a set of component streams connected to a robot, as disclosed by Chen, and merge the set of component streams into a single unified component stream, as taught by Nandan.
Based on the above discussion, and based on the new grounds of rejection of claim 1 based on Chen in view of Mronga, examiner withdraws 35 U.S.C. 103 rejection of claim 7 based on Chen in view of Nandan, but introduces new grounds for rejection of claim 7 based on 35 U.S.C. 103 based on Chen in view of Mronga and further in view Nanda.
Applicant's arguments, see Remarks pages 10-11, filed 8/18/2025, with respect to rejection of claims 12-20 under 35 U.S.C. 103 based on Chen in view of Nandan have been fully considered but they are not persuasive.
Regarding claims 12-20, applicant submits that it would not be obvious to combine Chen with the teachings of Nandan to form the based for 35 U.S.C. 103 rejection. Examiner respectfully disagrees noting the same discussion as above for claim 7. For the same reasons as discussed above for claim 7, examiner maintains 35 U.S.C. 103 rejection of claims 12-20 based on Chen in view of Nandan.
Applicant’s arguments, see Remarks pages 8-10, filed 8/18/2025, with respect to the rejections of claims 12-20 under 35 U.S.C. 103 have been fully considered and are persuasive in view of the amendments to claim 12. Therefore, the rejections have been withdrawn. However, upon further consideration, new grounds of rejections based on 35 U.S.C. 103 are made in view of Mronga et al. (Dennis Mronga & Frank Kirchner, “Learning Context-Adaptive Task Constraints for Robotic Manipulation”, Journal of Robotics and Autonomous Systems, April 14, 2021)(herein after “Mronga”).
Regarding claim 12, applicant submits that amendments to this claim traverse the 35 U.S.C. 103 rejection of this claim based on Chen in view of Nanda. Examiner agrees and withdraws 35 U.S.C. 102 rejection of claims 12 based on Chen in view of Nandan.
Applicant argues that amendment to claim 12 to include the limitation “based on a detected component state change, modifying a constraint of a previous calibration specification; and calculating the updated calibration using the modified constraint,” is not disclosed by Chen in view of Nandan. Examiner agrees, but submits new references Mronga which teaches “based on a detected component state change, modifying a constraint of a previous calibration specification; and calculating the updated calibration using the modified constraint” (see fig 2 and section 4.3 & 4.4). Thus, it would have been obvious to combine Chen in view of Nanda with the teachings of Mronga to disclose all of the limitations of claim 12. Based on the above discussion, examiner withdraws 35 U.S.C. 103 rejection of claim 12 based on Chen in view of Nandan, but introduces new grounds for rejection of claim 12 based on 35 U.S.C. 103 based on Chen in view Nandan and further in view of Mronga.
Regarding claims 15-17, 19 & 20, applicant submits that these claims are not disclosed by Chen in view of Nandan given their dependency on claim 12 and amendments to claim 12. Examiner agrees, but has provided above new grounds for rejection of claim 12 based 35 U.S.C. 103 in view of Mronga. Applicant makes no further arguments regarding these claims. Therefore, examiner withdraws rejection of claims 15-17, 19 & 20 based on 35 U.S.C. 103 based on Chen in view Nandan, but introduces new grounds for rejection of these claims based on 35 U.S.C. 103 based on Chen in view of Nandan and further in view of Mronga.
Regarding claims 13, 14 & 18, applicant submits that these claims are patentable given the amendments to claim 12 for which they depend. Examiner agrees, but has provided above new grounds for rejection of claim 12 based 35 U.S.C. 103 in view of Mronga. Applicant makes no further arguments regarding these claims. Therefore, examiner withdraws rejection of claims 13, 14 & 18 based on 35 U.S.C. 103 based on Chen in view Nandan and further in view of their respective other prior art references, but introduces new grounds for rejection of these claims based on 35 U.S.C. 103 based on Chen in view of Nandan and Mronga and further in view of their respective other prior art references.
Claim Interpretation
Claim 12 recites “a stream merge module” followed by functional language, but fails to invoke 112(f) because of being preceded by modifying structure consisting of a set of processors and a memory storing instructions that can be executed by the set of processors to perform the functions of the stream merge module. For the same reasons, the limitations reciting the “system specification module” and “calibration module” do not invoke 112(f).
Claims 13-14 recite “a user interface”, “ a serializer” and a “message transport protocol” followed by functional language, but fail to invoke 112(f) because of being dependent on claim 12 and thus preceded by modifying structure consisting of a set of processors and a memory storing instructions that can be executed by the set of processors to perform the functions of the “user interface”, “ a serializer” and a “message transport protocol” respectively.
Claims 1-4, 9 & 10 are rejected under pre-AIA 35 U.S.C. 102(a)(1) as being anticipated by Chen et al. (“Heterogeneous Multi-sensor Calibration based on Graph Optimization”, Hongyu Chen & Soren Schwertfeger, Proceedings of the 2019 IEEE International Conference on Real-time Computing and Robotics, Pages 158-163, 8/4/2019)(herein after “Chen”).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under pre-AIA 35 U.S.C. 103(a) are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims under pre-AIA 35 U.S.C. 103(a), the examiner presumes that the subject matter of the various claims was commonly owned at the time any inventions covered therein were made absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and invention dates of each claim that was not commonly owned at the time a later invention was made in order for the examiner to consider the applicability of pre-AIA 35 U.S.C. 103(c) and potential pre-AIA 35 U.S.C. 102(e), (f) or (g) prior art under pre-AIA 35 U.S.C. 103(a).
Claims 1-3, 9 & 10 rejected under pre-AIA 35 U.S.C. 103(a) as being unpatentable over Chen et al. (“Heterogeneous Multi-sensor Calibration based on Graph Optimization”, Hongyu Chen & Soren Schwertfeger, Proceedings of the 2019 IEEE International Conference on Real-time Computing and Robotics, Pages 158-163, 8/4/2019)(herein after “Chen”) in view of Mronga et al. (Dennis Mronga & Frank Kirchner, “Learning Context-Adaptive Task Constraints for Robotic Manipulation”, Journal of Robotics and Autonomous Systems, April 14, 2021)(herein after “Mronga”).
Regarding Claim 1, Chen discloses a method, comprising: providing a set of component streams, each generated by a component of a set of components connected to a robot (Fig 1 & Section IV-B, 1st paragraph disclose data collection each camera from a set of 9 cameras that are connected to a MARS Mapper robot.),
to a host system of the robot (Section II, 3rd paragraph discloses a single Intel i7 CPU used as a host system to collect data from 9 cameras, 2 lidars, an IMU and odometry data.);
determining a component state change within the set of components (Fig 1 & Section IV-B, 2nd & 3rd paragraph disclose the MARS Mapper moving around which would lead to a state change in the images captured by the 9 cameras, 2 lidars, the IMU and the odometry data.);
responsive to the component state change: determining a calibration specification for each component of the set of components (Fig. 1 & Section III-A disclose calibration specifications for components including Stereo Cameras, Non-overlapping cameras, 3D Lidar to camera, 3D Lidar to 3D Lidar and Tracking System to camera.);
determining an updated calibration for the robot based on the set of calibration specifications (Fig 1 & Sections III-B, 3rd to 6th paragraphs disclose a method for performing global calibration for the MARS Mapper robot based on a set of calibration specifications from a plurality of sensors.);
and providing the updated calibration to the host system (Table 1 and Section IV-B, 3rd paragraph disclose accuracy results based on global calibration updates being provided to the single Intel i7 CPU on the MARS Mapper robot. Section IV-B, 2nd paragraph discloses an Optitrack tracking system for providing global calibration updates to the single Intel i7 CPU on the MARS Mapper robot.)
wherein the host system processes subsequent component streams from the set of components using the updated calibration (Table 1 and Section IV-B, 2nd & 3rd paragraph disclose accuracy results from the MARS Mapper robot using the single Intel i7 CPU to process subsequent camera data from a set of 9 cameras using the global calibration updates from the Optitrack tracking system.).
Chen fails to disclose based on the component state change, modifying a constraint of a previous calibration specification.
However, Mronga teaches based on the component state change, modifying a constraint of a previous calibration specification (Page 14, Fig 2 & Section 4.3 discloses a constrained-based control framework (i.e. calibration specification) based on task constraints x(t) and v(t) that are provided by a learning module for a given context [Symbol font/0x6B]. Pages 17-18, Section 4.4 discloses modifying task weights of constraints based on context changes (i.e. state changes).).
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have a method that, responsive to the component state change, determines a calibration specification for each component of the set of components, as disclosed by Chen, wherein the determined calibration specification is based on modifying a constraint of a previous calibration specification, as taught by Mronga. The motivation to do so would be to have a method where a constrained-based control system used to calibrate a robot can modify constraints associated with the constrained-based control system in order to perform varying contexts that change the task that the robot is to perform.
Regarding Claim 2, Chen in view of Mronga disclose the method of claim 1.
Chen disclose wherein the updated calibration is determined contemporaneously with robot operation (Fig. 1 & Section IV-B, 2nd & 3rd paragraphs disclose a MARS Mapping robot moving around while the Optitrack tracking system is simultaneously determining and providing global calibration updates.).
Regarding Claim 3, Chen in view of Mronga disclose the method of claim 1.
Chen discloses wherein the calibration specification for each component identifies an intrinsic submodel from a set of candidate intrinsic submodels for the respective component (Fig 1 & Section IV, 1st paragraph discloses a MARS Mapper robot using intrinsic parameters from component cameras. Section III-A disclose a set of calibration submodels for Stereo-Camera, Non-Overlapping Camera, 3D Lidar to Camera and 3D Lidar to 3D Lidar calibration using the intrinsic parameters from the component cameras.),
wherein the identified intrinsic submodel is used to determine intrinsic calibration equations for the respective component (Section III-A, paragraphs 4-8 disclose intrinsic calibration equations for non-overlapping camera pair components.),
wherein the intrinsic calibration equations are included in a calibration set equation set used to determine the updated calibration (Fig 2 & Section III-B, 4th to 6th paragraphs disclose use of component pairwise calibration equations to perform equations for global calibration updates.).
Regarding Claim 9, Chen in view of Mronga disclose the method of claim 1.
Chen discloses wherein providing the set of component streams, determining the component state change, and determining the updated calibration are processes that are atomic, execute asynchronously, and are non-blocking (Section II, 3rd paragraph & Section IV-B disclose real data results from experiments with a MARS Mapper robot, moving around leading to state changes, with 9 camera components and an Optitrack tracking system remotely fixed to a ceiling providing global calibration updates. These results demonstrate the performance of component image streams from 9 cameras, based on state changes from a moving robot, and global calibration updates from a tracking system, all operating in an atomic, asynchronous and non-blocking fashion.).
Regarding Claim 10, Chen in view of Mronga disclose the method of claim 1.
Chen discloses wherein each component comprises a sensor package (Fig 1 and Section II, 1st & 2nd paragraphs disclose a MARS Mapper robot is a sensor platform where each component is mounted to the MARS Mapper robot platform as a sensor.).
Claim 4 rejected under pre-AIA 35 U.S.C. 103(a) as being unpatentable over Chen et al. (“Heterogeneous Multi-sensor Calibration based on Graph Optimization”, Hongyu Chen & Soren Schwertfeger, Proceedings of the 2019 IEEE International Conference on Real-time Computing and Robotics, Pages 158-163, 8/4/2019)(herein after “Chen”) in view of Mronga et al. (Dennis Mronga & Frank Kirchner, “Learning Context-Adaptive Task Constraints for Robotic Manipulation”, Journal of Robotics and Autonomous Systems, April 14, 2021)(herein after “Mronga”), as applied to claim 1, and further in view of Nemallan et al. (US 10828790)(herein after “Nemallan”).
Regarding Claim 4, Chen in view of Mronga disclose the method of claim 1.
Chen discloses an updated calibration is determined based on a calibration specification for an added component (Section III-B, 2nd to 6th paragraphs discloses a global calibration based on the additional Sensor C (and in general any number of additional sensors).).
Chen fails to discloses wherein determining the component state change comprises detection of an added component.
However, Nemallan further teaches wherein determining the component state change comprises detection of an added component (Col 4, lines 19-37 disclose a system that can perform object feature recognition (i.e. detection) to changes in components used to complete a task for a robot. An added component would be detected as a change in components (i.e. a state change).).
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have the method of claim 1 wherein updated calibration is determined based on a calibration specification for an added component, as disclosed by Chen in view of Mronga, wherein determining the component state change comprises detection of an added component, as taught by Nemallan. The motivation to do so would be to have a method where a robot can detect the addition of a component in performing a new task, such as detecting a new screw and screw hole for assembling a product, and automatically update its calibration specification without having to be re-programmed or re-configured to perform the new task.
Claim 5 rejected under pre-AIA 35 U.S.C. 103(a) as being unpatentable over Chen et al. (“Heterogeneous Multi-sensor Calibration based on Graph Optimization”, Hongyu Chen & Soren Schwertfeger, IEEE International Conference on Real-time Computing and Robotics (RCAR), 8/24/2019)(herein after “Chen”) in view of Mronga et al. (Dennis Mronga & Frank Kirchner, “Learning Context-Adaptive Task Constraints for Robotic Manipulation”, Journal of Robotics and Autonomous Systems, April 14, 2021)(herein after “Mronga”), as applied to claim 1, and further in view of Hsu et al. (US 2015/0002194)(herein after “Hsu”).
Regarding Claim 5, Chen in view of Mronga discloses the method of Claim 1.
Chen fails to disclose wherein each calibration specification is received from a driver for the respective component.
However, Hsu further teaches wherein each calibration specification is received from a driver for the respective component (Fig 1 & [0015] disclose calibration signals received by receiver comparators 121 & 122 from a driver for each respective component (driver 111 for receiver comparator 121 and driver 112 for receiver comparator 122).
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have the method of Claim 1, as disclosed by Chen in view of Mronga, wherein each calibration specification is received from a driver for the respective component as further taught by Hsu. The motivation to do so would be to enable sending of a plurality of calibration signals with individually adjustable signal strengths.
Claim 6 rejected under pre-AIA 35 U.S.C. 103(a) as being unpatentable over Chen et al. (“Heterogeneous Multi-sensor Calibration based on Graph Optimization”, Hongyu Chen & Soren Schwertfeger, IEEE International Conference on Real-time Computing and Robotics (RCAR), 8/24/2019)(herein after “Chen”) in view of Mronga et al. (Dennis Mronga & Frank Kirchner, “Learning Context-Adaptive Task Constraints for Robotic Manipulation”, Journal of Robotics and Autonomous Systems, April 14, 2021)(herein after “Mronga”), as applied to claim 1, and further in view of Zinkovsky et al. (US 2012/0151452)(herein after “Zinkovsky”).
Regarding Claim 6, Chen in view of Mronga discloses the method of Claim 1.
Chen fails to disclose wherein the set of component streams is provided to the host system using a serializer and a message transport protocol, wherein the serializer and the message transport protocol are selected from a set of supported serializers and message transport protocols, respectively.
However, Zinkovsky further teaches wherein the set of component streams is provided to the host system using a serializer (Fig 1 & [0026] disclose a set of component streams 118a to 118n from a plurality of user sessions 104 to 108 provided to a network 150 using a serializer 120.] and
a message transport protocol (Fig 1 & [0029] disclose a network transport protocol 122.),
wherein the serializer and the message transport protocol are selected from a set of supported serializers and message transport protocols, respectively (Fig 1 & [0026] disclose that different serializers can be accommodated for different processes. Fig 1 & [0029] discloses that network transport protocols may be selected from either custom or standard network transport protocols such as TCP, UDP, HTTP or SOAP/REST.).
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have the method of Claim 1, as disclosed by Chen in view of Mronga, wherein the set of component streams is provided to the host system using a serializer and a message transport protocol, wherein the serializer and the message transport protocol are selected from a set of supported serializers and message transport protocols, respectively, as further taught by Zinkovsky. The motivation to do so would be to enable simple, reliable and ordered data transmission between a set of user sessions and a network.
Claim 7 rejected under pre-AIA 35 U.S.C. 103(a) as being unpatentable over Chen et al. (“Heterogeneous Multi-sensor Calibration based on Graph Optimization”, Hongyu Chen & Soren Schwertfeger, IEEE International Conference on Real-time Computing and Robotics (RCAR), 8/24/2019)(herein after “Chen”) in view of Mronga et al. (Dennis Mronga & Frank Kirchner, “Learning Context-Adaptive Task Constraints for Robotic Manipulation”, Journal of Robotics and Autonomous Systems, April 14, 2021)(herein after “Mronga”), as applied to claim 1, and further in view of Zandan et al. (“How Merging Companies Will Give Rise to Unified Data Streams”, https://www.confluent.io/blog/merging-data-streams/ , Naveen Nandan, 6/23/2020)(herein after “Zandan”).
Regarding Claim 7, Chen in view of Mronga discloses the method of Claim 1.
Chen fails to disclose wherein the set of component streams are merged into a unified stream, wherein providing the set of component streams to the host system comprises sending the unified stream to the host system.
However, Nandan further teaches wherein the set of component streams are merged into a unified stream (“Why unify data streams?” section, 2nd figure & paragraphs 4-6 disclose merging data streams from 3 organizations into a unified stream.),
wherein providing the component stream to the host system comprises sending the unified stream to the host system (“Why unify data streams?” section, 2nd figure & paragraphs 4-6 disclose providing a merged data stream to a network for a merged set of organizations.).
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have the method of Claim 1, as disclosed by Chen in view of Mronga, wherein the set of component streams are merged into a unified stream, wherein providing the component stream to the host system comprises sending the unified stream to the host system, as further taught by Nandan. The motivation to do so would be to have a system for robot data stream management that merges streams of camera data, from a set of 9 different cameras, into a single stream that can be used to perform a global calibration across the 9 cameras to avoid error propagation in performing successive pairwise calibration across the 9 cameras.
Claim 8 rejected under pre-AIA 35 U.S.C. 103(a) as being unpatentable over Chen et al. (“Heterogeneous Multi-sensor Calibration based on Graph Optimization”, Hongyu Chen & Soren Schwertfeger, IEEE International Conference on Real-time Computing and Robotics (RCAR), 8/24/2019)(herein after “Chen”) in view of Mronga et al. (Dennis Mronga & Frank Kirchner, “Learning Context-Adaptive Task Constraints for Robotic Manipulation”, Journal of Robotics and Autonomous Systems, April 14, 2021)(herein after “Mronga”), as applied to claim 1, and further in view of Memon et al. (US 2019/0012197)(herein after “Memon”).
Regarding Claim 8, Chen in view of Mronga discloses the method of Claim 1.
Chen fails to disclose wherein the host system interacts with observations within a component stream of the set of component streams using an API specific to the respective component.
However, Memon further teaches wherein the host system interacts with observations within a component stream of the set of component streams using an API specific to the respective component (Fig 1 & [0022] disclose a host system that interacts with applications through respective APIs.).
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have the method of Claim 1, as disclosed by Chen in view of Mronga, wherein the host system interacts with observations within a component stream of the set of component streams using an API specific to the respective component, as further taught by Memon. The motivation to do so would be to simplify how application software components interact host system hardware components.
Claim 11 rejected under pre-AIA 35 U.S.C. 103(a) as being unpatentable over Chen et al. (“Heterogeneous Multi-sensor Calibration based on Graph Optimization”, Hongyu Chen & Soren Schwertfeger, IEEE International Conference on Real-time Computing and Robotics (RCAR), 8/24/2019)(herein after “Chen”) in view of Mronga et al. (Dennis Mronga & Frank Kirchner, “Learning Context-Adaptive Task Constraints for Robotic Manipulation”, Journal of Robotics and Autonomous Systems, April 14, 2021)(herein after “Mronga”), as applied to claim 1, and further in view of Cella et al. (US 2019/0041840)(herein after “Cella”).
Regarding Claim 11, Chen in view of Mronga discloses the method of Claim 1.
Chen fails to disclose further comprising a user interface, configured to: present device events emitted by drivers of the set of components to a user; and receive user-specified stream configurations that are sent to a driver for a component of the set of components.
However, Cella further teaches further comprising a user interface, configured to: present device events emitted by drivers of the set of components to a user (Fig 151 & [1177-1178] disclose a haptic user interface device for providing haptic stimuli to a user based on data collected in an industrial environment. Fig 23 & [0062] disclose that a data collection architecture may include driver APIs to facilitate communication in a network.);
and receive user-specified stream configurations that are sent to a driver for a component of the set of components (Fig 151 & [1177-1178] disclose the haptic user interface device may receive results from analysis of data collected from a plurality of sensors. Fig 23 & [0062] disclose that a data collection architecture may include driver APIs to facilitate communication in a network.).
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have the method of Claim 1, as disclosed by Chen in view of Mronga, further comprising a user interface, configured to: present device events emitted by drivers of the set of components to a user; and receive user-specified stream configurations that are sent to a driver for a component of the set of components, as further taught by Cella. The motivation to do so would be to provide physical or audible feedback and stimuli to a user based on data collection and analysis from a surrounding sensor network.
Claim 12 rejected under pre-AIA 35 U.S.C. 103(a) as being unpatentable over Chen et al. (“Heterogeneous Multi-sensor Calibration based on Graph Optimization”, Hongyu Chen & Soren Schwertfeger, IEEE International Conference on Real-time Computing and Robotics (RCAR), 8/24/2019)(herein after “Chen”) in view of Zandan et al. (“How Merging Companies Will Give Rise to Unified Data Streams”, https://www.confluent.io/blog/merging-data-streams/ , Naveen Nandan, 6/23/2020)(herein after “Zandan”) and further in view of Mronga et al. (Dennis Mronga & Frank Kirchner, “Learning Context-Adaptive Task Constraints for Robotic Manipulation”, Journal of Robotics and Autonomous Systems, April 14, 2021)(herein after “Mronga”).
Regarding Claim 12, Chen discloses a system for robot data stream management (Fig 1 & Section IV-B, 1st & 2nd paragraphs disclose data collection from 9 cameras that are connected to a MARS Mapper robot, and management of the 9 cameras through calibration.), comprising:
a host system comprising a set of processors and a memory storing instructions that, when executed by the set of processors, cause the set of processors to run each of (Section II, 3rd paragraph discloses custom micro-processors and an Intel i7 CPU (i.e. a host system) that is able to gather and compress images from 9 cameras (i.e. by executing instructions stored in memory):
a system specification module, configured to determine a set of calibration specifications for a set of components connected to the robot (Section III-A disclose calibration specifications for components including Stereo Cameras, Non-overlapping cameras, 3D Lidar to camera, 3D Lidar to 3D Lidar and Tracking System to camera. Fig 1 & Section IV-B, 1st & 2nd paragraphs disclose data collection of calibration specifications from 9 cameras that are connected to the MARS Mapper robot.);
a set of component data streams, each generated by a different component connected to a robot; and (Fig 1 & Section IV-B, 1st & 2nd paragraphs disclose data collection from data generated from 9 different cameras that are connected to a MARS Mapper robot.)
a calibration module, configured to determine an updated calibration based on the set of calibration specifications (Section IV-B discloses use of an Optitrack tracking system that determines calibration for the 9 cameras.),
wherein the robot processes subsequent component streams from the set of connected components using the updated calibration (Section II, 2nd & 3rd paragraphs disclose a single Intel i7 CPU used as a host processor for a MARS Mapper robot. Table 1 and Section IV-B, 3rd paragraph disclose accuracy results from the MARS Mapper robot using the single Intel i7 CPU to process subsequent camera data from a set of cameras using global calibration.).
Chen fails to discloses a stream merge module configured to merge a set of component data streams into a unified stream; and wherein the calibration module determines an updated calibration by: based on a detected component state change, modifying a constraint of the set of calibration specifications; and calculating the updated calibration using the modified constraint.
However, Nandan teaches a stream merge module configured to merge a set of component data streams, into a unified stream; (“Why unify data streams?” section, 2nd figure & paragraphs 4-6 disclose an Apache Kafka® platform for merging data streams from 3 organizations into a unified stream.).
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have a system for robot data stream management, comprising: a system specification module, configured to determine a set of calibration specifications for a set of components connected to the robot; a set of component data streams, each generated by a different component connected to a robot; and a calibration module, configured to determine an updated calibration based on the set of calibration specifications, wherein a host system of the robot processes subsequent component streams from the set of connected components using the updated calibration, as disclosed by Chen, further comprising: a stream merge module configured to merge a set of component data streams into a unified stream, as taught by Nandan. The motivation to do so would be to have a system for robot data stream management that merges streams of camera data, from a set of 9 different cameras, into a single stream that can be used to perform a global calibration across the 9 cameras to avoid error propagation in performing successive pairwise calibration across the 9 cameras.
Mronga further teaches wherein the calibration module determines an updated calibration by: based on a detected component state change, modifying a constraint of the set of calibration specifications (Page 14, Fig 2 & Section 4.3 discloses a constrained-based control framework (i.e. calibration specification) based on task constraints x(t) and v(t) that are provided by a learning module for a given context [Symbol font/0x6B]. Pages 17-18, Section 4.4 discloses modifying task weights of constraints based on context changes (i.e. detecting a state change).);
and calculating the updated calibration using the modified constraint (Page 14, fig 2 & Section 4.3 discloses the constrained-based control framework computes joint velocities for robot joints based on estimated task constraints that are fed into the constrained-based control framework).
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to a have system for robot data stream management configured to determine an updated calibration based on the set of calibration specifications, as disclosed by Chen, the updated calibration is based on detecting a component state change and modifying a constraint of the set of calibration specifications; and calculating the updated calibration using the modified constraint, as further taught by Mronga. The motivation to do so would be to have a system for robot data stream management where a constrained-based control system used to calibrate a robot can detect a state change and modify constraints associated with the constrained-based control system in order to perform varying contexts that change the task that the robot is to perform.
Claims 15-17, 19 & 20 rejected under pre-AIA 35 U.S.C. 103(a) as being unpatentable over Chen et al. (“Heterogeneous Multi-sensor Calibration based on Graph Optimization”, Hongyu Chen & Soren Schwertfeger, IEEE International Conference on Real-time Computing and Robotics (RCAR), 8/24/2019)(herein after “Chen”) in view of Zandan et al. (“How Merging Companies Will Give Rise to Unified Data Streams”, https://www.confluent.io/blog/merging-data-streams/ , Naveen Nandan, 6/23/2020)(herein after “Zandan”), as applied to claim 12, and further in view of Mronga et al. (Dennis Mronga & Frank Kirchner, “Learning Context-Adaptive Task Constraints for Robotic Manipulation”, Journal of Robotics and Autonomous Systems, April 14, 2021)(herein after “Mronga”).
Regarding Claim 15, Chen in view of Zandan and further in view of Mronga disclose the system of Claim 12.
Chen further discloses wherein each calibration specification identifies an intrinsic submodel for each of a set of intrinsic parameters for the respective component (Fig 1 & Section IV, 1st paragraph discloses a MARS Mapper robot using intrinsic parameters from component cameras.),
wherein each intrinsic submodel is selected from a set of candidate intrinsic submodels for the respective intrinsic parameter (Section III-A discloses intrinsic submodels selected for different pairwise calibration parameters including stereo-camera, non-overlapping camera pairs, 3D Lidar to camera and 3D Lidar to 3D lidar parameters.).
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have the system of Claim 12, as disclosed by Chen in view of Zandan and further in view of Mronga, wherein each calibration specification identifies an intrinsic submodel for each of a set of intrinsic parameters for the respective component, wherein each intrinsic submodel is selected from a set of candidate intrinsic submodels for the respective intrinsic parameter, as further taught by Chen. The motivation to do so would be leverage internal characteristics of cameras in performing calibration.
Regarding Claim 16, Chen in view of Zandan and further in view of Mronga disclose the system of Claim 15.
Chen further discloses wherein the calibration module uses equations associated with the identified intrinsic submodels when determining the updated calibration specification (Section III-A, paragraphs 4-8 disclose intrinsic calibration equations for a non-overlapping camera pair component. Fig 2 & Section III-B, 3rd paragraph disclose use of component pairwise calibration results to perform a global calibration.).
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have the system of Claim 15, as disclosed by Chen in view of Zandan and further in view of Mronga, wherein the calibration module uses equations associated with the identified intrinsic submodels when determining the updated calibration, as further disclosed by Chen. The motivation to do so would be to leverage internal characteristics of cameras in determining a global calibration.
Regarding Claim 17, Chen in view of Zandan and further in view of Mronga disclose the system of Claim 12.
Chen further discloses wherein the component comprises a sensor package (Fig 1 and Section II, 1st & 2nd paragraphs disclose a MARS Mapper robot is a sensor platform where each component is mounted to the MARS Mapper robot platform as a sensor.).
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have the system of Claim 12, as disclosed by Chen in view of Zandan and further in view of Mronga, wherein the component comprises a sensor package, as further disclosed by Chen. The motivation to do so would be include camera sensors as part of a robot system.
Regarding Claim 19, Chen in view of Zandan and further in view of Mronga disclose the system of Claim 12.
Chen further discloses wherein the calibration module is remote from the system specification module and a subset of the set of processors of the host system (Section IV-B, 2nd paragraph discloses an Optitrack tracking system fixed to a truss on a ceiling, remote from a robot, that performs calibration for 9 cameras on the robot. Fig 1, Section II, 2nd & 3rd paragraphs and Section III-A disclose a sensor platform that is part of a MARS Mapper robot, remote from the ceiling, that includes a single Intel i7 CPU used as a host system to collect data and provide calibration specifications for 9 cameras.).
Zandan further teaches wherein the calibration module is remote from the stream merge module (“Why unify data streams?” section, 2nd figure & paragraphs 4-6 disclose an Apache Kafka® platform for merging data streams from 3 organizations into a unified stream. The Apache Kafka® platform could be at any location remote from a calibration module.).
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have the system of Claim 12, as disclosed by Chen in view of Zandan and further in view of Mronga, wherein the calibration module is remote from the stream merge module, as further taught by Zandan, and the system specification module and the host system, as further disclosed by Chen. The motivation to do so would be to have a tracking system with a plurality of cameras located remotely from a robot that can wirelessly perform global calibration for the robot.
Regarding Claim 20, Chen in view of Zandan and further in view of Mronga discloses the system of Claim 12.
Chen further discloses wherein the stream merge module, the system specification module, and the calibration module are atomic processes and execute asynchronously. (Section II, 3rd paragraph & Section IV-B disclose real data results from experiments with a MARS Mapper robot with 9 camera components providing system specifications and an Optitrack tracking system remotely fixed to a ceiling providing global calibration updates. These results demonstrate the performance of modules providing system specifications from 9 cameras, that could incorporate an Apache Kafka® stream merge module, and a module for performing global calibration updates from a tracking system all operating in an atomic, asynchronous and non-blocking fashion.).
Claim 13 rejected under pre-AIA 35 U.S.C. 103(a) as being unpatentable over Chen et al. (“Heterogeneous Multi-sensor Calibration based on Graph Optimization”, Hongyu Chen & Soren Schwertfeger, IEEE International Conference on Real-time Computing and Robotics (RCAR), 8/24/2019)(herein after “Chen”) in view of Zandan et al. (https://www.confluent.io/blog/merging-data-streams/ , Naveen Nandan, 6/23/2020)(herein after “Zandan”) and Mronga et al. (Dennis Mronga & Frank Kirchner, “Learning Context-Adaptive Task Constraints for Robotic Manipulation”, Journal of Robotics and Autonomous Systems, April 14, 2021)(herein after “Mronga”) and further in view of Cella et al. (US 2019/0041840)(herein after “Cella”).
Regarding Claim 13, Chen in view of Zandan and Mronga discloses the system of Claim 12.
Chen in view of Zandan and Mronga fails to disclose further comprising a user interface, configured to: present device events emitted by a driver for a component of the set of components to a user; and receive user-specified stream configurations that are sent to a driver for a component of the set of components.
However, Cella further teaches further comprising a user interface, configured to: present device events emitted by a driver for a component of the set of components to a user (Fig 151 & [1177-1178] disclose a haptic user interface device for providing haptic stimuli to a user based on data collected in an industrial environment. Fig 23 & [0062] disclose that a data collection architecture may include driver APIs to facilitate communication in a network.);
and receive user-specified stream configurations that are sent to a driver for a component of the set of components (Fig 151 & [1177-1178] disclose the haptic user interface device may receive results from analysis of data collected from a plurality of sensors. Fig 23 & [0062] disclose that a data collection architecture may include driver APIs to facilitate communication in a network.).
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have the method of Claim 12, as disclosed by Chen in view of Nandan and Mronga, further comprising a user interface, configured to: present device events emitted by drivers of the set of components to a user; and receive user-specified stream configurations that are sent to a driver for a component of the set of components, as further taught by Cella. The motivation to do so would be to provide physical or audible feedback and stimuli to a user based on data collection and analysis from a surrounding sensor network.
Claim 14 rejected