DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 5/12/2026 was in compliance with the provisions of 37 CFR 1.97. Accordingly, the IDS is being considered by the examiner.
Response to Amendment
Claims 2-6, 8-10, 12-14, and 16-17 are amended, claims 1, 7, and 15 are cancelled, and claims 18-22 are new. Claims 2-6, 8-14, and 16-22 are pending.
Response to Arguments
Applicant's arguments filed 5/12/2026 have been fully considered.
Regarding the objections to claims 2, 6, 9-10, 14, and 17, as noted by Applicant on pages 9-10 of the response, the claims amendments overcome the objections, which are withdrawn.
Regarding the objections to the drawings, as noted by Applicant on page 10 of the response, the amendments to the drawings overcome the objections, which are withdrawn.
Regarding the rejections of independent claims 2 and 10 under 101 as being directed to a judicial exception without significantly more, Examiner respectfully disagrees with Applicant’s arguments on pages 13-21 that the amendments to claims 2 and 10 overcome the rejections for the following reasons.
On page 12 of the response, Applicant contends that claim 2 is patent eligible under the streamlined analysis set forth by MPEP 2106.06 because claim 2 improves computer functionality such as by remediating abnormal movement of a robot. In support, Applicant cites portions of the specification describing remedial actions that may be performed based on various sensor imaging data and associates such description with elements recited in claim 2 such as use of “a first series of RGB data” “a second series of spatial distance data,” and the steps of “fusing together, by the processor, the first series of RGB image data and the second series of spatial distance data” and “generating … based on the fusion of the first series and the second series, a remediation instruction for execution by the robot traversing the environment.”
Examiner submits that no combination of the elements constitutes an improvement to computer technology because the method includes no particularized/specialized processing techniques. Instead, the processing entails receiving collecting data that have been sensed by sensor activity falling outside the functions positively recited as being part of the method, and implementing mathematical algorithms (e.g., SLAM) to fuse RGB and spatial distance imaging data, which constitutes insignificant extra solution activity. Regarding generating the remediation instruction, Examiner notes that claim 2 does not appear to convey any specialized processing exception that the output of the fusion (e.g., overlayed data via SLAM) is evaluated to determine how to remediate the robot in some manner. Using combined/fused RGB and spatial distance data for producing enhanced imaging is very well known as evidenced by the Ligochi and Kueny references such that the claim considered as a whole does not appear to constitute an improvement to the functioning of a computer.
Regarding Step 2A Prong 1, on pages 15-16 of the response, Applicant contends that the step of “fusing together … the first series of RGB image data and the second series of spatial distance data” does not fall within the mathematical concepts exception. In support, Applicant asserts that this element does not recite a mathematical relation because it does not “recite a relationship between variables or numbers” or “a numerical formula or equation” or “a mathematical operation … or an act of calculating using mathematical methods to determine a variable or number.” Applicant further asserts that at most, the subject matter may be considered as “only based on or involve[ing] a mathematical concept.”
Examiner submits that while claim 2 (and similarly claims 10 and 22) does not expressly recite a specific mathematical equation/formula, the function of fusing RGB image data and spatial distance data as described in Applicant’s specification (describing the respective 2D and 3D coordinate systems of the respective imaging data and SLAM sensor fusion) clearly entails an overlaying of corresponding image data, that inherently entails aligning of coordinates/geometries between the RGB and spatial distance images that requires mathematical relations/calculations as fundamental, rather than being merely involved, to such alignment/fusion. Therefore, Examiner submits that the “fusing together” step falls within the mathematical relations subcategory of the mathematical concepts exception.
Regarding Step 2 Prong 2, on pages 18-20 Applicant contends that claim 2 embodies technical improvements to the computer-related field of remediating abnormal movement of a robotic system and in relation thereto, that “fusing together, by the processor, the first series of RGB image data and the second series of spatial distance data” and “generating, by the processor, and based on the fusion of the first series and the second series, a remediation instruction for execution by the robot traversing the environment” is not merely post-solution activity. In support, Applicant asserts on page 20 that claim 2 integrates any judicial exception into a practical application because “… the ‘remediation instruction for execution by the robot traversing the environment’ is generated using the disclosed improvement, e.g., ‘based on the fusion of the first series [of RGB image data generated by an RGB sensor deployed on a robot traversing an environment’ and the second series [of spatial distance data generated by a spatial distance sensor deployed on the robot traversing the environment].”
Examiner submits that individually and in combination the steps of “fusing together” and “generating” “a remediation instruction” fall within the abstract idea exception. The additional elements “receiving” “a first series of RGB image data generated by an RGB sensor deployed on a robot” and “receiving” “a second series of spatial distance data generated by a spatial distance sensor deployed on the robot” constitute known (e.g., Ligocki and Kueny) and importantly sources of data used by robotic systems that do not appear to play a particularized functional role with respect to the manner of determining or the nature of the broadly claimed “remediation instruction,” such that no integration of the judicial exception such as in terms of improving any particular technology such as computer technology is apparent from the use of such fused data for generating some form of remediation instruction.
Regarding Step 2B, Applicant contends on pages 20-21 that the claims amount to significantly more than any alleged judicial exception, instead reciting a “specific way of remediating abnormal movement of a robotic system.” In support, Applicant asserts on page 21 that the claimed use of fused RGB image data and spatial distance data to determine a remediation instruction “does not tie up every way that one may remediate abnormal movement of a robotic system” and “does not seek a monopoly over an alleged abstract idea and in no manner seeks to ‘wholly pre-empt’ all manners” in which such may be performed.
Examiner acknowledges that the use of fusing the particular combination of RGB image data and spatial distance data provides a limitation to generating a remedial instruction. However, this combination of imaging data types itself is recited at a high level of generality (i.e., fusing/combining the two imaging types) that is commonly used for robotic imaging and control (e.g., direction, mapping) as disclosed by the Ligocki and Kueny references. Related to this generality in the nature of the fused data is the lack of any particularized functional relation between the fused data and the nature of the remediation instruction or manner of determining the remediation instruction, such that claim 2 effectively entails recites a broad application of the abstract idea in the field of multi-sensor robotic imaging and control that does not appear to amount to significantly more than the abstract idea.
Regarding the rejections of independent claims 2 and 10 under 103, Examiner agrees with Applicant’s arguments on pages 22-23 that the amendments overcome the rejections. Therefore, the previous grounds for rejecting claims 2 and 10 under 103 are withdrawn. However, based on further search and analysis new grounds for rejecting claims 2 and 10 under 103 are set forth herein.
Claim Objections
Claims 18 and 21 are objected to because of the following informalities:
In claim 18 line 3, “based on the of the first series” should read “based on the first series.”
In claim 21 line 2, “wherein robot traversing the environment comprises robot” should read “wherein the robot traversing the environment comprises the robot” for clarity purposes.
In claim 21 lines 5-6, “comprises at least one of at least one of” should read “comprises at least one of.”
In each of independent claims 1, 10, and 22, the first instance of RGB should be spelled out to clearly indicate the meaning of the acronym.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
Claims 19-21 are rejected under 35 U.S.C. 112(a) as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor had possession of the claimed invention.
Claim 19 recites “wherein generating the remediation instruction comprises generating a remediation instruction to cause the robot to adjust a motor of a wheel of the robot or a body part of the robot,” in which per antecedent relation with claim 2 “the remedial instruction” is “based on the fusion of the first series and the second series” in which the first series is “RGB image data” and the second series is “spatial distance data generated by a spatial distance sensor.” Applicant’s original disclosure does not disclose, with reasonable clarity, that the inventors had possession of claimed invention including this combination of features.
Applicant’s original disclosure discloses generating a remediation instruction to cause the robot to adjust a motor of a wheel of the robot or a body part of the robot such as in [0142] of the specification. In [0142]-[0143], the remediation instruction (motor instruction) is described as having been generated by sensor data including data received from an IMU and/or encoder and in some cases from two or more sensors. Neither in [0142]-[0143] nor elsewhere does Applicant’s original disclosure appear to teach that the particularized sensor processing configuration entailing a fusion of RGB image data and spatial distance data generated by a spatial distance sensor is used as a basis for generating a remediation instruction to cause the robot to adjust a motor of a wheel of the robot or a body part of the robot. Using a fusion of RGB image data and spatial distance data generated by a spatial distance sensor for generating may at least inferentially considered to be a remediation instruction is described in Applicant’s specification in [0076], which describes a “calibration transformation” that is implemented based on a fusion/combination of IR image data mapped into 3D space and RGB image data mapped into 3D space, and which entails accounting for detected differences between the respective encoders and IMUs accounting for the different frames of reference of the IR and RGB cameras. Paragraph [0057] describes what appears to be similar calibrations as a calibration of the sensors to harmonize the readings of the encoder and IMU. In this manner, Applicant’s specification discloses using a fusion of RGB image data and spatial distance data generated by a spatial distance sensor to calibrate disparity between encoders and IMUs to align the respective sensor readings. Nowhere does Applicant’s original disclosure appear to disclose, with reasonable clarity, using the fusion of RGB image data and spatial distance data generated by a spatial distance sensor as a basis for generating a remediation instruction to cause the robot to adjust a motor of a wheel of the robot or a body part of the robot.
Claim 20 recites “wherein generating the remediation instruction comprises: determining, based on the fusion of the first series and the second series, a slippage of a wheel of the robot, and generating a remediation instruction to cause the robot to adjust a motor of the wheel,” in which per antecedent relation with claim 2 “the remedial instruction” is “based on the fusion of the first series and the second series” in which the first series is “RGB image data” and the second series is “spatial distance data generated by a spatial distance sensor.” Applicant’s original disclosure does not disclose, with reasonable clarity, that the inventors had possession of claimed invention including this combination of features.
Regarding the recited process for “generating the remediation instruction” including “determining … a slippage of a wheel…,” Applicant’s original disclosure only infers “generating the remediation instruction” (e.g., [0076] calibration implemented and [0142] preform remediation task, motor instruction applied), and does not appear to teach a process of generating a remediation instruction such as including “determining … a slippage of a wheel…”
Regarding “determining” “slippage of a wheel of the robot,” Applicant’s specification discloses using wheel encoder/IMU disparities to determine wheel slippage ([0057], [0071], [0133], and [0142]). Applicant’s specification further discloses in [0076] using the fusion of RGB image data and spatial distance data generated by a spatial distance sensor for calibrating to correct for misalignments between encoder/IMU measurements, and further teaches that encoder/IMU misalignments may result from wheel slippage ([0057] and [0133]). However, Applicant’s original disclosure does not appear to disclose, with reasonable clarity, that a process of “generating the remedial instruction” (that per antecedent relation with claim 2 is based on a fusion of RGB image data and spatial distance data generated by a spatial distance sensor) includes “determining, based on the fusion of the first series and the second series, a slippage of a wheel of the robot.”
Claim 21 recites “wherein generating the remediation instruction comprises: determining, based on the fusion of the first series and the second series, a condition of the robot traversing the pipeline, wherein the condition comprises at least one of at least one of abnormal movement of the robot, a tool of the robot has accomplished a task, a slippage of a wheel of the robot, or a drift in at least one of the first series of RGB sensor data or the second series of spatial distance data,” in which per antecedent relation with claim 2 the first series is “RGB image data” and the second series is “spatial distance data generated by a spatial distance sensor.” Applicant’s original disclosure does not disclose, with reasonable clarity, that the inventors had possession of claimed invention including this combination of features.
Regarding the recited process for “generating the remediation instruction” including “determining … a condition of the robot traversing the pipeline …,” Applicant’s original disclosure only infers “generating the remediation instruction” (e.g., [0076] calibration implemented and [0142] preform remediation task, motor instruction applied), and does not appear to teach a process of generating a remediation instruction such as including “determining … a slippage of a wheel…”
Regarding “determining, based on the fusion of the first series and the second series, a condition of the robot traversing the pipeline, wherein the condition comprises at least one of at least one of abnormal movement of the robot, a tool of the robot has accomplished a task, a slippage of a wheel of the robot, or a drift in at least one of the first series of RGB sensor data or the second series of spatial distance data,” Applicant’s specification discloses determining a condition of the robot such as in [0142]-[0143], which describes determining a condition (e.g., wheel slippage, task accomplished, abnormal movement, dataset drift) of a robot based on sensor data including data received from an IMU and/or encoder and in some cases from two or more sensors. Neither in [0142]-[0143] nor elsewhere does Applicant’s original disclosure appear to teach that the particularized sensor processing configuration entailing a fusion of RGB image data and spatial distance data generated by a spatial distance sensor is used as a basis for determining a robot condition including at least one of abnormal movement of the robot, a tool of the robot has accomplished a task, a slippage of a wheel of the robot, or a drift in at least one of the first series of RGB sensor data or the second series of spatial distance data.
Using a fusion of RGB image data and spatial distance data generated by a spatial distance sensor for generating may at least inferentially considered to be a remediation instruction is described in Applicant’s specification in [0076], which describes a “calibration transformation” that is implemented based on a fusion/combination of IR image data mapped into 3D space and RGB image data mapped into 3D space, and which entails accounting for detected differences between the respective encoders and IMUs accounting for the different frames of reference of the IR and RGB cameras. Paragraph [0057] describes what appears to be similar calibrations as a calibration of the sensors to harmonize the readings of the encoder and IMU. In this manner, Applicant’s specification discloses using a fusion of RGB image data and spatial distance data generated by a spatial distance sensor to calibrate disparity between encoders and IMUs to align the respective sensor readings. Nowhere does Applicant’s original disclosure appear to disclose, with reasonable clarity, using the fusion of RGB image data and spatial distance data generated by a spatial distance sensor as a basis for determining a condition of the robot traversing the pipeline, wherein the condition comprises at least one of at least one of abnormal movement of the robot, a tool of the robot has accomplished a task, a slippage of a wheel of the robot, or a drift in at least one of the first series of RGB sensor data or the second series of spatial distance data.
Examiner further notes that while Applicant’s specification discloses the “condition” as including “drift in a dataset” ([0142]), Applicant’s original disclosure does not appear to disclose, with reasonable clarity, the more specific condition of “drift in at least one of the first series of RGB sensor data or the second series of spatial distance data.”
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 2-6, 8-14, and 16-22 are rejected under 35 U.S.C. 101 because the claimed invention in each of these claims is directed to the abstract idea judicial exception without significantly more.
Independent claim 10, substantially representative also of independent claims 2 and 22, recites:
“[a]n apparatus comprising:
a processor; and
a memory configured to store instructions that, executed by the processor, cause the apparatus to:
receive by the processor, a first series of RGB image data generated by an RGB sensor deployed on a robot traversing an environment, wherein the first series is a function of a distance traversed by the robot;
receive, by the processor, a second series of spatial distance data generated by a spatial distance sensor deployed on the robot traversing the environment, wherein the second series is a function of the distance traversed by the robot;
fuse together, by the processor, the first series of RGB image data and the second series of spatial distance data; and
generate, by the processor and based on the fusion of the first series and the second series, a remediation instruction for execution by the robot traversing the environment.”
The claim limitations considered to fall within in the abstract idea are highlighted in bold font above and the remaining features are “additional elements.”
Step 1 of the subject matter eligibility analysis entails determining whether the claimed subject matter falls within one of the four statutory categories of patentable subject matter identified by 35 U.S.C. 101: process, machine, manufacture, or composition of matter. Claim 10 an apparatus, claim 2 recites a method, and claim 22 recites an article of manufacture and each therefore falls within a statutory category.
Step 2A, Prong One of the analysis entails determining whether the claim recites a judicial exception such as an abstract idea. Under a broadest reasonable interpretation, the highlighted portions of claim 10 fall within the abstract idea judicial exception. Specifically, under the 2019 Revised Patent Subject Matter Eligibility Guidance, the highlighted subject matter falls within the mathematical concepts category (mathematical relationships, mathematical formulas or equations, mathematical calculations) and the mental processes exception (e.g., evaluation, judgment, opinion). MPEP § 2106.04(a)(2).
The recited function “fuse together” “the first series of RGB image data and the second series of spatial distance data,” is determined by the Examiner as falling within the mathematical relationships sub-category of mathematical concepts (MPEP 2106.04(a)(2)) because fusing RGB imaging data (two-dimensional data as described in Applicant’s specification) with three-dimensional point cloud data is fundamentally characterized by mathematical relations and calculations in terms, for example, of alignment of the respective two-dimensional and three-dimension coordinates and in terms of associated methods described in Applicant’s specification for effectuating such fusing (e.g., localization via SLAM, which is fundamentally characterized by mathematical relations/calculations).
The recited function “generate” “based on the fusion of the first series and the second series, a remediation instruction for execution by the robot traversing the environment,” is found to fall within mental processes exception because a broadest reasonable interpretation of what may constitute generating a remediation instruction based on the fusion of the first and second series includes a mental determination (what remedial action to perform) based on evaluation (mental) of the fused data.
Step 2A, Prong Two of the analysis entails determining whether the claim includes additional elements that integrate the recited judicial exception into a practical application. “A claim that integrates a judicial exception into a practical application will apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the judicial exception” (MPEP § 2106.04(d)).
MPEP § 2106.04(d) sets forth considerations to be applied in Step 2A, Prong Two for determining whether or not a claim integrates a judicial exception into a practical application. Based on the individual and collective limitations of claim 10 and applying a broadest reasonable interpretation, the most applicable of such considerations appear to include: improvements to the functioning of a computer, or to any other technology or technical field (MPEP 2106.05(a)); applying the judicial exception with, or by use of, a particular machine (MPEP 2106.05(b)); and effecting a transformation or reduction of a particular article to a different state or thing (MPEP 2106.05(c)).
Regarding improvements to the functioning of a computer or other technology, none of the “additional elements” including “a processor,” “a memory configured to store instructions that, when executed by the processor” perform the recited receiving, fusing, and generating functions, “receive by the processor, a first series of RGB image data generated by an RGB sensor deployed on a robot traversing an environment, wherein the first series is a function of a distance traversed by the robot,” and “receive, by the processor, a second series of spatial distance data generated by a spatial distance sensor deployed on the robot traversing the environment, wherein the second series is a function of the distance traversed by the robot,” in any combination appear to integrate the abstract idea in a manner that technologically improves any aspect of a device or system that may be used to implement the highlighted step or a device for implementing the highlighted step such as a signal processing device or a generic computer. For example, the structural features including the processor and memory represent standard data processing functionality for implementing the fusing step that falls within the judicial exception and therefore constitute extra solution activity that fails to integrate the judicial exception into a practical application. The two “receiving” steps, individually and in combination, only convey that particular types of information are received, as the apparatus recited in claim 10 does not positively recite that the apparatus includes an RGB sensor and/or a spatial distance sensor, such that both of these steps, individually and in combination, represent high level data collection that constitutes extra solution activity that fails to integrate the judicial exception into a practical application.
Regarding application of the judicial exception with, or by use of, a particular machine, the additional elements are recited broadly as being configured and implemented in a manner reflective of mere data gathering rather than in a particularized manner of implementing multi-sensor imaging such as for determining remediation instructions.
Regarding a transformation or reduction of a particular article to a different state or thing, claim 10 does not include any such transformation or reduction. Instead, claim 10 as a whole entails receiving input information (RGB and spatial distance data that is measured independent of the scope of the claim), applying standard processing techniques (standard computer processor) and mathematics (processing entailed in fusing 2D and 3D image data) to combine the information to determine a remediation instruction with the additional elements failing to provide a meaningful integration of the abstract idea in an application that transforms an article to a different state. Instead, the additional elements represent extra-solution activity that does not integrate the judicial exception into a practical application. In view of the various considerations encompassed by the Step 2A, Prong Two analysis, claim 10 does not include additional elements that integrate the recited abstract idea into a practical application.
Therefore, claim 10 is directed to a judicial exception and requires further analysis under Step 2B.
Regarding Step 2B, and as explained in the Step 2A Prong Two analysis, the additional elements constitute extra-solution activity and therefore fail to result in the claim as a whole amounting to significantly more than the judicial exception as well as failing to integrate the judicial exception into a practical application. Furthermore, the additional elements in claim 10 appear to be generic and well understood as evidenced by the disclosures of Ligocki et al., "Atlas Fusion - Modern Framework for Autonomous Agent Sensor Data Fusion," 2022 ELEKTRO (ELEKTRO), Krakow, Poland, 2022, pp. 1-6 (Ligocki), Kueny (US 2019/0285555 A1), and X. Xu, L. Zhang, J. Yang, C. Cao, Z. Tan and M. Luo, "Object Detection Based on Fusion of Sparse Point Cloud and Image Information," in IEEE Transactions on Instrumentation and Measurement, vol. 70, pp. 1-12, 2021 (Xu), which teach substantially similar data processing functions using the same structural features.
As explained in the grounds for rejecting claim 10 under 103, Ligocki teaches “a processor,” “a memory configured to store instructions that, when executed by the processor” perform the recited receiving, fusing, and generating functions, “receive by the processor, a first series of RGB image data generated by an RGB sensor deployed on a robot [inferred by Ligocki] traversing an environment, wherein the first series is a function of a distance traversed by the robot,” and “receive, by the processor, a second series of spatial distance data generated by a spatial distance sensor deployed on the robot [inferred by Ligocki] traversing the environment, wherein the second series is a function of the distance traversed by the robot.” Similarly, Kueny discloses an apparatus for implementing multi-sensor inspection that includes processor, and memory for processing sensed imaging data (FIG. 5 computer 500) and includes receiving multi-sensor data obtained from robot-deployed sensors (Abstract structured laser imaging and LiDAR) to generate fused images (Abstract, FIG. 3 block 307). Xu further discloses a framework for fusing multi-sensor data (Abstract and FIG. 1) in which both LiDAR and RGB imaging data are received/processed (FIG. 1; page 2, Introduction, paragraph beginning with “In response to the above problems…” explaining the camera image data is RGB data).
Therefore, the additional elements are insufficient to amount to significantly more than the judicial exception.
Independent claim 10 is therefore not patent eligible under 101.
Independent claims 2 and 22 each includes substantially the same elements falling within the judicial exception as claim 10 and neither includes further significant additional elements that either integrate the judicial exception into a practical application or result in the claim as a whole amounting to significantly more than the judicial exception.
Therefore, claims 2 and 22 are also not patent eligible under 101.
Claims 3-6, 8-9 and 18-21 depending from claim 2, and claims 11-14, and 16-17 depending from claim 10, provide additional features/steps that are part of an expanded algorithm that includes the abstract idea of claims 2, 10, and 22 (Step 2A, Prong One). None of dependent claims 3-6, 8-9, 11-14, 16-21 recite additional elements that integrate the abstract idea into practical application (Step 2A, Prong Two), and all fail the “significantly more” test under the step 2B for substantially similar reasons as discussed with regards to the independent claims.
For example, claim 3 substantially representative also of claim 11, recites a function of detecting features of the environment in real time, which is found by the Examiner to fall within the mental processes judicial exception (including an observation, evaluation, judgment, opinion). MPEP § 2106.04(a)(2). Detecting features of an environment in real time may be performed via mental processes such as evaluation of sensor data (e.g., real-time camera video) and judgement in ascertaining features revealed by the data in real-time. Claim 3 further recites that the detection is performed by a machine learning model. which represents routine, conventional program instruction implementation of the step falling within the judicial exception and therefore constitutes extra solution activity that neither integrates the judicial exception into a practical application nor results in the claim as a whole amounting to significantly more than the judicial exception. Claim 3 further recites that the machine learning model is deployed in the processor that is attached to a robot, which is a structural feature related to data collection and having no particularized functional relation to the steps falling within the judicial exception and therefore constitutes extra solution activity that neither integrates the judicial exception into a practical application nor results in the claim as a whole amounting to significantly more than the judicial exception.
Claims 4 and 12 recite mapping the first and second series to a robot position relative to the environment, which falls within the mental processes exception because mapping RGB and spatial distance data to data indicating robot position may be performed via mental processes (e.g., evaluation possibly aided by pen-and-paper and judgement). This element is also found to fall within the mathematical concepts exception because as disclosed in Applicant’s specification such mapping may be implemented via SLAM, which is fundamentally characterized by mathematical relations/calculations. Claim 4 further recites generating a digital representation of the environment based on the mapping represents insignificant post-solution activity.
Claims 5-6, 9, 13-14, and 17 recite additional sensor types (infrared for claims 5 and 13, and IMU for claims 6, 9, 14 and 17) for detecting information relevant to multi-sensor inspection. These elements represent high-level data collection typical of imaging inspection systems and having no particularized functional relation to the steps falling within the judicial exception and therefore constitute extra solution activity that neither integrates the judicial exception into a practical application nor results in the claim as a whole amounting to significantly more than the judicial exception.
Claim 8, substantially representative also of claim 16, further recites tracking a distance moved by the robot within the environment, which falls within the mental processes judicial exception because it may be performed via mental processes (e.g., evaluation of information such as position, speed, etc., possible aided by pen-and-paper and judgement). Claim 8 further recites that the processor for performing the tracking is attached to the robot and that the tracking is performed using data received by a motor encoder associated with a wheel, which represents high-level data collection and therefore constitutes extra solution activity that neither integrates the judicial exception into a practical application nor results in the claim as a whole amounting to significantly more than the judicial exception. Examiner notes that the apparatus/method recited in claims 8 and 16 does not positively incorporate the motor encoder as a functional entity, instead the motor encoder is merely a source of information processed by the recited method/apparatus.
Claim 18 recites that generating the remediation instruction further includes “producing, based on the of the first series and the second series, a calibration transformation,” which falls within the mental processes exception because it may be performed via mental processes (e.g., evaluate the first and second series data (individually and/or in combination such as in a fused/overlayed representation) and judgment to determine a calibration adjustment/transformation). Generating, based on the calibration transformation, a remediation instruction to remediate a difference between data produced by at least one encoder and data produced by at least one inertial measurement unit” may also be performed via mental processes (e.g., evaluation of a warranted calibration adjustment to further determine via judgment a remediation to address a different between encoder and IMU data), and therefore also falls within the mental processes exception.
Claim 19 recites “generating a remediation instruction to cause the robot to adjust a motor of a wheel of the robot or a body part of the robot,” which falls within the mental processes exception for substantially the same reasons as the generating step in claim 2.
Claim 20 recites that generating the remediation instruction further comprises “determining, based on the fusion of the first series and the second series, a slippage of a wheel of the robot,” which falls within the mental processes exception because it may be performed via mental processes (e.g., evaluation of fusion data, which may be overlayed image data and judgment in recognizing a slippage condition based thereon). The further step of “generating a remediation instruction to cause the robot to adjust a motor of the wheel” also falls within the mental processes exception for substantially the same reasons as the generating step in claim 2.
Claim 21 further recites that the robot is traversing a “pipeline” environment, which only contextually limits the application of the recited method and does not otherwise entail an additional element that ingrates the judicial exception into a practical application or result in the claim as a whole amounting to significantly more than the judicial exception. Claim 21 further recites that generating the remediation instruction further comprises “determining, based on the fusion of the first series and the second series, a condition of the robot traversing the pipeline, wherein the condition comprises at least one of at least one of abnormal movement of the robot, a tool of the robot has accomplished a task, a slippage of a wheel of the robot, or a drift in at least one of the first series of RGB sensor data or the second series of spatial distance data,” which falls within the mental processes exception because it may be performed via mental processes (e.g., evaluation of fusion data, which may be overlayed image data and judgment in recognizing a robot condition such as slippage based thereon). The further step of “generating, based on the condition of the robot, a remediation instruction to remediate the condition of the robot” also falls within the mental processes exception for substantially the same reasons as the generating step in claim 2.
Dependent claims 3-6, 8-9, 11-14, 16-21 therefore also constitute ineligible subject matter under 101.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2-6, 10-14, 19, and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Ligocki et al., "Atlas Fusion - Modern Framework for Autonomous Agent Sensor Data Fusion," 2022 ELEKTRO (ELEKTRO), Krakow, Poland, 2022, pp. 1-6 (Ligocki), in view of Kueny (US 2019/0285555 A1) and in further view of Ebrahimi Afrouzi (US 11,548,159 A1).
As to claim 2, Ligocki teaches “[a] method (Abstract describing framework including autonomous robot and imaging sensors for implementing method that includes fusing multi-sensor data including RGB and 3D LiDAR data (digital) for environmental visualization that, per page 2 C. Algorithms, D. Local Maps, and E. Visualizers, is generated via computer processing (is a digital representation)), comprising:
receiving by a processor (pages 1-2, B. Core Pipeline, paragraph beginning with “At the startup …” through paragraph beginning with “The entire pipeline …” describing processing/program “pipeline” including loading data into memory (entails processor-based information processing); page 2, C. Algorithms, paragraph beginning with “The ‘Algorithms’ module …” further characterizing the method as implemented via data processing code (implemented via processor)), a first series of RGB image data generated by an RGB sensor (page 1, A. Input Data, paragraph beginning with “The data stored as …” describing RGB camera data as input data; pages 3-4, C. Camera-LiDAR Object Detection, paragraph beginning with “LiDAR can measure…” through paragraph beginning with “For this purpose, we have …” describing processing of RGB image data) deployed” [with respect to] “a robot traversing an environment (Abstract describing framework including multi-sensor configuration including RGB camera deployed with respect to an autonomous robot; page 2, C. Outputs, paragraph beginning with “Secondary, there are several …” describing that the agent (robot) travels during a “mapping session” that includes collection of RGB camera data (the RGB data is collected in association with agent travel such that the camera would be deployed on the agent robot)), wherein the first series is a function of a distance traversed by the robot (the RGB camera data would reflect the environment over the path traversed during a “mapping session” including detected features per pages 3-4, C. Camera-LiDAR Object Detection, paragraph beginning with “LiDAR can measure …”);
receiving, by the processor, a second series of spatial distance data generated by a spatial distance sensor (Abstract and page 1, A. Input Data, paragraph beginning with “The data stored as …” describing LiDAR scan data as input data for data fusion; page 3, B. LiDAR data aggregation, paragraphs beginning with “As we are using …” and “The input LiDAR data …” describing obtaining LiDAR data) deployed” [with respect to] “the robot traversing the environment (Abstract describing framework including multi-sensor configuration including 3D LiDAR deployed with respect to an autonomous robot; page 3, B. LiDAR data aggregation, paragraph beginning with “As we are using …” and “The input LiDAR data could come …” describing the scanning function performed during and affected by robot re-positioning (clearly inferring that the scanners are mounted to the robot)), wherein the second series is a function of the distance traversed by the robot (the LiDAR data would reflect the environment over the path traversed during agent repositioning);
fusing together, by the processor, the first series of RGB image data and the second series of spatial distance data (Abstract describing fusion of RGB cameras and 3D LiDAR; page 1, I. Introduction, paragraph beginning with “As a result, our team …” describing fusion of various sensor types (as depicted in FIG. 1); FIG. 1 depicting visualization that combines (fuses) RGB image data and LiDAR data; page 4, C. Camera-LiDAR Object Detection, paragraphs beginning with “For this purpose, we have created …” explaining that the system fuses the LiDAR data and camera detections into a single representation, and paragraph beginning with “There is an estimated …” describing the RGB as the camera source and describing the detected object aspect of the RGB data (frustum color coded as shown in FIG. 1)).”
As set forth above, Ligocki strongly suggests but does not expressly teach that the RGB sensor and spatial distance sensor are mounted “on” the robot.
Kueny discloses a method for performing multi-sensor inspection using a multi-sensor robot (Abstract) that includes using a robot on which multiple imaging sensors are deployed ([0004] and [0023] multi-sensor inspection robot includes a variety of multiple sensor types including 3D LIDAR and optical imaging sensors such as cameras; [0027] and [0029] sensor data may include LIDAR (point cloud) and laser scan (image) obtained by LiDAR unit and laser scanner).
It would have been obvious to one of ordinary skill in the art before the effective filing date, to have applied Kueny’s teaching of deploying multiple imaging sensors onto a mobile robot including LIDAR and optical imaging (e.g., cameras) for implementing multi-sensing inspection to the method taught by Ligocki, such that in combination the method includes deploying the RGB sensor and spatial distance sensor on the robot traversing the environment.
The motivation would have been to effectuate multi-sensor surveying/inspection of an environment as the robot travels through the environment as disclosed by Kueny.
Neither Ligocki nor Kueny appear to expressly teach generating a remediation instruction for execution by the robot traversing the environment based on the fusion of the first series and the second series.
Ebrahimi Afrouzi discloses a method/apparatus for implementing and controlling a robot (Abstract) that includes fusing imaging data from multiple sensors (col. 2 lines 18-28 first and second imaging data combined for an overlapping field of view to generate a combined representation of the environment) deployed on a robot (col. 2 lines 4-16) in which one of the sensors may be an RGB camera (col. 7 lines 3-5; col. 26 lines 49-56) and another may be a spatial distance sensor (col. 7 lines 3-6 LIDAR sensor) and further teaches that that the environment information determined by such fusion may be used to control/remediate actions (inherently entails instruction) performed by the robot (col. 2 lines 28-40 spatial environment data used for estimating a corrected position of the robot which is then used for corrected robot navigation; col. 59 line 21 through col. 60 line 2 sensor determined environmental characteristics used for controlling robot motor).
It would have been obvious to one of ordinary skill in the art before the effective filing date, to have applied Ebrahimi Afrouzi’s teaching of used fused RGB/camera imaging data for remediating robot actions to the method taught by Ligocki as modified by Kueny such that in combination the method includes generating a remediation instruction for execution by the robot traversing the environment based on the fusion of the first series and the second series.
The motivation would have been to leverage the enhanced environmental imaging obtained by fusion to improve robot operations as disclosed by Ebrahimi Afrouzi.
As to claim 3, the combination of Ligocki, Kueny, and Ebrahimi Afrouzi teaches “[t]he method of claim 2, further comprising a machine learning model deployed in the processor (Ligocki: pages 3-4, C. Camera-LiDAR Objection Detection, paragraph beginning with “LiDAR can measure …” describing use of neural network (inherently executed via a processor)),” “wherein the method further comprises detecting, by the machine learning model, features of the environment in real time (Ligocki: pages 3-4, C. Camera-LiDAR Objection Detection, paragraph beginning with “LiDAR can measure …” describing use of neural network for object detection in real time).”
Ligocki does not appear to teach that a processor for implementing the machine learning model is incorporated with the robot structure.
Kueny discloses a method for performing multi-sensor inspection using a multi-sensor robot (Abstract) in which machine learning is used for detecting features of the environment ([0030]) and in which the processing for implementing the inspection tasks is a processor attached to the robot (FIG. 5 computer 500 including CPU 510 that per [0042] configured for implementing processing functionality with respect to inspection robot 570; [0037] computer may be deployed within inspection robot 570).
It would have been obvious to one of ordinary skill in the art before the effective filing date, to have applied Kueny’s teaching of using an “on-board” processor attached to the robot for performing processing function associated with the multi-sensor data collection that includes machine learning feature detection to the method taught by Ligocki as modified by Kueny to include a multi-sensor robot, such that in combination the processor that implements the machine learning detection is configured to be attached to the robot.
Such a combination would amount to selecting a known design option for deploying processing functionality for a multi-sensor robot to achieve predicable results.
As to claim 4, the combination of Ligocki, Kueny, and Ebrahimi Afrouzi teaches “[t]he method of claim 2, further comprising:
mapping, by the processor, each of the first series and second series to a position of the robot relative to the environment (Ligocki: page 3, A. Precise Positioning, paragraph beginning with “Without precise positioning …” through the paragraph beginning with “As the system models …” explaining that the agent position is tracked for building the image map model that is generated, as depicted and explained with reference to FIGS. 1 and 6, based on both the 3D LiDAR data and the RGB camera data, such that the 3D LiDAR data and RGB camera data track with the agent position (e.g., RGB camera data aligned as in images in FIGS. 1 and 6 with LiDAR data that per page 3, B. LiDAR data aggregation, paragraph beginning with “The input LiDAR data …” through paragraph beginning with “We have already estimated …” is mapped to the agent’s location); and
generating, by the processor, a digital representation of the environment (FIGS. 1 and 6 are generated by computer processing (per page 2, C. Algorithms; and E. Visualizers) based on digital data (RGB camera and LiDAR) and therefore constitute digital representations) based on the mapping (the aligned optical imaging/LiDAR imaging results such as depicted in Ligocki FIGS. 1 and 6 are dependent on the position tracking/mapping as explained on page 3, A. Precise Positioning, paragraph beginning with “Without precise positioning …”; page 3, B. LiDAR data aggregation, paragraph beginning with “The input LiDAR data …” through paragraph beginning with “We have already estimated …” explaining the use of agent positioning information on the LiDAR scanning results (from which the final digital representation is generated)).”
As to claim 5, the combination of Ligocki, Kueny, and Ebrahimi Afrouzi teaches “[t]he method of claim 2, further comprising detecting by an infrared sensor associated with the robot (Ligocki: Abstract explaining that multi-sensor data associated with robot may include data from thermal (infrared) camera; page 1, A. Input Data, paragraph beginning with “The data are stored …” describing input data as including data from thermal camera; page 4, D. RGB YOLO Detections to IR Image, paragraph beginning with “If we focus …” through paragraph beginning with “We have proposed …” designating the thermal images as “IR images”), temperature gradients in the environment (Ligocki: FIG. 6 (bottom) depicting thermal/IR imaging that shows thermal profiles of objects. Examiner note that thermal/IR sensors inherently implement imaging via thermal profiles that are manifested via temperature gradients).”
As to claim 6, the combination of Ligocki, Kueny, and Ebrahimi Afrouzi teaches “[t]he method of claim 4, further comprising determining by an inertial measurement unit associated with the robot (Ligocki: Abstract explaining that multi-sensor data associated with robot may include data from a 3D IMU; page 1, A. Input Data, paragraph beginning with “The data are stored …” describing input data as including data from IMU), a pose and position of the robot (Ligocki: FIG. 2 depicting IMU providing orientation information (roll, pitch) and linear acceleration data (position/location information) for determining position and orientation outputs).”
As to claim 10, Ligocki teaches “[a]n apparatus (Abstract describing framework including autonomous robot and imaging sensors for implementing method that includes fusing multi-sensor data including RGB and 3D LiDAR data (digital) for environmental visualization that, per page 2 C. Algorithms, D. Local Maps, and E. Visualizers, is generated via computer processing (is a digital representation); pages 1-2, II. General Architecture Description, describing data processing structure) comprising:
a processor (pages 1-2, B. Core Pipeline, paragraph beginning with “At the startup …” through paragraph beginning with “The entire pipeline …” describing processing/program “pipeline” including loading data into memory (entails processor-based information processing); page 2, C. Algorithms, paragraph beginning with “The ‘Algorithms’ module …” further characterizing the method as implemented via data processing code (implemented via processor)); and
a memory configured to store instructions (pages 1-2, B. Core Pipeline, paragraph beginning with “At the startup …” through paragraph beginning with “The entire pipeline …” describing processing/program “pipeline” including loading data into memory for processing; page 2, C. Algorithms, paragraph beginning with “The ‘Algorithms’ module …” further characterizing the method as implemented via data processing code (entails execution of instructions)) that, when executed by the processor, cause the apparatus to:
receive by the processor, a first series of RGB image data generated by an RGB sensor (page 1, A. Input Data, paragraph beginning with “The data stored as …” describing RGB camera data as input data; pages 3-4, C. Camera-LiDAR Object Detection, paragraph beginning with “LiDAR can measure…” through paragraph beginning with “For this purpose, we have …” describing processing of RGB image data) deployed” [with respect to] “a robot traversing an environment (Abstract describing framework including multi-sensor configuration including RGB camera deployed with respect to an autonomous robot; page 2, C. Outputs, paragraph beginning with “Secondary, there are several …” describing that the agent (robot) travels during a “mapping session” that includes collection of RGB camera data (the RGB data is collected in association with agent travel such that the camera would be deployed on the agent robot)), wherein the first series is a function of a distance traversed by the robot (the RGB camera data would reflect the environment over the path traversed during a “mapping session” including detected features per pages 3-4, C. Camera-LiDAR Object Detection, paragraph beginning with “LiDAR can measure …”);
receive, by the processor, a second series of spatial distance data generated by a spatial distance sensor (Abstract and page 1, A. Input Data, paragraph beginning with “The data stored as …” describing LiDAR scan data as input data for data fusion; page 3, B. LiDAR data aggregation, paragraphs beginning with “As we are using …” and “The input LiDAR data …” describing obtaining LiDAR data) deployed” [with respect to] “the robot traversing the environment (Abstract describing framework including multi-sensor configuration including 3D LiDAR deployed with respect to an autonomous robot; page 3, B. LiDAR data aggregation, paragraph beginning with “As we are using …” and “The input LiDAR data could come …” describing the scanning function performed during and affected by robot re-positioning (clearly inferring that the scanners are mounted to the robot)), wherein the second series is a function of the distance traversed by the robot (the LiDAR data would reflect the environment over the path traversed during agent repositioning);
fuse together, by the processor, the first series of RGB image data and the second series of spatial distance data (Abstract describing fusion of RGB cameras and 3D LiDAR; page 1, I. Introduction, paragraph beginning with “As a result, our team …” describing fusion of various sensor types (as depicted in FIG. 1); FIG. 1 depicting visualization that combines (fuses) RGB image data and LiDAR data; page 4, C. Camera-LiDAR Object Detection, paragraphs beginning with “For this purpose, we have created …” explaining that the system fuses the LiDAR data and camera detections into a single representation, and paragraph beginning with “There is an estimated …” describing the RGB as the camera source and describing the detected object aspect of the RGB data (frustum color coded as shown in FIG. 1)).”
As set forth above, Ligocki strongly suggests but does not expressly teach that the RGB sensor and spatial distance sensor are mounted “on” the robot.
Kueny discloses a method for performing multi-sensor inspection using a multi-sensor robot (Abstract) that includes using a robot on which multiple imaging sensors are deployed ([0004] and [0023] multi-sensor inspection robot includes a variety of multiple sensor types including 3D LIDAR and optical imaging sensors such as cameras; [0027] and [0029] sensor data may include LIDAR (point cloud) and laser scan (image) obtained by LiDAR unit and laser scanner).
It would have been obvious to one of ordinary skill in the art before the effective filing date, to have applied Kueny’s teaching of deploying multiple imaging sensors onto a mobile robot including LIDAR and optical imaging (e.g., cameras) for implementing multi-sensing inspection to the method taught by Ligocki, such that in combination the method includes deploying the RGB sensor and spatial distance sensor on the robot traversing the environment.
The motivation would have been to effectuate multi-sensor surveying/inspection of an environment as the robot travels through the environment as disclosed by Kueny.
Neither Ligocki nor Kueny appear to expressly teach generating a remediation instruction for execution by the robot traversing the environment based on the fusion of the first series and the second series.
Ebrahimi Afrouzi discloses a method/apparatus for implementing and controlling a robot (Abstract) that includes fusing imaging data from multiple sensors (col. 2 lines 18-28 first and second imaging data combined for an overlapping field of view to generate a combined representation of the environment) deployed on a robot (col. 2 lines 4-16) in which one of the sensors may be an RGB camera (col. 7 lines 3-5; col. 26 lines 49-56) and another may be a spatial distance sensor (col. 7 lines 3-6 LIDAR sensor) and further teaches that that the environment information determined by such fusion may be used to control/remediate actions (inherently entails instruction) performed by the robot (col. 2 lines 28-40 spatial environment data used for estimating a corrected position of the robot which is then used for corrected robot navigation; col. 59 line 21 through col. 60 line 2 sensor determined environmental characteristics used for controlling robot motor).
It would have been obvious to one of ordinary skill in the art before the effective filing date, to have applied Ebrahimi Afrouzi’s teaching of used fused RGB/camera imaging data for remediating robot actions to the apparatus taught by Ligocki as modified by Kueny such that in combination the apparatus is configured to generate a remediation instruction for execution by the robot traversing the environment based on the fusion of the first series and the second series.
The motivation would have been to leverage the enhanced environmental imaging obtained by fusion to improve robot operations as disclosed by Ebrahimi Afrouzi.
As to claim 11, the combination of Ligocki, Kueny, and Ebrahimi Afrouzi teaches “[t]he apparatus of claim 10, further comprising a machine learning model deployed in the processor (Ligocki: pages 3-4, C. Camera-LiDAR Objection Detection, paragraph beginning with “LiDAR can measure …” describing use of neural network (inherently executed via a processor)),” “wherein the machine learning model is configured to detect features of the environment in real time (Ligocki: pages 3-4, C. Camera-LiDAR Objection Detection, paragraph beginning with “LiDAR can measure …” describing use of neural network for object detection in real time).”
Ligocki does not appear to teach that a processor for implementing the machine learning model is incorporated with the robot structure.
Kueny discloses a method/system for performing multi-sensor inspection using a multi-sensor robot (Abstract) in which machine learning is used for detecting features of the environment ([0030]) and in which the processing for implementing the inspection tasks is a processor attached to the robot (FIG. 5 computer 500 including CPU 510 that per [0042] configured for implementing processing functionality with respect to inspection robot 570; [0037] computer may be deployed within inspection robot 570).
It would have been obvious to one of ordinary skill in the art before the effective filing date, to have applied Kueny’s teaching of using an “on-board” processor attached to the robot for performing processing function associated with the multi-sensor data collection that includes machine learning feature detection to the apparatus taught by Ligocki as modified by Kueny to include a multi-sensor robot, such that in combination the processor that implements the machine learning detection is configured to be attached to the robot.
Such a combination would amount to selecting a known design option for deploying processing functionality for a multi-sensor robot to achieve predicable results.
As to claim 12, the combination of Ligocki, Kueny, and Ebrahimi Afrouzi teaches “[t]he apparatus of claim 11, wherein the instructions further cause the apparatus to:
map each of the first series and second series to a position of the robot relative to the environment (Ligocki: page 3, A. Precise Positioning, paragraph beginning with “Without precise positioning …” through the paragraph beginning with “As the system models …” explaining that the agent position is tracked for building the image map model that is generated, as depicted and explained with reference to FIGS. 1 and 6, based on both the 3D LiDAR data and the RGB camera data, such that the 3D LiDAR data and RGB camera data track with the agent position (e.g., RGB camera data aligned as in images in FIGS. 1 and 6 with LiDAR data that per page 3, B. LiDAR data aggregation, paragraph beginning with “The input LiDAR data …” through paragraph beginning with “We have already estimated …” is mapped to the agent’s location); and
generate a digital representation of the environment based on the mapping (the aligned optical imaging/LiDAR imaging results such as depicted in Ligocki FIGS. 1 and 6 are dependent on the position tracking/mapping as explained on page 3, A. Precise Positioning, paragraph beginning with “Without precise positioning …”; page 3, B. LiDAR data aggregation, paragraph beginning with “The input LiDAR data …” through paragraph beginning with “We have already estimated …” explaining the use of agent positioning information on the LiDAR scanning results (from which the final digital representation is generated)).
As to claim 13, the combination of Ligocki, Kueny, and Ebrahimi Afrouzi teaches “[t]he apparatus of claim 11, further comprising an infrared sensor associated with the robot (Ligocki: Abstract explaining that multi-sensor data associated with robot may include data from thermal (infrared) camera; page 1, A. Input Data, paragraph beginning with “The data are stored …” describing input data as including data from thermal camera; page 4, D. RGB YOLO Detections to IR Image, paragraph beginning with “If we focus …” through paragraph beginning with “We have proposed …” designating the thermal images as “IR images”), the infrared sensor configured to detect temperature gradients in the environment (Ligocki: FIG. 6 (bottom) depicting thermal/IR imaging that shows thermal profiles of objects. Examiner note that thermal/IR sensors inherently implement imaging via thermal profiles that are manifested via temperature gradients).”
As to claim 14, the combination of Ligocki and Kueny teaches “[t]he apparatus of claim 11, further comprising an inertial measurement unit associated with the robot (Ligocki: Abstract explaining that multi-sensor data associated with robot may include data from a 3D IMU; page 1, A. Input Data, paragraph beginning with “The data are stored …” describing input data as including data from IMU), the inertial measurement unit configured to determine a pose and position of the robot (Ligocki: FIG. 2 depicting IMU providing orientation information (roll, pitch) and linear acceleration data (position/location information) for determining position and orientation outputs).”
As to claim 19, the combination of Ligocki, Kueny, and Ebrahimi Afrouzi teaches “[t]he method of claim 2,” and Ebrahimi Afrouzi further teaches “wherein generating a remediation instruction comprises generating a remediation instruction to cause the robot to adjust a motor of a wheel of the robot or a body part of the robot (FIG. 80 steps 8000 through 8003 depicting motor control based spatial data used to determine movement path; col. 59 lines 27-59 various control actions/instructions implemented including motor control based on multi-sensor data relating to environment characteristics; col. 2 lines 18-42 first and second imaging data combined/fused for an overlapping field of view to generate a spatial representation of the environment (environment characteristics)).”
It would have been obvious to one of ordinary skill in the art before the effective filing date, to have applied Ebrahimi Afrouzi’s teaching of determining environmental spatial data via fused imaging data which, per the grounds for rejecting claim 2, include fusing RGB image and spatial sensor data and using environmental spatial data to generating a remediation instruction to cause the robot to adjust a motor of a wheel of the robot or a body part of the robot to the method taught by Ligocki as modified by Kueny and Ebrahimi Afrouzi, such that in combination the method includes wherein generating the remediation instruction (based on fusion of RGB image and spatial sensor data) comprises generating a remediation instruction to cause the robot to adjust a motor of a wheel of the robot or a body part of the robot.
The motivation would have been to leverage improved environmental spatial data resulting from fusion to consequently improve robot control as suggested by Ebrahimi Afrouzi.
As to claim 22, Ligocki teaches “[a] non-transitory computer readable storage medium storing instructions that, when executed, cause one or more processors (pages 1-2, B. Core Pipeline, paragraph beginning with “At the startup …” through paragraph beginning with “The entire pipeline …” describing processing/program “pipeline” including loading data into memory (entails processor-based information processing); page 2, C. Algorithms, paragraph beginning with “The ‘Algorithms’ module …” further characterizing the method as implemented via data processing code (implemented via processor and instructions that inherently entails storage of instructions in some form of memory/storage)) to:
receive a first series of RGB image data generated by an RGB sensor (page 1, A. Input Data, paragraph beginning with “The data stored as …” describing RGB camera data as input data; pages 3-4, C. Camera-LiDAR Object Detection, paragraph beginning with “LiDAR can measure…” through paragraph beginning with “For this purpose, we have …” describing processing of RGB image data) deployed” [with respect to] “a robot traversing an environment (Abstract describing framework including multi-sensor configuration including RGB camera deployed with respect to an autonomous robot; page 2, C. Outputs, paragraph beginning with “Secondary, there are several …” describing that the agent (robot) travels during a “mapping session” that includes collection of RGB camera data (the RGB data is collected in association with agent travel such that the camera would be deployed on the agent robot)), wherein the first series is a function of a distance traversed by the robot (the RGB camera data would reflect the environment over the path traversed during a “mapping session” including detected features per pages 3-4, C. Camera-LiDAR Object Detection, paragraph beginning with “LiDAR can measure …”);
receiving, by the processor, a second series of spatial distance data generated by a spatial distance sensor (Abstract and page 1, A. Input Data, paragraph beginning with “The data stored as …” describing LiDAR scan data as input data for data fusion; page 3, B. LiDAR data aggregation, paragraphs beginning with “As we are using …” and “The input LiDAR data …” describing obtaining LiDAR data) deployed” [with respect to] “the robot traversing the environment (Abstract describing framework including multi-sensor configuration including 3D LiDAR deployed with respect to an autonomous robot; page 3, B. LiDAR data aggregation, paragraph beginning with “As we are using …” and “The input LiDAR data could come …” describing the scanning function performed during and affected by robot re-positioning (clearly inferring that the scanners are mounted to the robot)), wherein the second series is a function of the distance traversed by the robot (the LiDAR data would reflect the environment over the path traversed during agent repositioning);
fusing together, by the processor, the first series of RGB image data and the second series of spatial distance data (Abstract describing fusion of RGB cameras and 3D LiDAR; page 1, I. Introduction, paragraph beginning with “As a result, our team …” describing fusion of various sensor types (as depicted in FIG. 1); FIG. 1 depicting visualization that combines (fuses) RGB image data and LiDAR data; page 4, C. Camera-LiDAR Object Detection, paragraphs beginning with “For this purpose, we have created …” explaining that the system fuses the LiDAR data and camera detections into a single representation, and paragraph beginning with “There is an estimated …” describing the RGB as the camera source and describing the detected object aspect of the RGB data (frustum color coded as shown in FIG. 1)).”
As set forth above, Ligocki strongly suggests but does not expressly teach that the RGB sensor and spatial distance sensor are mounted “on” the robot.
Kueny discloses a method for performing multi-sensor inspection using a multi-sensor robot (Abstract) that includes using a robot on which multiple imaging sensors are deployed ([0004] and [0023] multi-sensor inspection robot includes a variety of multiple sensor types including 3D LIDAR and optical imaging sensors such as cameras; [0027] and [0029] sensor data may include LIDAR (point cloud) and laser scan (image) obtained by LiDAR unit and laser scanner).
It would have been obvious to one of ordinary skill in the art before the effective filing date, to have applied Kueny’s teaching of deploying multiple imaging sensors onto a mobile robot including LIDAR and optical imaging (e.g., cameras) for implementing multi-sensing inspection to the method taught by Ligocki, such that in combination the method includes deploying the RGB sensor and spatial distance sensor on the robot traversing the environment.
The motivation would have been to effectuate multi-sensor surveying/inspection of an environment as the robot travels through the environment as disclosed by Kueny.
Neither Ligocki nor Kueny appear to expressly teach generating a remediation instruction for execution by the robot traversing the environment based on the fusion of the first series and the second series.
Ebrahimi Afrouzi discloses a method/apparatus for implementing and controlling a robot (Abstract) that includes fusing imaging data from multiple sensors (col. 2 lines 18-28 first and second imaging data combined for an overlapping field of view to generate a combined representation of the environment) deployed on a robot (col. 2 lines 4-16) in which one of the sensors may be an RGB camera (col. 7 lines 3-5; col. 26 lines 49-56) and another may be a spatial distance sensor (col. 7 lines 3-6 LIDAR sensor) and further teaches that that the environment information determined by such fusion may be used to control/remediate actions (inherently entails instruction) performed by the robot (col. 2 lines 28-40 spatial environment data used for estimating a corrected position of the robot which is then used for corrected robot navigation; col. 59 line 21 through col. 60 line 2 sensor determined environmental characteristics used for controlling robot motor).
It would have been obvious to one of ordinary skill in the art before the effective filing date, to have applied Ebrahimi Afrouzi’s teaching of used fused RGB/camera imaging data for remediating robot actions to the method taught by Ligocki as modified by Kueny such that in combination the method includes generating a remediation instruction for execution by the robot traversing the environment based on the fusion of the first series and the second series.
The motivation would have been to leverage the enhanced environmental imaging obtained by fusion to improve robot operations as disclosed by Ebrahimi Afrouzi.
Claims 8-9 and 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Ligocki in view of Kueny and Ebrahimi Afrouzi as applied to claims 2 and 11 above, and further in view of Alzuhiri, Mohand, et al. "IMU-assisted robotic structured light sensing with featureless registration under uncertainties for pipeline inspection." NDT & E International 139 (2023), (Alzuhiri).
As to claim 8, the combination of Ligocki, Kueny, and Ebrahimi Afrouzi teaches “[t]he method of claim 2, further comprising:”
“tracking, by the processor” “a distance moved by the robot within the environment (Ligocki: page 3, A. Precise Positioning, paragraph beginning with “In the beginning, the first GNSS …” describing tracking of agent movement (as depicted in FIG. 2 as tracking position) beginning with the mapping session origin. Examiner notes that such position tracking over a mapping session would entail tracking distance).”
Ligocki does not appear to teach that a processor for tracking distance traveled is incorporated with the robot structure.
Kueny discloses a method for performing multi-sensor inspection using a multi-sensor robot (Abstract) in which the processing for implementing the inspection tasks is a processor attached to the robot (FIG. 5 computer 500 including CPU 510 that per [0042] configured for implementing processing functionality with respect to inspection robot 570; [0037] computer may be deployed within inspection robot 570).
It would have been obvious to one of ordinary skill in the art before the effective filing date, to have applied Kueny’s teaching of using an “on-board” processor attached to the robot for performing processing function associated with the multi-sensor data collection to the method taught by Ligocki as modified by Kueny, which teaches tracking positions and therefore respective robot distances as part of inspection processing, such that in combination the processor that implements the distance tracking is configured to be attached to the robot.
Such a combination would amount to selecting a known design option for deploying processing functionality for a multi-sensor robot to achieve predicable results.
Neither Ligocki nor Kueny appear to teach “receiving, by the processor, data from a motor encoder associated with a wheel attached to the robot;” and tracking “based on the data from the motor encoder,” a distance moved by the robot.
Alzuhiri discloses a method for robotic imaging for pipeline inspection (Abstract) that including using wheel odometry that translates to motor encoding for tracking robot motion and positioning (Abstract and FIG. 3 describing and depicting wheel odometry used for positioning aspect of registration and reconstruction of sensor data; page 2, Introduction, paragraph beginning with “The main information sources for global positioning …” describing encoder data used for estimating distance of robot within pipeline; page 6, 3.3 Wheel Odometry, full paragraph beginning with “Another set of sensors …” through paragraph following equation (26) explaining that a wheel odometry entails an encoder for tracking rotation that is motor-driven (motor rotation translating to wheel rotation and hence motor encoder associated with a wheel)).
It would have been obvious to one of ordinary skill in the art before the effective filing date, to have applied Alzuhiri’s teaching of using a motor encoder for tracking positions/distances traveled by an inspection robot to the method taught by Ligocki as modified by Kueny and Ebrahimi Afrouzi such that in combination the method including using a motor encoder associated with a robot wheel in addition or as an alternative to the satellite and IMU tracking implemented by Ligocki for tracking robot position and distance.
The motivation would have been to provide position/distance information such as in an enclosed space such as a pipe interior in which external positioning (e.g., satellite) may be unavailable or unreliable as suggested by Alzuhiri. Furthermore, such a combination would amount to selecting a known design option for tracking inspection robot position/distance to achieve predictable results.
As to claim 9, the combination of Ligocki, Kueny, Ebrahimi Afrouzi, and Alzuhiri teaches “[t]he method of claim 8, further comprising determining, by an inertial measurement unit associated with the robot (Ligocki: Abstract explaining that multi-sensor data associated with robot may include data from a 3D IMU; page 1, A. Input Data, paragraph beginning with “The data are stored …” describing input data as including data from IMU), a pose and position of the robot (Ligocki: FIG. 2 depicting IMU providing orientation information (roll, pitch) and linear acceleration data (position/location information) for determining position and orientation outputs).”
As to claim 16, the combination of Ligocki, Kueny, and Ebrahimi Afrouzi teaches “[t]he apparatus of claim 11, further comprising:”
“wherein the instructions further cause the apparatus to track,” “a distance moved by the robot within the environment (Ligocki: page 3, A. Precise Positioning, paragraph beginning with “In the beginning, the first GNSS …” describing tracking of agent movement (as depicted in FIG. 2 as tracking position) beginning with the mapping session origin. Examiner notes that such position tracking over a mapping session would entail tracking distance).”
Ligocki does not appear to teach that a processor for tracking distance traveled is incorporated with the robot structure.
Kueny discloses a method/system for performing multi-sensor inspection using a multi-sensor robot (Abstract) in which the processing for implementing the inspection tasks is a processor attached to the robot (FIG. 5 computer 500 including CPU 510 that per [0042] configured for implementing processing functionality with respect to inspection robot 570; [0037] computer may be deployed within inspection robot 570).
It would have been obvious to one of ordinary skill in the art before the effective filing date, to have applied Kueny’s teaching of using an “on-board” processor attached to the robot for performing processing function associated with the multi-sensor data collection to the apparatus taught by Ligocki as modified by Kueny, which teaches tracking positions and therefore respective robot distances as part of inspection processing, such that in combination the processor that implements the distance tracking is configured to be attached to the robot.
Such a combination would amount to selecting a known design option for deploying processing functionality for a multi-sensor robot to achieve predicable results.
Neither Ligocki nor Kueny appear to teach “a motor encoder associated with a wheel attached to the robot, the motor encoder configured to receive data,” and wherein the instructions further cause the apparatus to track, “based on the data received by the motor encoder,” “a distance moved by the robot.
Alzuhiri discloses a method for robotic imaging for pipeline inspection (Abstract) that including using wheel odometry that translates to motor encoding for tracking robot motion and positioning (Abstract and FIG. 3 describing and depicting wheel odometry used for positioning aspect of registration and reconstruction of sensor data; page 2, Introduction, paragraph beginning with “The main information sources for global positioning …” describing encoder data used for estimating distance of robot within pipeline; page 6, 3.3 Wheel Odometry, full paragraph beginning with “Another set of sensors …” through paragraph following equation (26) explaining that a wheel odometry entails an encoder for tracking rotation that is motor-driven (motor rotation translating to wheel rotation and hence motor encoder associated with a wheel)).
It would have been obvious to one of ordinary skill in the art before the effective filing date, to have applied Alzuhiri’s teaching of using a motor encoder for tracking positions/distances traveled by an inspection robot to the apparatus taught by Ligocki as modified by Kueny and Ebrahimi Afrouzi such that in combination the apparatus is configured for using a motor encoder associated with a robot wheel in addition or as an alternative to the satellite and IMU tracking implemented by Ligocki for tracking robot position and distance.
The motivation would have been to provide position/distance information such as in an enclosed space such as a pipe interior in which external positioning (e.g., satellite) may be unavailable or unreliable as suggested by Alzuhiri. Furthermore, such a combination would amount to selecting a known design option for tracking inspection robot position/distance to achieve predictable results.
As to claim 17, the combination of Ligocki, Kueny, Ebrahimi Afrouzi, and Alzuhiri teaches “[t]he apparatus of claim 11, further comprising an inertial measurement unit associated with the robot (Ligocki: Abstract explaining that multi-sensor data associated with robot may include data from a 3D IMU; page 1, A. Input Data, paragraph beginning with “The data are stored …” describing input data as including data from IMU), the inertial measurement unit configured to determine a pose and position of the robot (Ligocki: FIG. 2 depicting IMU providing orientation information (roll, pitch) and linear acceleration data (position/location information) for determining position and orientation outputs).”
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Ligocki in view of Kueny and Ebrahimi Afrouzi as applied to claim 2 above, and further in view of Zhang (US 11,774,983 B1).
As to claim 18, the combination of Ligocki, Kueny, and Ebrahimi Afrouzi teaches “[t]he method of claim 2,” but does not appear to teach the combined features “wherein generating the remediation instruction comprises:
producing, based on the of the first series and the second series, a calibration transformation; and
generating, based on the calibration transformation, a remediation instruction to remediate a difference between data produced by at least one encoder and data produced by at least one inertial measurement unit (IMU).”
Zhang discloses a method/system for controlling multi-sensor autonomous platforms/robots (Abstract; FIG. 9) that include multiple imaging sensors (col. 5 lines 63-67 imaging may include combination of RGB and LidAR) and in which imaging is used to produce a calibration transformation (FIG. 14 depicting steps including using imaging data (blocks 1420, 1425, 1430, 1435) for implementing a calibration that transforms pose data generated by IMU and wheel odometry unit (encoder) into corrected pose data, col. 23 line 60 through col. 24 line 30) that includes a remedial instruction (FIG. 14 block 1455 minimize error to find corrected pose) to remediate a difference between data produced by at least one encoder and data produced by at least one inertial measurement unit (FIG. 14 blocks 1445, 1450, and 1455 difference between pose generated by wheel odometer unit and pose generated based on IMU determined to find/determine corrected pose (remediate); col. 24 lines18-30).
It would have been obvious to one of ordinary skill in the art before the effective filing date, to have applied Zhang’s teaching of using in which imaging is used to produce a calibration transformation that includes a remedial instruction to remediate a difference between data produced by at least one encoder and data produced by at least one inertial measurement unit to the method taught by Ligocki as modified by Kueny and as further modified by Ebrahimi Afrouzi to including using fused RGB imaging and spatial distance sensor data for improved imaging accuracy, such that in combination the method includes producing, based on the of the first series and the second series, a calibration transformation, and generating, based on the calibration transformation, a remediation instruction to remediate a difference between data produced by at least one encoder and data produced by at least one inertial measurement unit.
The motivation would have been to leverage the imaging data (with improved accuracy as provided by the fusion taught by Ebrahimi Afrouzi) to increase accuracy of pose/position finding as disclosed by Zhang.
Claims 20-21 are rejected under 35 U.S.C. 103 as being unpatentable over Ligocki in view of Kueny and Ebrahimi Afrouzi as applied to claim 2 above, and further in view of Zhang (US 2023/0363609 A1), “Zhang ‘609.”
As to claim 20, the combination of Ligocki, Kueny, and Ebrahimi Afrouzi teaches “[t]he method of claim 2,” and Ebrahimi Afrouzi further teaches “wherein generating a remediation instruction comprises generating a remediation instruction to cause the robot to adjust a motor of a wheel of the robot (FIG. 80 steps 8000 through 8003 depicting motor control based spatial data used to determine movement path; col. 59 lines 27-59 various control actions/instructions implemented including motor control based on multi-sensor data relating to environment characteristics; col. 2 lines 18-42 first and second imaging data combined/fused for an overlapping field of view to generate a spatial representation of the environment (environment characteristics)).”
It would have been obvious to one of ordinary skill in the art before the effective filing date, to have applied Ebrahimi Afrouzi’s teaching of determining environmental spatial data via fusing imaging data which, per the grounds for rejecting claim 2, include fusing RGB image and spatial sensor data and using environmental spatial data to generating a remediation instruction to cause the robot to adjust a motor of a wheel of the robot to the method taught by Ligocki as modified by Kueny and Ebrahimi Afrouzi, such that in combination the method includes wherein generating the remediation instruction (based on fusion of RGB image and spatial sensor data) comprises generating a remediation instruction to cause the robot to adjust a motor of a wheel of the robot.
The motivation would have been to leverage improved environmental spatial data resulting from fusion to consequently improve robot control as suggested by Ebrahimi Afrouzi.
Zhang ‘609 discloses a method/system for using 3D point cloud imaging data for guiding/controlling an autonomous vehicle/robot (Abstract; FIG. 3A; FIG. 4) in which the 3D point cloud is generated based on combining/fusing RGB image data and spatial distance sensor data ([0022] 3D point cloud formed by combination of RGB data and depth measuring camera; FIG. 3A occupancy grid map (3D point cloud) generated by combining imaging from time-of-flight camera 1 and RGB camera 2; FIG. 3B RGB camera imaging combined with depth camera imaging via pixel mapping to form RGB-D information, [0092]; [0110] 3D point cloud including combined RGB and depth image data), and which the point cloud data is used to detect wheel slippage ([0308]-[0310] robot’s pose based on point cloud used in combination with wheel odometry to detect wheel slip) and a remedial instruction relating to robot motion is performed in response thereto ([0311]-[0314] describing motion adjustment in response to wheel slip determination).
It would have been obvious to one of ordinary skill in the art before the effective filing date, to have applied Zhang ‘609’s teaching of using point cloud data in the form of fused RGB image data and spatial distance sensor data to detect wheel slippage to the method taught by Ligocki as modified by Kueny and Ebrahimi Afrouzi such that in combination the method includes determining, based on the fusion of the first series and the second series, a slippage of a wheel of the robot.
The motivation would have been to leverage the combined camera image and depth information to enable robot wheel slip detection as disclosed by Zhang ‘609 to optimize robot control.
As to claim 21, the combination of Ligocki, Kueny, and Ebrahimi Afrouzi teaches “[t]he method of claim 2,” and Kueny further teaches that the robot traversing the environment may be a robot traversing a pipeline (Abstract multi-sensor inspection robot traversing pipe interior).
It would have been obvious to one of ordinary skill in the art before the effective filing date, to have applied Kueny’s teaching that a multi-sensor robot for traversing through an environment may be a robot traversing a pipeline to the method taught by Ligocki as modified by Kueny and Ebrahimi Afrouzi such that the method is implemented with respect to such a robot.
Applying the method using such a robot would have been readily recognized by one of ordinary skill as an available design option (controllable multi-sensor robot in the form of a pipeline inspection robot that applies multi-sensor sensing to navigate via images) for implementing the combined method.
Ebrahimi Afrouzi further teaches wherein generating a remediation instruction comprises “generating” “a remediation instruction to remediate” [a] “condition of the robot (FIG. 80 steps 8000 through 8003 depicting motor control based spatial data used to determine movement path; col. 59 lines 27-59 various control actions/instructions implemented including motor control based on multi-sensor data relating to environment characteristics; col. 2 lines 18-42 first and second imaging data combined/fused for an overlapping field of view to generate a spatial representation of the environment (environment characteristics)).”
It would have been obvious to one of ordinary skill in the art before the effective filing date, to have applied Ebrahimi Afrouzi’s teaching of determining environmental spatial data via fusing imaging data which, per the grounds for rejecting claim 2, include fusing RGB image and spatial sensor data and using environmental spatial data to generating a remediation instruction to cause the robot to adjust a condition of the robot to the method taught by Ligocki as modified by Kueny and Ebrahimi Afrouzi, such that in combination the method includes generating a remediation instruction to remediate a condition of the robot.
The motivation would have been to leverage improved environmental spatial data resulting from fusion to consequently improve robot control as suggested by Ebrahimi Afrouzi.
The combination of Ligocki, Kueny, and Ebrahimi Afrouzi do not appear to expressly teach the combined features “determining, based on the fusion of the first series and the second series, a condition of the robot traversing the pipeline, wherein the condition comprises at least one of at least one of abnormal movement of the robot, a tool of the robot has accomplished a task, a slippage of a wheel of the robot, or a drift in at least one of the first series of RGB sensor data or the second series of spatial distance data; and generating, based on the condition of the robot, a remediation instruction to remediate the condition of the robot.”
Zhang ‘609 discloses a method/system for using 3D point cloud imaging data for guiding/controlling an autonomous vehicle/robot (Abstract; FIG. 3A; FIG. 4) in which the 3D point cloud is generated based on combining/fusing RGB image data and spatial distance sensor data ([0022] 3D point cloud formed by combination of RGB data and depth measuring camera; FIG. 3A occupancy grid map (3D point cloud) generated by combining imaging from time-of-flight camera 1 and RGB camera 2; FIG. 3B RGB camera imaging combined with depth camera imaging via pixel mapping to form RGB-D information, [0092]; [0110] 3D point cloud including combined RGB and depth image data), and which the point cloud data is used to detect wheel slippage ([0308]-[0310] robot’s pose based on point cloud used in combination with wheel odometry to detect wheel slip) and a remedial instruction relating to robot motion is performed in response thereto ([0311]-[0314] describing motion adjustment in response to wheel slip determination).
It would have been obvious to one of ordinary skill in the art before the effective filing date, to have applied Zhang ‘609’s teaching of using point cloud data in the form of fused RGB image data and spatial distance sensor data to detect wheel slippage to the method taught by Ligocki as modified by Kueny and Ebrahimi Afrouzi, which teaches generating a remediation instruction to cause the robot to adjust a motor of a wheel of the robot, such that in combination the generation of a remedial instruction includes “determining, based on the fusion of the first series and the second series, a condition of the robot traversing the pipeline, wherein the condition comprises at least one of” “a slippage of a wheel of the robot,” and “generating, based on the condition of the robot, a remediation instruction to remediate the condition of the robot.”
The motivation would have been to leverage the combined camera image and depth information to enable robot wheel slip detection as disclosed by Zhang ‘609 to optimize robot control with respect to wheel slippage.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW W BACA whose telephone number is (571)272-2507. The examiner can normally be reached Monday - Friday 8:00 am - 5:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Schechter can be reached at (571) 272-2302. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW W. BACA/Examiner, Art Unit 2857
/ALEXANDER SATANOVSKY/Primary Examiner, Art Unit 2857
Prosecution Insights