DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This is a Final Office Action on the merits.
Claims 5 and 8-10 were cancelled. Therefore, claims 1-4, 6-7, and 11-20 are pending in the current application.
Response to Amendment/Arguments
Applicant' s arguments filed on 12/15/2025 have been fully considered as below.
Regarding the rejection made under 35 U.S.C. 101, Applicant's amendments and arguments have been fully considered but are not persuasive. Contrary to Applicant’s argument, the language of the amended claim 13 does not explicitly reflect that the computer program product actually comprises a sensor. The sensor in the claim merely describes the control unit that is being used to read the computer program product. Hence, the claim remains directed to a signal per se, See MPEP 2106.03(I). The 101 rejection is maintained as explained below.
Regarding the objections to claims 2-3, 5, 11-12, and 16, Applicant's amendments and arguments have been fully considered and are persuasive. Therefore, the objections are withdrawn. It appears that claims 21-23 are cancelled, see page 5 of Remarks. However, there is no properly marking to indicate the cancellation. Therefore, claims 21-23 are objected as discussed below, and it assumes the claims are cancelled. w
Regarding the objections to drawings, Applicant's amendments and arguments have been fully considered and are persuasive. Therefore, the objections are withdrawn.
Regarding the rejection under 35 U.S.C. 112(b), Applicant's amendments and arguments have been fully considered and are persuasive. Therefore, the objections are withdrawn. However, new issues under this section are discussed below.
Regarding the rejection made under 35 U.S.C. 102 and 103, Applicant's amendments and arguments have been fully considered but are moot in view of the new grounds of rejection provided below, in light of newly found prior art, which was necessitated based on Applicant' s amendments which changed the scope of the claims.
Claim Objections
Regarding claims 21-23, it appears that the claims were not included in the filed amendment on 12/15/2025. Therefore, they are being interpreted as being cancelled. The missing claims are also objected as failure to provide properly marked up claims.
Specification
The disclosure is objected to because of the following informalities: line 17 of page 6, the specification discloses “The trainable agent may also apply the or a second action policy for controlling motions of the robot and/or the end-effector”. It is unclear what “apply the” means.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 13 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because the claim is directed to a signal per se, mere information in the form of data (i.e., a computer program product including a storage readable). See MPEP 2106.03(I).
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 1-4, 6-7, and 11-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 1, the Applicant provides claim limitation, “A method of acquiring sensor data on a construction site by at least one sensor of a mobile construction robot system comprising at least one construction robot, the method comprising controlling the at least one sensor using a trainable agent utilizing a semantic mask that generates an action consisting of a set of weights to assign to semantic classes, wherein a set of weights is assigned to each semantic class,” however, based on currently provided claim language, and given the broadest reasonable interpretation of the currently provided claim language, it is unclear what the metes and bounds regarding the claimed “set of weights” encompasses and further how the claimed “set of weights” are applied to semantic classes and how the claimed “set of weights” are related to an action for controlling the at least one sensor. Moreover, the claimed “a set of weights to assign to semantic classes” and “wherein a set of weights is assigned to each semantic class” are contradict each other. It is unclear whether there is a set of weights to all semantics classes or a set of weights is assigned to each semantic class. In addition, it is unclear whether the claimed “a set of weights” in line 4 is referring to “a set of weights” in line 5. There is insufficient antecedent basis for this limitation in the claim. Therefore, these reasons render the claim indefinite. Appropriate correction and/or clarification is/are required.
Claims 11, 13, and 14 are also rejected for the same reasons since the claims are dependent on claim 1. Appropriate correction and/or clarification is/are required.
The dependent claims are also rejected for being dependent on previous rejected based claim. Appropriate correction and/or clarification is/are required.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim 1-4, 6-7, and 11-20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Metzler et al. (US 20220157136 A1, hereinafter “Metzler”).
Regarding claim 1, Metzler discloses a method of acquiring sensor data on a construction site (Metzler, see at least abstract, Figs. 1, par. [0003, 0031], a method of acquiring sensor data for surveillance of a construction site) by at least one sensor of a mobile construction robot system comprising at least one construction robot (Metzler, see at least abstract, Figs. 1, par. [0287, 0361], sensors 4 are installed at fixed positions and/or are mobile as for example a surveillance drone or the surveillance robot 43 as depicted, which serves as a mobile station for several of the surveillance sensor types mentioned above (e.g. camera, microphone, IR-sensor etc.). The robot 43 equipped with adequate manipulation tools), the method comprising controlling the at least one sensor using a trainable agent (Metzler, see at least par. [0440-0445], a machine learning system based on training data which comprises and classifies the contextual information for each given scene (i.e. region in pixel coordinates)) utilizing a semantic mask (Metzler, see at least Figs. 30-36b, par. [0435-0440], a semantic mask to mask a region of a scene, e.g. top region 90 is masked with an overlaid dot matrix) that generates an action consisting of a set of weights to assign to semantic classes, wherein a set of weights is assigned to each semantic class (Metzler, see at least Figs. 30-36b, par. [0435-0440, 0443], “In a specific embodiment, the contextual information can also provide a preference weighting of different machine learned detection and/or classification attempt. For example, in darkness, a classifier specifically trained on the detection of persons with flashlights will be precedented, whereas in case of illumination, a classifier specifically trained on the detection of persons illumined environments will be precedented”. A top region 90 is masked by the overlaid dot matrix, and the top region 90 can in this example be ignored by all of the detectors as it is very unlikely to find a potential intruder there).
Regarding claim 2, Metzler teaches all the limitations of claim 1 as discussed above. Metzler further teaches including selecting the at least one sensor by the trainable agent (Metzler, see at least par. [0058], “The planning thereby considers for instance choice of sensor, choice of position and/or orientation/direction for measuring resp. interaction or choice of other variables or parameters of a (surveillance) sensor or of the action controller such as measuring accuracy, measuring time or applied power of a robot's actuator”; Fig. 31a-c, 32, 33, par. [0445-0453], the machine learning system based on training data which comprises and classifies the contextual information for selecting an appropriate IR detector/RGB detector/depth detector).
Regarding claim 3, Metzler teaches all the limitations of claim 1 as discussed above. Metzler further teaches including controlling a pose of the at least one sensor using the trainable agent (Metzler, see at least par. [0058], “The planning thereby considers for instance choice of sensor, choice of position and/or orientation/direction for measuring resp. interaction or choice of other variables or parameters of a (surveillance) sensor or of the action controller such as measuring accuracy, measuring time or applied power of a robot's actuator”; Figs. 10, 34a-34c, par. [0333-0340, 0407, 0457-0459], the machine learning system based on training data which comprises and classifies the contextual information for controlling a pose of the at least one sensor based on level of ambiguity, e.g., “The triggered action 117 is e.g. acquisition of another image of door 108 at a second imaging position, different from the first position P10”).
Regarding claim 4, Metzler teaches all the limitations of claim 1 as discussed above. Metzler further teaches including acquiring at least one of image data or depth image data by the at least one sensor (Metzler, see at least par. [0004], “The plurality of surveillance sensors is configured to provide data about one or more facility elements, either via a respective sensor on its own or two or more sensors generating surveillance data in working relationship, suitable for monitoring or observe a state of a respective facility element, and comprise for example one or more RGB camera, depth camera, infrared camera”).
Regarding claim 6, Metzler teaches all the limitations of claim 1 as discussed above. Metzler further teaches comprising at least one of localizing the construction robot, trajectory planning of the construction robot, or mapping of at least a part of the construction site (Metzler, see at least par. [0078], “On the other hand, for navigation purposes, the UGV can be equipped with more sensors, such as e.g. LIDAR (light detection and ranging) and perform for instance LIDAR SLAM (simultaneous localization and mapping) and thus navigate very precisely”; par. [0387], “In particular, the computing unit 218 can be adapted to perform a simultaneous localization and mapping (SLAM) functionality based on the sensor data while moving through the environment”)).
Regarding claim 7, Metzler teaches all the limitations of claim 1 as discussed above. Metzler further teaches including inferring an informativeness measure by the trainable agent (Metzler, see at least Fig. 30, par. [0437], “For example, a top region 90 is indicated by the overlaid dot matrix. This top region 90 can in this example be ignored by all of the detectors as it is very unlikely to find a potential intruder there. This knowledge can contribute to an acceleration of an automated detection and/or classification task by reducing the size of the search region. By excluding this top region 90, also potential false positives in this area can be avoided”).
Regarding claim 11, Metzler discloses a mobile construction robot system comprising a construction robot, at least one sensor for acquiring sensor data (Metzler, see at least abstract, Figs. 1, par. [0003, 0031], a system of acquiring sensor data for surveillance of a construction site; par. [0287, 0361], sensors 4 are installed at fixed positions and/or are mobile as for example a surveillance drone or the surveillance robot 43 as depicted, which serves as a mobile station for several of the surveillance sensor types mentioned above (e.g. camera, microphone, IR-sensor etc.). The robot 43 equipped with adequate manipulation tools)), and a control unit wherein the control unit comprises a trainable agent (Metzler, see at least par. [0005, 0010], “the central computing unit is also configured to establish the criticality classification model and—if present—optionally the normality-anomaly classification model, i.e. to start “blank” and generate or create (and if necessary refine) the classification model on its own, e.g. by machine learning”), wherein the mobile construction robot system is configured to acquire sensor data using the method according to claim 1 (see claim 1 rejection as discussed above).
Regarding claim 12, Metzler teaches all the limitations of claim 11 as discussed above. Metzler further teaches wherein the mobile construction robot comprises at least one of an image sensor or a depth image sensor (Metzler, see at least par. [0004], “The plurality of surveillance sensors is configured to provide data about one or more facility elements, either via a respective sensor on its own or two or more sensors generating surveillance data in working relationship, suitable for monitoring or observe a state of a respective facility element, and comprise for example one or more RGB camera, depth camera, infrared camera”).
Regarding claim 13, Metzler discloses a computer program product including a storage readable by a control unit of a mobile construction robot system (Metzler, see at least Fig. 1, par. [0056], “a computer programme product comprising programme code which is stored on a machine-readable medium, or being embodied by an electromagnetic wave comprising a programme code segment, and having computer-executable instructions for performing, in particular when run on calculation means of a central computing unit of a surveillance system comprising a mobile surveillance robot, the steps of the method”) comprising at least one sensor for acquiring sensor data (Metzler, see at least abstract, Figs. 1, par. [0003, 0031], a system of acquiring sensor data for surveillance of a construction site; par. [0287, 0361], sensors 4 are installed at fixed positions and/or are mobile as for example a surveillance drone or the surveillance robot 43 as depicted, which serves as a mobile station for several of the surveillance sensor types mentioned above (e.g. camera, microphone, IR-sensor etc.). The robot 43 equipped with adequate manipulation tools)), the storage carrying instructions which, when executed by the control unit, cause the construction robot to acquire sensor data using the method according to claim 1, wherein, when executed by the control unit, the at least one sensor is controlled using a trainable agent (Metzler, see at least par. [0440-0445], a machine learning system based on training data which comprises and classifies the contextual information for each given scene (i.e. region in pixel coordinates)) utilizing a semantic mask (Metzler, see at least Figs. 30-36b, par. [0435-0440], a semantic mask to mask a region of a scene, e.g. top region 90 is masked with an overlaid dot matrix) that generates an action consisting of a set of weights to assign to semantic classes, wherein a set of weights is assigned to each semantic class (Metzler, see at least Figs. 30-36b, par. [0435-0440, 0443], “In a specific embodiment, the contextual information can also provide a preference weighting of different machine learned detection and/or classification attempt. For example, in darkness, a classifier specifically trained on the detection of persons with flashlights will be precedented, whereas in case of illumination, a classifier specifically trained on the detection of persons illumined environments will be precedented”. A top region 90 is masked by the overlaid dot matrix, and the top region 90 can in this example be ignored by all of the detectors as it is very unlikely to find a potential intruder there).
Regarding claim 14, Metzler discloses a training method for training the trainable agent of the control unit of the mobile construction robot system according to claim 11, the method comprising training the trainable agent using at least one artificially generated set of sensor data (Metzler, see at least Figs. 42-43b, least par. [0235-0236, 0515-0519], training the machine learning system using synthetic training data by combining multiple virtual objects, preferably in a plurality of different combinations, with the virtual objects themselves varied, under different illumination conditions, with deliberately introduced disturbances, etc.).
Regarding claim 15, Metzler teaches all the limitations of claim 14 as discussed above. Metzler further teaches including introducing noise into the at least one artificially generated set of sensor data (Metzler, see at least par. [0490], “To prevent an over-fitting in the learning process, additional data can be helpful in many embodiments in order to achieve more realistic and robust results. For example, also additional data that is not directly valuable and used to train the classifiers and/or detectors in the direction of the actually intended purpose, such a noise, disturbances, etc. can be subjoined in the training data, e.g. at random”).
Regarding claim 16, Metzler teaches all the limitations of claim 14 as discussed above. Metzler further teaches including controlling a pose of the at least one sensor using the trainable agent (Metzler, see at least par. [0058], “The planning thereby considers for instance choice of sensor, choice of position and/or orientation/direction for measuring resp. interaction or choice of other variables or parameters of a (surveillance) sensor or of the action controller such as measuring accuracy, measuring time or applied power of a robot's actuator”; Figs. 10, 34a-34c, par. [0333-0340, 0407, 0457-0459], the machine learning system based on training data which comprises and classifies the contextual information for controlling a pose of the at least one sensor based on level of ambiguity, e.g., “The triggered action 117 is e.g. acquisition of another image of door 108 at a second imaging position, different from the first position P10”).
Regarding claim 17, Metzler teaches all the limitations of claims 1 and 2 as discussed above. Metzler further teaches including acquiring at least one of image data or depth image data by the at least one sensor (Metzler, see at least par. [0004], “The plurality of surveillance sensors is configured to provide data about one or more facility elements, either via a respective sensor on its own or two or more sensors generating surveillance data in working relationship, suitable for monitoring or observe a state of a respective facility element, and comprise for example one or more RGB camera, depth camera, infrared camera”).
Regarding claim 18, Metzler teaches all the limitations of claims 1 and 2 as discussed above. Metzler further teaches comprising semantic classification (Metzler, see at least par. [0027], “Optionally, the classification is based on a class based on a semantic property”).
Regarding claim 19, Metzler teaches all the limitations of claims 1 and 2 as discussed above. Metzler further teaches comprising at least one of localizing the construction robot, trajectory planning of the construction robot, or mapping of at least a part of the construction site (Metzler, see at least par. [0078], “On the other hand, for navigation purposes, the UGV can be equipped with more sensors, such as e.g. LIDAR (light detection and ranging) and perform for instance LIDAR SLAM (simultaneous localization and mapping) and thus navigate very precisely”; par. [0387], “In particular, the computing unit 218 can be adapted to perform a simultaneous localization and mapping (SLAM) functionality based on the sensor data while moving through the environment”).
Regarding claim 20, Metzler teaches all the limitations of claims 1 and 2 as discussed above. Metzler further teaches including inferring an informativeness measure by the trainable agent (Metzler, see at least Fig. 30, par. [0437], “For example, a top region 90 is indicated by the overlaid dot matrix. This top region 90 can in this example be ignored by all of the detectors as it is very unlikely to find a potential intruder there. This knowledge can contribute to an acceleration of an automated detection and/or classification task by reducing the size of the search region. By excluding this top region 90, also potential false positives in this area can be avoided”).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TRANG DANG whose telephone number is (703)756-1049. The examiner can normally be reached Monday-Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Khoi Tran can be reached at (571)272-6919. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TRANG DANG/Examiner, Art Unit 3656 /KHOI H TRAN/Supervisory Patent Examiner, Art Unit 3656