DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claims 1-20 are pending and examined below. This action is in response to the claims filed 7/6/26.
Response to Amendment
Applicant’s arguments, see Applicant Remarks 35 USC § 103. filed on 7/6/26, regarding 35 USC § 103 rejections are persuasive in view of amendments filed 7/6/26.
However, upon further consideration, new grounds of rejection are made in view of Cohen et al. (US 10,898,999) below.
Claim Rejections - 35 USC § 103
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claims 1-6, 8-13, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over D’Ercoli et al. (US 2020/0189103) in view of Cohen et al. (US 10,898,999).
Regarding claims 1, 8, and 15, D’Ercoli discloses an automated guidance vehicle system for industrial environments including a bystander control system for a material handling vehicle (MHV) having a sensor and a vehicle control system (VCS), the bystander control system comprising (¶16-20 – AGV corresponding to the recited MHV including sensors and a PLC to control operation of the vehicle corresponding to the recited VCS):
an automation processing system (APS) having a processor and a memory, the APS coupled with the sensor and the VCS and having a machine learning control program stored in the memory, wherein the APS is configured to (¶16-20 and ¶92-94 - multi-channel virtual sensor 126 corresponding to the recited APS including a processor and a memory with multiple machine learning models corresponding to the recited machine learning control program stored in the memory):
receive sensor data based on the sensor output indicative of the environment proximate to the MHV (¶16-20 – receiving data from the cameras indicative of the environment proximate to the MHV);
process the sensor data, using the machine learning control program, wherein the machine learning control program comprises a first trained machine learning model configured to determine a bystander presence and a second trained machine learning model configured to determine a bystander pose (¶19 and ¶36-44 – feature extraction sub-module 316b/316c corresponding to the recited first trained machine learning model which extracts shape information from the sensor data which is used to identify cand classify objects corresponding to the recited determining the presence of a bystander and feature extraction sub-module 316d/316e corresponding to the recited second trained machine learning model which estimates proximity between distinct objects in a scene corresponding to the recited determine bystander pose. Since the feature extraction module 130 may be configured to generate feature data or interaction predictors 131 from multiple machine learning models, feature extraction sub-module 316b/316c and feature extraction sub-module 316d/316e are interpreted as different machine learning models);
While D’Ercoli discloses numerous sub-modules each consisting of different machine learning models to detect, identify, classify, and then determine finer details such as pose, it does not explicitly disclose using a first ML model to determine a bystander presence based on sensor data then using a second ML model to determine bystander pose based on an output from the first ML model, however Cohen discloses a system for automated human-robot interaction including a first trained machine learning model configured to determine a bystander presence based on the sensor data and a second trained machine learning model configured to determine a bystander pose based on an output of the first trained machine learning model (Col 10:63-11:43 – person detection model corresponding to the recited first trained machine learning model configured to determine a bystander presence based on the sensor data which then directs the pose estimation model corresponding to the recited second trained machine learning model to determine a bystander pose for each detected person corresponding to the recited based on an output of the first trained machine learning model);
generate, based on the determined bystander pose, an output comprising an indication of a control action for the MHV (Col 13:31-14:9 – using the features identified for the detected people including pose estimations, the selection module determines an action for the robot corresponding to the recited generate an output comprising an indication of a control action for the MHV); and
send the generated output to the VCS (Col 10:1-18 – robot may provide information regarding an interaction with a person to a server corresponding to the recited send the generated output to the VCS).
The combination of the numerous sub-modules each consisting of different machine learning models to detect, identify, classify, and then determine finer details such as pose of D’Ercoli with the human specific detection and pose estimation models of Cohen fully discloses the elements as claimed.
It would have been obvious to one of ordinary skill in the art before the filing date to have combined the numerous sub-modules each consisting of different machine learning models to detect, identify, classify, and then determine finer details such as pose of D’Ercoli with the human specific detection and pose estimation models of Cohen in order to increase likelihood of task completion by the robot and reduce the amount of time to complete tasks based on complex multi-step plans in human environments (Cohen – Col 5:18-34).
Regarding claims 2, 3, 4, 9, 10, and 11, D’Ercoli further discloses wherein the first trained machine learning model comprises a bystander detection machine learning model stored in the memory of the APS and the second trained machine learning model comprises a bystander pose estimation machine learning model stored in the memory of the APS (¶19, ¶36-44, and Fig. 3 - feature extraction sub-module 316b/316c corresponding to the recited first trained machine learning model and feature extraction sub-module 316d/316e corresponding to the recited second trained machine learning model where the feature extraction sub-modules are part of the multi-channel virtual sensor 126 corresponding to the recited APS including a processor and a memory therefore the models are stored in the APS),
wherein: processing the sensor data comprises: determining, using the first trained machine learning model, whether a bystander is in proximity of the MHV; and determining, using the second trained machine learning model, the pose of the bystander (¶19 and ¶32-44 – feature extraction sub-module 316b/316c corresponding to the recited first trained machine learning model determines blobs/shapes of the objects corresponding to the recited determining whether a bystander is in proximity of the MHV and feature extraction sub-module 316d/316e corresponding to the recited second trained machine learning model estimates proximity between distinct objects in a scene and specific features typically associated with human body parts, such as faces, hands and legs corresponding to the recited the pose of the bystander); and
the APS is configured to generate the output based on the determined bystander pose (¶17 - in the event of detecting that a human, robot, or object enters an environment in which the robot 102, which may be stationary or mobile, is operating, the safety PLC 120 may output command data 122 to the robot 102 to slow or stop operation thereof so as to avoid a collision).
Regarding claims 5 and 12, D’Ercoli further discloses wherein the APS is configured to send an indication of the determined bystander pose to a second MHV (¶19-20 – when risky robot-to-human interactions are determined, the AGV may communicate control data, informational data, classification data, etc. corresponding to the recited determined bystander pose to any or all of the control PLC[s] corresponding to the recited a second MHV given each MHV is controlled by a PLC).
Regarding claims 6 and 13, D’Ercoli further discloses wherein the sensor comprises a first sensor of a first type and a second sensor of a second type, and wherein generating sensor data indicative of the environment proximate to the MHV comprises generating a first sensor data based on the output of the first sensor and a second sensor data based on the output of the second sensor (¶25 – cameras may include 2D cameras, 3D cameras, or any other type of optical or spectral camera corresponding to the recited first sensor type and second sensor type which respectively generate different sensor data of the environment).
Claims 7, 14, and 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over D’Ercoli et al. (US 2020/0189103) in view of Cohen et al. (US 10,898,999), as applied to claims 1 and 8 above, in view of Hashimoto et al. (US 2021/0272434).
Regarding claims 7 and 14, D’Ercoli does not explicitly disclose comparing sensor data to a predetermined map however Hashimoto discloses a smart navigation system including wherein generating the sensor data indicative of the environment proximate to the MHV comprises comparing an output of the sensor to a predetermined map of the environment (¶66 - the detection unit 2 may be configured to detect the presence of the moving object 9 based on a comparison between a detection result Da of the ranging sensor 74 and the map information M of the target area).
The combination of the system for detecting and identifying the presence and pose of a bystander of D’Ercoli in view of Cohen with the map discrepancy based moving object analysis of Hashimoto fully discloses the elements as claimed.
It would have been obvious to one of ordinary skill in the art before the filing date to have combined the system for detecting and identifying the presence and pose of a bystander of D’Ercoli in view of Cohen with the map discrepancy based moving object analysis of Hashimoto in order to monitor large areas for suspicious persons without labor costs associated with human patrols and blind spots generated with cameras in a known environment (Hashimoto - ¶3).
Regarding claims 16 and 19, while D’Ercoli does disclose detecting and identifying the presence and pose of a bystander, it does not explicitly utilize an identified discrepancy between sensor and map data.
However, Hashimoto further discloses wherein processing the sensor data comprises: receiving, by the first trained machine learning model, an identified discrepancy between the sensor data and a predetermined map of an environment of the MHV (¶118 and Fig. 7 – S2 corresponding to the recited identifying a discrepancy between the sensor data and a predetermined map of an environment of the MHV); and
determining, by the first trained machine learning model, whether the identified discrepancy is a bystander (¶118-119 and Fig. 7 – S3 corresponding to the recited determining whether the identified discrepancy is a bystander).
The combination of the system for detecting and identifying the presence and pose of a bystander of D’Ercoli in view of Cohen with the map discrepancy based moving object analysis of Hashimoto fully discloses the elements as claimed.
It would have been obvious to one of ordinary skill in the art before the filing date to have combined the system for detecting and identifying the presence and pose of a bystander of D’Ercoli in view of Cohen with the map discrepancy based moving object analysis of Hashimoto in order to monitor large areas for suspicious persons without labor costs associated with human patrols and blind spots generated with cameras in a known environment (Hashimoto - ¶3).
Regarding claims 17 and 20, D’Ercoli further discloses wherein processing the sensor data comprises: in response to a determination by the first trained machine learning model that the [object] is a bystander, outputting an indication of the bystander to the second trained machine learning model (¶19 and ¶32-44 – feature extraction sub-module 316b/316c corresponding to the recited first trained machine learning model determines blobs/shapes of the objects corresponding to the recited determining whether a bystander is in proximity of the MHV and feature extraction sub-module 316d/316e corresponding to the recited second trained machine learning model estimates proximity between distinct objects in a scene and specific features typically associated with human body parts, such as faces, hands and legs corresponding to the recited the pose of the bystander where the results of the feature extraction sub-module 316b/316c corresponding to the recited first trained machine learning model is used in the feature extraction sub-module 316d/316e corresponding to the recited second trained machine learning model); and
in response to a determination by the trained machine learning model that the [object] does not correspond to a bystander, identifying the discrepancy as a potential obstacle and register at least one of a position or a size of the obstacle (¶19, ¶36-44, ¶84-87, and Fig. 3 – each time a decision is to be made, an historical database may be maintained inside an ensemble classifier to update an historical database by updating a row or adding a new row or record in the database where the classifier modules compile the results of the determinations including whether the object is a human or not and its extracted features such as position and size and the resulting features are stored for machine learning purposes).
While D’Ercoli does disclose detecting and identifying the presence and pose of a bystander, it does not explicitly utilize an identified discrepancy.
However, Hashimoto further discloses in response to S3 identifying the discrepancy as a bystander, proceed to S4 to confirm position and behavior of the person corresponding to the recited outputting to the second trained machine learning model to determine pose information (¶118-119 and Fig. 7).
The combination of the system for detecting and identifying the presence and pose of a bystander of D’Ercoli in view of Coehn with the map discrepancy based moving object analysis of Hashimoto fully discloses the elements as claimed.
It would have been obvious to one of ordinary skill in the art before the filing date to have combined the system for detecting and identifying the presence and pose of a bystander of D’Ercoli in view of Cohen with the map discrepancy based moving object analysis of Hashimoto in order to monitor large areas for suspicious persons without labor costs associated with human patrols and blind spots generated with cameras in a known environment (Hashimoto - ¶3).
Regarding claim 18, D’Ercoli further discloses wherein the APS is configured to: in response to a determination by the first trained machine learning model that the discrepancy corresponds to the bystander, isolate a portion of the sensor data that includes the discrepancy (¶19, ¶36-46, ¶84-87, and Fig. 3 – each time a decision is to be made, an historical database may be maintained inside an ensemble classifier to update an historical database by updating a row or adding a new row or record in the database where the extracted features are saved as training data corresponding to the recited isolate a portion of the sensor data that includes the discrepancy).
While D’Ercoli does disclose detecting and identifying the presence and pose of a bystander, it does not explicitly utilize an identified discrepancy.
However, Hashimoto further discloses in response to S3 identifying the discrepancy as a bystander, proceed to S4 to confirm position and behavior of the person corresponding to the recited outputting to the second trained machine learning model to determine pose information (¶118-119 and Fig. 7).
The combination of the system for detecting and identifying the presence and pose of a bystander of D’Ercoli in view of Cohen with the map discrepancy based moving object analysis of Hashimoto fully discloses the elements as claimed.
It would have been obvious to one of ordinary skill in the art before the filing date to have combined the system for detecting and identifying the presence and pose of a bystander of D’Ercoli in view of Cohen with the map discrepancy based moving object analysis of Hashimoto in order to monitor large areas for suspicious persons without labor costs associated with human patrols and blind spots generated with cameras in a known environment (Hashimoto - ¶3).
Additional References Cited
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Van Oldenborgh et al. (US 2021/0233234) discloses a system of using a second machine learning model for detecting the pose of an identified baby (¶228).
Francesca (US 2022/0198802) discloses a system of using distinctly separate object neural network models and human pose detection models for increased efficiency and reliability (¶11-14).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Matthew J Reda whose telephone number is (408)918-7573. The examiner can normally be reached Monday - Friday 7-4 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hunter Lonsberry can be reached at (571) 272-7298. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW J. REDA/ Primary Examiner, Art Unit 3665
Prosecution Insights