DETAIL ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Notice on Prior Art Rejections
2. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Status of Claims
3. This Office Action is in response to the Applicant's application filed October 14, 2024. Claims 1-20 are presently pending and are presented for examination.
Claim Rejections - 35 USC § 101
4. 35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
5. Claim 1-9 is/are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter because the claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter.
6. Claim 1 requires a non-transitory computer readable store medium, which stores a program (See claim 19). The specification does not set forth what constitutes a non-transitory computer readable store medium, and therefore, in view of the ordinary and customary meaning of computer readable media and in accordance with the broadest reasonable interpretation of the claim, said medium could be directed towards a transitory propagating signal per se and considered to be non-statutory subject matter. See In re Nuijten, 500 F.3d 1346, 1356-57 (Fed. Cir. 2007) and Interim Examination Instructions for Evaluating Subject Matter Eligibility Under 35 U.S.C. 101, Aug 24, 2009, p. 2. Claims that recite nothing but the physical characteristics of a form of energy, such as a frequency, voltage, or the strength of a magnetic field, define energy or magnetism, per se, and as such are nonstatutory natural phenomena. O'Reilly, 56 U.S. (15 How.) at 112-14. Moreover, it does not appear that a claim reciting a signal encoded with functional descriptive material falls within any of the categories of patentable subject matter set forth in §101. Please refer to MPEP 2111.01 and the USPTO’s “Subject Matter Eligibility of Computer Readable Media” memorandum dated January 26, 2010.
7. Claims 2-9 depend from claim 1 and therefore include the same limitation as claim 1 so they are rejected for the same reason.
Claim Rejections - 35 USC § 103
8. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
9. Claims 1-20 are rejected under 35 U.S.C 103 as being unpatentable over Sajjan et al, US 2023/0213945, in view of Tremblay et al. US 2021/0390653, hereinafter referred to as Sajjan and Tremblay, respectively.
Regarding claim 1, Sajjan discloses a system, comprising: a non-transitory memory; and one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations (See at least fig 1-11, ¶ 26, 52, 63, 87, 206, 207, “The computer-storage media may include both volatile and nonvolatile media and/or removable and nonremovable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, and/or other data types”) comprising:
receiving, at a plurality of neural network models, data from a plurality of sensors associated with a robotic device (See at least fig 1-11, ¶ 28, 30, 33, 35, 40, 41, 43, 198, 226, 27, (“the process 100 may include one or more machine learning model(s) 104 receiving one or more inputs…the sensor data 102 may include other types of sensor data used for obstacle to path assignments, such as LiDAR data, SONAR data, RADAR data, and/or the like---e.g., as generated by one or more sensors”), wherein each neural network model of the plurality of neural network models receives a subset of the data from a different subset of sensors of the plurality of sensors (See at least fig 1-11, ¶ 28, 29, 71, 100, 107, 141, 42, “using deep neural networks (DNNs), and specifically convolutional neural networks (CNN s ), as the machine learning model(s) 104, this is not intended to be limiting. For example, and without limitation, the machine learning model( s) 104 may include any type of machine learning model, such as a machine learning model(s)”);
generating, by the plurality of neural network models and based on the data from the plurality of sensors, a plurality of outputs, wherein each output of the plurality of outputs is generated by a particular neural network model of the plurality of neural network models, and wherein each output corresponds to a respective subset of data received by the particular neural network model of the plurality of neural network models (See at least fig 1-11, ¶ 26, 27, 58, 78, 89, 90, 101, 40, “the machine learning model(s) 104 may take as input the sensor data 102 ( e.g., an image, a point cloud, a range image, etc.), the rasterized paths 116, and the rasterized objects 118 and may compute outputs including the one or more output channels 106. The one or more output channels 106 may include, in non-limiting embodiments, one output channel 106 per path”);
selecting an output from the plurality of outputs, the output comprising data for controlling the robotic device (See at least fig 1-11, ¶ 5, 22, 23, 24, 26, 27, 32, 33, 34, 4, “obstacle to path assignment for autonomous machine systems and applications. Systems and methods are disclosed that use one or more output channels of a deep neural network (DNN) to determine assignments of obstacles ( e.g., vehicles, bicyclists, pedestrians, etc.) to paths”); and
controlling the robotic device based on the output (See at least fig 1-11, ¶ 100, 115, 141, 159, 188, 99, “The controller(s) 936 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals, and output operation commands ( e.g., signals representing commands) to enable autonomous driving and/or to assist a human driver in driving the vehicle”).
Sajjan fails to explicitly disclose a plurality of outputs, wherein each output of the plurality of outputs is generated by a particular neural network model of the plurality of neural network models.
However, Tremblay teaches a plurality of outputs, wherein each output of the plurality of outputs is generated by a particular neural network model of the plurality of neural network models (See at least fig 1-11, ¶ 17, 24, 26, 30, 71, 82, 29, “a perception network 302 can be a deep neural network that accepts the demonstration data captured of the performance, such as may include image, distance, and other data as discussed herein…a plan generation network 304, which can be another deep neural network that can process the percepts to generate, or infer, a human-readable plan corresponding to the task…an execution network 306, which can be a deep neural network capable of processing data for the plan and generating, or inferring, one or more robot-readable instructions”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Sajjan and include controlling, by the high-level controller, all load outputs as taught by Tremblay because it would a system to have a set of neural networks to perform the tasks associated with perception, program generation, and program execution (Tremblay ¶ 17).
Regarding claim 2, Sajjan discloses the system of claim 1, wherein the output comprises steering data and throttle data (See at least fig 1-11, ¶ 99, 100, 113, 153, 184, 96 “The propulsion system 950 may be controlled in response to receiving signals from the throttle/accelerator 952”), (See at least fig 1-11, ¶ 99, 97 “A steering system 954, which may include a steering wheel, may be used to steer the vehicle 900”).
Regarding claim 3, Sajjan discloses the system of claim 1, wherein the operations further comprise: converting the output into control signals; and controlling the robotic device based on the control signals (See at least fig 1-11, ¶ 99, 100, 113, 153, 184, 96 “The propulsion system 950 may be controlled in response to receiving signals from the throttle/accelerator 952”).
Regarding claim 4, Sajjan discloses the system of claim 1, wherein the subset of data received by respective first and second models of the plurality of neural network models is different (See at least fig 1-11, ¶ 65, 198, 226, 42, “the machine learning model(s) 104 may include any type of machine learning model, such as a machine learning model(s) using linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k-nearest neighbor (Knn), K means clustering, random forest, dimensionality reduction algorithms,”).
Regarding claim 5, Sajjan discloses the system of claim 1, wherein the data is received from the plurality of sensors in real-time or at predefined time intervals (See at least fig 1-11, ¶ 131, 137, 147, 155, 199, 200, 201, 68, “In addition, by increasing processing speeds, the run-time of the system may be decreased thereby enabling real-time or near real-time deployment of the process 600 within the system”).
Regarding claim 6, Sajjan discloses the system of claim 1, wherein the selecting the output is performed by an agent module, in communication with the plurality of neural network models, based on a target task for the robotic device (See at least fig 1-11, ¶ 100, 115, 141, 159, 188, 99, “The controller(s) 936 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals, and output operation commands ( e.g., signals representing commands) to enable autonomous driving and/or to assist a human driver in driving the vehicle”).
Regarding claim 7, Sajjan discloses the system of claim 1, wherein the plurality of neural network models are configured to perform a plurality of different tasks associated with the controlling the robotic device (See at least fig 1-11, ¶ 42, 43, 47, 50, 126, 127, 128, 129, 156, 99, “Autonomous and semi-autonomous driving systems, as well as advanced driver assistance systems (ADAS) use sensors, such as cameras, LiDAR, RADAR, etc., to perform various tasks-such as lane keeping, lane changing, lane assigmnent, camera calibration, and localization”).
Regarding claim 8, Sajjan discloses the system of claim 1, wherein two or more models of the plurality of neural network models are configured to perform a same task associated with the controlling the robotic device, and wherein the selecting the output is performed by an agent module, in communication with the plurality of neural network models, based on a performance metric for each of the two or more models for performing the same task (See at least fig 1-11, ¶ 42, 43, 47, 50, 126, 127, 128, 129, 156, 99, 226, 141, “The DLA may be used to run any type of network to enhance control and driving safety, including for example, a neural network that outputs a measure of confidence for each object detection. Such a confidence value may be
interpreted as a probability, or as providing a relative "weight" of each detection compared to other detections. This confidence value enables the system to make further decisions regarding which detections should be considered”).
Regarding claim 9, Sajjan discloses the system of claim 1, wherein at least some of the plurality of neural network models comprise models trained using a plurality of data augmentation transformations (See at least fig 1-11, ¶ 10, 11, 22, 28, 29, 30, 35, 53, 6, “to increase the size of the training data set to account for less common scenarios or path detection inaccuracies, data augmentation may be used. For example, lane boundaries may be demarcated using points, and these points may be used to generate polylines that may define the boundaries of lanes”).
Regarding claim 10, Sajjan discloses the system of claim 6, wherein a neural network engine comprises the plurality of neural network models and the agent module, and wherein the robotic device comprises the neural network engine (See at least fig 1-11, ¶ 42, 43, 47, 50, 61, 126, 127, 128, 129, 132, 145, 146, 147, 153, 156, 99, 41, “The output channels 106 may then be compared to the GT output channels 120A-120N using training engine 108.”).
Regarding claim 11, Sajjan discloses the system of claim 6, wherein a neural network engine comprises the plurality of neural network models and the agent module, and wherein the robotic device is configured to communicate with the neural network engine over a network (See at least fig 1-11, ¶ 42, 43, 47, 50, 61, 126, 127, 128, 129, 132, 145, 146, 147, 153, 156, 99, 41, 113, “The bus 902 may include a Controller Area Network (CAN) data interface (alternatively referred to herein as a "CAN bus"). A CAN may be a network inside the vehicle 900 used to aid in control of various features and functionality of the vehicle 900”).
Regarding claim 12, Sajjan discloses a method, comprising:
receiving, at a plurality of neural network models, data from a plurality of sensors associated with a robotic device (See at least fig 1-11, ¶ 28, 30, 33, 35, 40, 41, 43, 198, 226, 27, (“the process 100 may include one or more machine learning model(s) 104 receiving one or more inputs…the sensor data 102 may include other types of sensor data used for obstacle to path assignments, such as LiDAR data, SONAR data, RADAR data, and/or the like---e.g., as generated by one or more sensors”), wherein each neural network model of the plurality of neural network models receives a subset of the data from a different subset of sensors of the plurality of sensors (See at least fig 1-11, ¶ 28, 29, 71, 100, 107, 141, 42, “using deep neural networks (DNNs), and specifically convolutional neural networks (CNN s ), as the machine learning model(s) 104, this is not intended to be limiting. For example, and without limitation, the machine learning model( s) 104 may include any type of machine learning model, such as a machine learning model(s)”);
generating, by the plurality of neural network models and based on the data from the plurality of sensors, a plurality of outputs, wherein each output of the plurality of outputs is generated by a particular neural network model of the plurality of neural network models, and wherein each output corresponds to a respective subset of data received by the particular neural network model of the plurality of neural network models (See at least fig 1-11, ¶ 26, 27, 58, 78, 89, 90, 101, 40, “the machine learning model(s) 104 may take as input the sensor data 102 ( e.g., an image, a point cloud, a range image, etc.), the rasterized paths 116, and the rasterized objects 118 and may compute outputs including the one or more output channels 106. The one or more output channels 106 may include, in non-limiting embodiments, one output channel 106 per path”);
selecting an output from the plurality of outputs, the output comprising data for controlling the robotic device (See at least fig 1-11, ¶ 5, 22, 23, 24, 26, 27, 32, 33, 34, 4, “obstacle to path assignment for autonomous machine systems and applications. Systems and methods are disclosed that use one or more output channels of a deep neural network (DNN) to determine assignments of obstacles ( e.g., vehicles, bicyclists, pedestrians, etc.) to paths”); and
controlling the robotic device based on the output (See at least fig 1-11, ¶ 100, 115, 141, 159, 188, 99, “The controller(s) 936 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals, and output operation commands ( e.g., signals representing commands) to enable autonomous driving and/or to assist a human driver in driving the vehicle”).
Sajjan fails to explicitly disclose a plurality of outputs, wherein each output of the plurality of outputs is generated by a particular neural network model of the plurality of neural network models.
However, Tremblay teaches a plurality of outputs, wherein each output of the plurality of outputs is generated by a particular neural network model of the plurality of neural network models (See at least fig 1-11, ¶ 17, 24, 26, 30, 71, 82, 29, “a perception network 302 can be a deep neural network that accepts the demonstration data captured of the performance, such as may include image, distance, and other data as discussed herein…a plan generation network 304, which can be another deep neural network that can process the percepts to generate, or infer, a human-readable plan corresponding to the task…an execution network 306, which can be a deep neural network capable of processing data for the plan and generating, or inferring, one or more robot-readable instructions”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Sajjan and include controlling, by the high-level controller, all load outputs as taught by Tremblay because it would a system to have a set of neural networks to perform the tasks associated with perception, program generation, and program execution (Tremblay ¶ 17).
Regarding claim 13, Sajjan discloses the method of claim 12, wherein the output comprises steering data and throttle data (See at least fig 1-11, ¶ 99, 100, 113, 153, 184, 96 “The propulsion system 950 may be controlled in response to receiving signals from the throttle/accelerator 952”), (See at least fig 1-11, ¶ 99, 97 “A steering system 954, which may include a steering wheel, may be used to steer the vehicle 900”).
Regarding claim 14, Sajjan discloses the method of claim 12, further comprising: converting the output into control signals; and controlling the robotic device based on the control signals (See at least fig 1-11, ¶ 99, 100, 113, 153, 184, 96 “The propulsion system 950 may be controlled in response to receiving signals from the throttle/accelerator 952”).
Regarding claim 15, Sajjan discloses the method of claim 12, wherein the subset of data received by respective first and second models of the plurality of neural network models is different (See at least fig 1-11, ¶ 65, 198, 226, 42, “the machine learning model(s) 104 may include any type of machine learning model, such as a machine learning model(s) using linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k-nearest neighbor (Knn), K means clustering, random forest, dimensionality reduction algorithms,”).
Regarding claim 16, Sajjan discloses the method of claim 12, wherein the plurality of neural network models are configured to perform a plurality of different tasks associated with the controlling the robotic device (See at least fig 1-11, ¶ 42, 43, 47, 50, 126, 127, 128, 129, 156, 99, “Autonomous and semi-autonomous driving systems, as well as advanced driver assistance systems (ADAS) use sensors, such as cameras, LiDAR, RADAR, etc., to perform various tasks-such as lane keeping, lane changing, lane assigmnent, camera calibration, and localization”).
Regarding claim 17, Sajjan discloses the method of claim 12, wherein at least some of the plurality of neural network models comprise models trained using a plurality of data augmentation transformations (See at least fig 1-11, ¶ 10, 11, 22, 28, 29, 30, 35, 53, 6, “to increase the size of the training data set to account for less common scenarios or path detection inaccuracies, data augmentation may be used. For example, lane boundaries may be demarcated using points, and these points may be used to generate polylines that may define the boundaries of lanes”).
Regarding claim 18, Sajjan discloses the method of claim 12, wherein a neural network engine comprises the plurality of neural network models and an agent module, and wherein the robotic device comprises the neural network engine or is configured to communicate with the neural network engine over a network (See at least fig 1-11, ¶ 42, 43, 47, 50, 61, 126, 127, 128, 129, 132, 145, 146, 147, 153, 156, 99, 41, 113, “The bus 902 may include a Controller Area Network (CAN) data interface (alternatively referred to herein as a "CAN bus"). A CAN may be a network inside the vehicle 900 used to aid in control of various features and functionality of the vehicle 900”).
Regarding claim 19, Sajjan discloses a non-transitory machine-readable medium having stored thereon machine-readable instructions executable to cause a machine to perform operations comprising:
receiving, at a plurality of neural network models, data from a plurality of sensors associated with a robotic device (See at least fig 1-11, ¶ 28, 30, 33, 35, 40, 41, 43, 198, 226, 27, (“the process 100 may include one or more machine learning model(s) 104 receiving one or more inputs…the sensor data 102 may include other types of sensor data used for obstacle to path assignments, such as LiDAR data, SONAR data, RADAR data, and/or the like---e.g., as generated by one or more sensors”), wherein each neural network model of the plurality of neural network models receives a subset of the data from a different subset of sensors of the plurality of sensors (See at least fig 1-11, ¶ 28, 29, 71, 100, 107, 141, 42, “using deep neural networks (DNNs), and specifically convolutional neural networks (CNN s ), as the machine learning model(s) 104, this is not intended to be limiting. For example, and without limitation, the machine learning model( s) 104 may include any type of machine learning model, such as a machine learning model(s)”);
generating, by the plurality of neural network models and based on the data from the plurality of sensors, a plurality of outputs, wherein each output of the plurality of outputs is generated by a particular neural network model of the plurality of neural network models, and wherein each output corresponds to a respective subset of data received by the particular neural network model of the plurality of neural network models (See at least fig 1-11, ¶ 26, 27, 58, 78, 89, 90, 101, 40, “the machine learning model(s) 104 may take as input the sensor data 102 ( e.g., an image, a point cloud, a range image, etc.), the rasterized paths 116, and the rasterized objects 118 and may compute outputs including the one or more output channels 106. The one or more output channels 106 may include, in non-limiting embodiments, one output channel 106 per path”);
selecting an output from the plurality of outputs, the output comprising data for controlling the robotic device (See at least fig 1-11, ¶ 5, 22, 23, 24, 26, 27, 32, 33, 34, 4, “obstacle to path assignment for autonomous machine systems and applications. Systems and methods are disclosed that use one or more output channels of a deep neural network (DNN) to determine assignments of obstacles ( e.g., vehicles, bicyclists, pedestrians, etc.) to paths”); and
controlling the robotic device based on the output (See at least fig 1-11, ¶ 100, 115, 141, 159, 188, 99, “The controller(s) 936 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals, and output operation commands ( e.g., signals representing commands) to enable autonomous driving and/or to assist a human driver in driving the vehicle”).
Sajjan fails to explicitly disclose a plurality of outputs, wherein each output of the plurality of outputs is generated by a particular neural network model of the plurality of neural network models.
However, Tremblay teaches a plurality of outputs, wherein each output of the plurality of outputs is generated by a particular neural network model of the plurality of neural network models (See at least fig 1-11, ¶ 17, 24, 26, 30, 71, 82, 29, “a perception network 302 can be a deep neural network that accepts the demonstration data captured of the performance, such as may include image, distance, and other data as discussed herein…a plan generation network 304, which can be another deep neural network that can process the percepts to generate, or infer, a human-readable plan corresponding to the task…an execution network 306, which can be a deep neural network capable of processing data for the plan and generating, or inferring, one or more robot-readable instructions”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Sajjan and include controlling, by the high-level controller, all load outputs as taught by Tremblay because it would a system to have a set of neural networks to perform the tasks associated with perception, program generation, and program execution (Tremblay ¶ 17).
Regarding claim 20, Sajjan discloses the non-transitory machine-readable medium of claim 19, wherein the output comprises steering data and throttle data, and wherein the operations further comprise: converting the output into control signals; and controlling the robotic device based on the control signals (See at least fig 1-11, ¶ 99, 100, 113, 153, 184, 96 “The propulsion system 950 may be controlled in response to receiving signals from the throttle/accelerator 952”).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LUIS A MARTINEZ BORRERO whose email is luis.martinezborrero@uspto.gov and telephone number is (571)272-4577. The examiner can normally be reached on M-F 8:00-5:00. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, HUNTER LONSBERRY can be reached on (571)272-7298. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LUIS A MARTINEZ BORRERO/Primary Examiner, Art Unit 3665