DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
The amendment filed July 6, 2026 has been entered. Claims 1-3, 5-8, 10, 14, 16, 17, and 20 have been amended. The remaining claims are in original or previously presented form. Therefore, claims 1-20 are pending in the application. Claims 1, 10, and 17 are the independent claims.
The Remarks filed July 6, 2026 have been fully considered. The applicant argues under the heading “35 USC 112 Rejection” that the amendments to claims 8, 16, and 20 removing the word “subset” should result in the withdrawal of the 35 USC 112 rejection given in the last detailed action, which was the Non-Final Rejection dated January 7, 2026. The examiner agrees and withdraws the rejection.
The applicant argues under the heading “35 USC 102 Rejection” that claim 17 has been amended with support from the present disclosure’s paragraph 0358 and 00392. The applicant states that claims 1 and 9 have been amended in the same way.
The applicant argues that the prior art of Arditi et al. (US2019/0197430) does not teach claim 17 as amended. In particular the applicant argues that Arditi does not teach, as present claim 17 does:
the computing system is to perform one or more operations associated with control of the autonomous or semi-autonomous machine based at least on processing, with the one or more graphics processing units (GPUs) capable of massively parallel processing and using one or more neural networks, a fused representation of sensor data obtained usingthe plurality of external sensors; and
the one or more neural networks were trained, at least in part, using one or more virtual simulations or training data generated using one or more virtual simulations.
Does Arditi teach this? Arditi teaches in paragraph 0104 that the host vehicle 1340 can have radar and lidar. These are to help the vehicle “understand the external world around it.” Paragraph 0105 teaches that the host vehicle can process the sensor data with GPUs. Therefore, Arditi teaches, as present claim 17 does “processing…sensor data” from a plurality of external sensors.
The examiner maintains that a GPU performs massively parallel processing by definition. This definition comes from Nvidia itself. As stated in the last detailed action, under “Additional Art” the examiner cited: Tarjan et al. (US2011/0161616 A1), a Nvidia application, teaches in paragraph 002 that “modern GPUs are massively parallel processors”. This defines GPUs.
Arditi also teaches that the system can use GPUs using one or more neural networks to process sensor data in order to perform vehicle operations. Paragraph 0096 teaches using “neural networks” that are “trained” to perform pick up and drop off.
Arditi focuses on training using personal preferences or on training using emergency scenarios regarding what is going on internal to the cabin. Arditi is not as focused on how the vehicle itself navigates.
However, other prior art teaches this. Due to the applicant’s amendments the grounds for rejection have changed. Please see the rejections below.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claims contain subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for pre-AIA the inventor(s), at the time the application was filed, had possession of the claimed invention.
Present claim 17 recites:
the computing system is to perform one or more operations associated with control of the autonomous or semi-autonomous machine based at least on processing, with the one or more graphics processing units (GPUs) capable of massively parallel processing and using one or more neural networks, a fused representation of sensor data obtained usingthe plurality of external sensors; and
the one or more neural networks were trained, at least in part, using one or more virtual simulations or training data generated using one or more virtual simulations.
Is there specification support for teaching that a GPU that uses a neural network processing a fused representation of sensor data obtained using a plurality of external sensors? Paragraph 0378 teaches that the Advanced SoC, which is later described as a GPU, is capable of make decisions “based on a fusion of both RADAR and LIDAR”. This is written description, in part. Paragraph 0384 states that the GPU and neural network can “perform the necessary functions for autonomous driving, including object detection, free space detection” etc. Paragraph 0345 teaches “sensor fusion algorithms” but in the context of the tire-pressure monitoring system. Paragraph 0084 teaches a “combined perception model”.
But the specification does not use the term “a fused representation of sensor data”. Nor does it teach that the system processes “a fused representation of sensor data”. It really only says that the GPU can use “a fusion of both radar and lidar”. It is not clear in the specification that these are merged together into some kind of “fused representation” before being processed, as taught in the claim.
The word “fusion” can reasonably mean that both sensor data sent to the neural network as inputs, which uses these inputs to create an output. The using of these inputs to create an output is the fusion. The fusion does not happen ahead of time and then that fusion is processed.
As stated in the previous detailed action, the examiner is weary of spec. creep. The claims cannot be amended to suggest anything more than what is in the specification, even slightly. For examination purposes, the phrase will be interpreted to only mean what the specification supports, and the phrase will need to be amended to more closely hue to the specification and its meaning.
The specification does not use the term “virtual simulations”. It only uses the term “simulations”. .While the term simulation” is frequently used, virtual simulation is not used. Paragraph 0141 teaches that a vehicle can be “trained” using a busy parking lot scene as a “scenario”. See also the section entitled “System Simulation” from paragraphs 0205-0214. In particular, see paragraph 0212.
Paragraph 00394, cited by the applicant, teaches that the shuttles are trained through the creation of a “virtual world.” The shuttle drives through this virtual world which is “a simulation.”
For examination purposes, the term virtual simulation will be interpreted as a simulation. Some amendment will likely need to be made.
A note about the last bullet: In the last bullet, the term “generated using one or more virtual simulations” is only intended to modify “training data”. That is clear, because if it were intended to also modify “virtual simulations” then the clause could appear to teach the absurd result that the neural network is trained using a virtual simulation generated using a virtual simulation. That is an unreasonable interpretation. This is not a rejection issue, but a note for clarification.
Independent claims 1 and 10 are substantially similar in their reciting of “a fused representation of sensor data” and “virtual simulations” are rejected for the same reasons, and will be interpreted for examination purposes in the same way.
Claim 14 is rejected for lacking written description. The claim recites:
The autonomous or semi-autonomous machine of claim 10, wherein
the computing system is capable of massively parallel processing at least due to the use of the one or more GPUs and the one or more hardware accelerators in combination.
Does the present specification teach that the system is capable of massively parallel processing “due to” the use of the one or more GPUs and the one or more hardware accelerators in combination? The specification teaches in paragraph 0353 that “Each GPU can be used for any advanced processing task, especially complex tasks that benefit from massively parallel processing.” Paragraph 0355 teaches that an “Acceleration Cluster (400)” can including deep learning accelerators, DLAs, that run a specific neural network with better performance than if executed on a general-purpose GPU alone. The DLA might detect features in the perception data (such as data from cameras, Radar, and lidar). Paragraph 0048 teaches that the “GPU may provide massively parallel processing”. Paragraph 0326 mentions that the GPU can perform massively parallel processing. In the examiner’s view, a GPU is capable of performing massively parallel processing by definition.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-4 and 9 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Lakshamanan et al. (US2018/0300964).
Regarding claim 1, Lakshamanan discloses:
An autonomous or semi-autonomous machine comprising (see Fig. 22):
a propulsion system (see paragraph 0236 for drive motors);
a passenger space (see Fig. 22);
a plurality of external external to see Figs. 22 and 24.),
the plurality of external sensors including at least a first sensor of a first sensor modality and a second sensor of a second sensor modality (see Fig. 24 for sensors 2420. See also paragraph 0216 for “cameras, radar, lidar, etc.”);
one or more internal see Fig. 22, item 2206 for “internal sensors”. See paragraph. See paragraph 0232 for “internal cameras” to assess passengers);
a computing system capable of performing massively parallel processing, the computing system including one or more systems-on-a-chip (SoCs) (see paragraph 0236. See paragraph 0322 for the beginning of a discussion of “System on a Chip”. See Figs. 1, 2A-2D, and 3A-3B and paragraph 0086 for a CPU and paragraph 0087 for a GPU can is coupled to the CPU. A GPU performs massively parallel processing by definition.)),
the one or more SoCs including one or more central processing units (CPU), one or more graphics processing units (GPUs), and one or more hardware accelerators (see Figs. 1, 2A-2D, and 3A-3B and paragraph 0086 for a CPU and paragraph 0087 for a GPU can is coupled to the CPU. See paragraphs 0095-0096 for an accelerator integration circuit. See paragraph 0100 for a “graphics accelerator modules 446 and/or other accelerator devices.” A GPU performs massively parallel processing by definition.),
wherein the computing system is to perform one or more planning, navigation, or control operations associated with control of the autonomous or semi-autonomous machine based at least on processing one or more fused representations of sensor data obtained using the plurality of external sensors (see paragraph 0216 for the vehicle’s processors analyzing camera, radar, and lidar sensor data. See paragraph 0184 for the system being applied to solve the technical problem of “autonomous driving and navigation”. See also paragraph 0185, especially the first sentence. See Fig. 24 and paragraph 0233 for sensor data being transmitted into the machine learning system of the vehicle. See paragraph 0146 for training the neural network for “image recognition, mapping and localization, autonomous navigation,” etc. In the context of paragraph 0144 and the entire disclosure, this reasonably means that the “problem” the neural network is being trained to solve is autonomous driving.).
Regarding claim 2, Lakshamanan discloses the autonomous or semi-autonomous machine of claim 1.
Lakshamanan further discloses:
The autonomous or semi-autonomous machine of claim 1, wherein
the massively parallel processing is achievable using, at least, the one or more GPUs and the one or more hardware accelerators in combination (see paragraphs 0095-0096 for an accelerator integration circuit. See paragraph 0100 for a “graphics accelerator modules 446 and/or other accelerator devices.”).
Regarding claim 3, Lakshamanan discloses the autonomous or semi-autonomous machine of claim 1.
Lakshamanan further discloses:
The autonomous or semi-autonomous machine of claim 1, wherein
the autonomous or semi-autonomous machine is capable of level 3 or greater autonomous vehicle functionalitysee Fig. 1, for “autonomous or partially-autonomous vehicle 110”. Those terms cover level 3 autonomy and greater.).
Regarding claim 4, Lakshamanan discloses the autonomous or semi-autonomous machine of claim 1.
Lakshamanan further discloses:
The autonomous or semi-autonomous machine of claim 1, further comprising
one or more modems for wireless communication over one or more cellular networks, wherein data is received from one or more remote computing devices, via the one or more modems, to update one or more neural networks, one or more algorithms, or one or more maps stored on the autonomous or semi-autonomous machine (see Fig. 14 for the lightening-bolt-looking transmissions. This indicates that the ego vehicle has a modem. See also Fig. 16 for the ego vehicle 1602 communicating wireless with nearby Avs and a server. See Fig. 20 for a vehicle identifying an obstacle or event using that vehicle’s sensors and transmitting the analysis to a trailing vehicle. See paragraph 0223 for this being done using Wifi or DSRC connections. See paragraph 0230 for periodic updating of occupied locations. In the context of the disclosure, this includes the reception of the location of the obstacle via another vehicle.).
Regarding claim 9, Lakshamanan discloses the autonomous or semi-autonomous machine of claim 1.
Lakshamanan further discloses:
The autonomous or semi-autonomous machine of claim 1, wherein
the one or more hardware accelerators include at least one of a deep learning accelerator (DLA) or a vision accelerator (see paragraphs 0184-0185. See also paragraph 0190 for a vision processor that can work with a media processor 1302 to accelerate computer vision operations. The media processor 1302 can accelerate neural network operations. See also Fig. 13.).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 5, 6, and 10-14 are rejected under 35 U.S.C. 103 as being unpatentable over Lakshamanan et al. (US2018/0300964) in view of Frtunikj (DE102017214531).
Regarding claim 5, Lakshamanan discloses the autonomous or semi-autonomous machine of claim 1.
Yet Lakshamanan does not further teach:
The autonomous or semi-autonomous machine of claim 1, wherein
one or more planning, navigation, or control operations of the autonomous or semi- autonomous machine are capable of achieving ISO 26262 automotive safety integrity level (ASIL) B or greater.
However, Frtunikj teaches:
one or more planning, navigation, or control operations of the autonomous or semi- autonomous machine are capable of achieving ISO 26262 automotive safety integrity level (ASIL) B or greater (see page 3 of the attached English translation for an autonomous vehicle that can operate at “SAE stage 3-5” and does so with, “for example, the ASIL-D standard or…ASIL-C standard”. The system meets “ASIL…A-D.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Lakshamanan, to add the additional features of: one or more operations of the autonomous or semi-autonomous machine are capable of achieving ISO 26262 automotive safety integrity level (ASIL) B or greater, as taught by Frtunikj. The motivation for doing so would be to have an autonomous vehicle that is “classified according to a given safety condition,” and recognized standard, as recognized by Frtunikj (see page 2)
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
Regarding claim 6, Lakshamanan and Frtunikj teach the autonomous or semi-autonomous machine of claim 1.
Yet Lakshamanan does not further teach:
The autonomous or semi-autonomous machine of claim 5, wherein
one or more planning, navigation, or control operations of the autonomous or semi- autonomous machine are capable of achieving ISO 26262 automotive safety integrity level (ASIL) D.
However, Frtunikj teaches:
one or more planning, navigation, or control operations of the autonomous or semi- autonomous machine are capable of achieving ISO 26262 automotive safety integrity level (ASIL) D (see page 3 of the attached English translation for an autonomous vehicle that can operate at “SAE stage 3-5” and does so with, “for example, the ASIL-D standard or…ASIL-C standard”. The system meets “ASIL…A-D.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Lakshamanan and Frtunikj, to add the additional features as indicated as being taught by Frtunikj. The motivation for doing so would be to have an autonomous vehicle that is “classified according to a given safety condition,” and recognized standard, as recognized by Frtunikj (see page 2)
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
Regarding claim 10, Lakshamanan teaches:
An autonomous or semi-autonomous machine comprising (see Fig. 22):
a propulsion system (see paragraph 0236 for drive motors);
a passenger space (see Fig. 22);
a plurality of external sensors, of two or more different sensor modalities, having fields of view or sensory fields outside of the autonomous or semi- autonomous machine (see Fig. 24 for sensors 2420. See also paragraph 0216 for “cameras, radar, lidar, etc.” See also Fig. 22.);
one or more internal see Fig. 22, item 2206 for “internal sensors”. See paragraph. See paragraph 0232 for “internal cameras” to assess passengers);
a computing system, the computing system including one or more systems-on-a-chip (SoCs) (see paragraph 0236. See paragraph 0322 for the beginning of a discussion of “System on a Chip”. See Figs. 1, 2A-2D, and 3A-3B and paragraph 0086 for a CPU and paragraph 0087 for a GPU can is coupled to the CPU. A GPU performs massively parallel processing by definition.),
the one or more SoCs including one or more central processing units (CPU), one or more graphics processing units (GPUs), and one or more hardware accelerators, wherein (see Figs. 1, 2A-2D, and 3A-3B and paragraph 0086 for a CPU and paragraph 0087 for a GPU can is coupled to the CPU. See paragraphs 0095-0096 for an accelerator integration circuit. See paragraph 0100 for a “graphics accelerator modules 446 and/or other accelerator devices.” A GPU performs massively parallel processing by definition.):
the computing system is to perform one or more operations associated with control of the autonomous or semi-autonomous machine based at least on processing, using one or more neural networks, a fused representation of sensor data obtained using the plurality of external sensors (see paragraph 0216 for the vehicle’s processors analyzing camera, radar, and lidar sensor data. See paragraph 0184 for the system being applied to solve the technical problem of “autonomous driving and navigation”. See also paragraph 0185, especially the first sentence. See Fig. 24 and paragraph 0233 for sensor data being transmitted into the machine learning system of the vehicle. See paragraph 0146 for training the neural network for “image recognition, mapping and localization, autonomous navigation,” etc. In the context of paragraph 0144 and the entire disclosure, this reasonably means that the “problem” the neural network is being trained to solve is autonomous driving.);
the autonomous or semi-autonomous machine is capable of achieving level 3 autonomous vehicle functionality or greater (see Fig. 1, for “autonomous or partially-autonomous vehicle 110”. Those terms cover level 3 autonomy and greater.).
Yet Lakshamanan does not further teach:
at least one operation of the one or more operations is capable of satisfying ISO 26262 automotive safety integrity level (ASIL) D.
However, Frtunikj teaches:
one or more operations of the autonomous or semi-autonomous machine are capable of achieving ISO 26262 automotive safety integrity level (ASIL) B or greater (see page 3 of the attached English translation for an autonomous vehicle that can operate at “SAE stage 3-5” and does so with, “for example, the ASIL-D standard or…ASIL-C standard”.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Lakshamanan, to add the additional features of: one or more operations of the autonomous or semi-autonomous machine are capable of achieving ISO 26262 automotive safety integrity level (ASIL) B or greater, as taught by Frtunikj. The motivation for doing so would be to have an autonomous vehicle that is “classified according to a given safety condition,” and recognized standard, as recognized by Frtunikj (see page 2)
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
Regarding claim 11, the claim is substantially similar to claim 9. Please see the rejection for that claim.
Regarding claim 12, the claim is substantially similar to claim 3. Please see the rejection for that claim.
Regarding claim 13, the claim is substantially similar to claim 4. Please see the rejection for that claim.
Regarding claim 14, the claim is substantially similar to claim 2. Please see the rejection for that claim.
Claims 7 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Lakshamanan et al. (US2018/0300964) in view of Myers (US2018/0074495).
Regarding claim 7, Lakshamanan teaches the autonomous or semi-autonomous machine of claim 1.
Yet Lakshamanan does not explicitly further teach:
The autonomous or semi-autonomous machine of claim 1, wherein
one or more planning, navigation, or control operations associated with control of the autonomous or semi-autonomous machine include autonomously picking up and dropping off one or more passengers.
However, Myers teaches:
one or more planning, navigation, or control operations associated with control of the autonomous or semi-autonomous machine include autonomously picking up and dropping off one or more passengers (see Fig. 4A, items 402-404 and paragraph 0002).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Lakshamanan to add the additional features as indicated as being taught by Myers. The motivation for doing so would be to identify the correct person as the passenger and monitor the passenger activity and customize the vehicle operation accordingly, including driving manner and destination reminders, as recognized by Myers (see paragraphs 0002, 0028, and 0041).
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
Regarding claim 8, Lakshamanan and Myers teach the autonomous or semi-autonomous machine of claim 7.
Yet Lakshamanan does not explicitly further teach:
The autonomous or semi-autonomous machine of claim 7, wherein
at least one passenger of the one or more passengers is picked up based at least on identifying the at least one passenger using first sensor data obtained using theplurality of external sensors, and
the at least one passenger is monitored between pick up and drop off using a-second internal sensors.
However, Myers teaches:
at least one passenger of the one or more passengers is picked up based at least on identifying the at least one passenger using first sensor data obtained using theplurality of external sensors (see Fig. 1 and paragraph 0020 for monitoring module 104 configured to “identify passengers, authenticate passengers, monitor passenger activity, and monitor passengers entering and exiting the vehicle”. See paragraph 0026 for module 104 using “a facial recognition algorithm that identifies a face of a person approaching the vehicle”. Other biometric information can also be used.), and
the at least one passenger is monitored between pick up and drop off using a-second internal sensors (see paragraph 0028 for a passenger analysis module 214 that monitors passengers inside the vehicle. See paragraph 0030 and Fig. 3 for the vehicle 300 have interior cameras that monitor passengers. See paragraph 0034 for doing this using a deep neural networks.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Lakshamanan and Myers to add the additional features as indicated as being taught by Myers. The motivation for doing so would be to identify the correct person as the passenger and monitor the passenger activity and customize the vehicle operation accordingly, including driving manner and destination reminders, as recognized by Myers (see paragraphs 0002, 0028, and 0041).
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
Since Lakshamanan teaches monitoring passengers using internal sensors, Lakshamanan could be thought of as teaching the second bullet. But Myers is even more explicitly so Myers has been used.
Claims 15 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Lakshamanan in view of Frtunikj in further view of Myers et al. (US2018/0074495).
Regarding claim 15, the claim is substantially similar to claim 7. Please see the rejection for that claim.
Regarding claim 16, the claim is substantially similar to claim 8. Please see the rejection for that claim.
Claims 17 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Lakshamanan et al. (US2018/0300964) in view of Redding et al. (US2018/0089563).
Regarding claim 17, Lakshamanan teaches:
An autonomous or semi-autonomous machine comprising (see Fig. 22);
a passenger space (see Fig. 22);
a plurality of external sensors, including two or more sensors of different modalities, having fields of view or sensory fields outside of the autonomous or semi- autonomous machine (see Fig. 22. See Fig. 24 for sensors 2420. See also paragraph 0216 for “cameras, radar, lidar, etc.”);
one or more internal see Fig. 22, item 2206 for “internal sensors”. See paragraph. See paragraph 0232 for “internal cameras” to assess passengers);
a computing system, the computing system including one or more central processing units (CPU) and one or more graphics processing units (GPUs) capable of massively parallel processing, wherein (see Figs. 1, 2A-2D, and 3A-3B and paragraph 0086 for a CPU and paragraph 0087 for a GPU can is coupled to the CPU. A GPU performs massively parallel processing by definition.):
the computing system is to perform one or more operations associated with control of the autonomous or semi-autonomous machine based at least on processing, with the one or more graphics processing units (GPUs) capable of massively parallel processing and using one or more neural networks, a fused representation of sensor data obtained usingthe plurality of external sensors (see paragraph 0141 for the start of a section on “Machine learning overview”. See paragraph 0143 for the machine learning system including a “neural network.” See paragraph 0158 for CNNs and paragraph 0159 for RNNs. GPUs are capable of massively parallel processing by definition. See paragraph 0216 for the vehicle’s processors analyzing camera, radar, and lidar sensor data. See Fig. 24 and paragraph 0233 for sensor data being transmitted into the machine learning system of the vehicle.).
However, Lakshamanan does not explicitly further teach:
the one or more neural networks were trained, at least in part, using one or more virtual simulations or training data generated using one or more virtual simulations.
Yet Redding et al. (US2018/0089563) teaches:
the one or more neural networks were trained, at least in part, using one or more virtual simulations or training data generated using one or more virtual simulations (see paragraph 0006. See Fig. 11, step 1107 for a “behavior planner” being part of the “subsystem of the vehicle”. This is also seen in Fig. 1 in which the behavior planner, item 117 is part of the autonomous vehicle 110. See paragraph 0065 for step 1107 involving “reinforcement learning (e.g., based on simulations of driving behavior)”. See also claim 6.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Lakshamanan, to add the additional features as indicated as being taught by Redding. The motivation for doing so would be to perform computations efficiently, as recognized by Redding (see paragraph 0006).
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
In summary, Lakshamanan teaches at least almost all the limitations of the claim. Lakshamanan teaches that the ego vehicle can also use distributed computing, such as computing for or by another nearby vehicle or using a server, but that is in additional to the ego vehicle performing computing. Lakshamanan can therefore be read as teaching claim 1 and additionally teaching optional distributed computing. Redding also teaches such optional distributed computing.
This combination is especially obvious because Lakshamanan at least strongly teaches toward it. See Fig. 11. See Fig. 36 for a “software simulation” and paragraph 0319 for the entire “SOC”—which includes the CPU and GPU—can use a simulation to “design, test, and verify the behavior of the IP core”. The training can include “functional, behavioral, and/or timing simulations. See paragraph 0144 for “training a neural network” using “a set of training data representing a problem being modeled”. See paragraph 0146 for training the neural network for “image recognition, mapping and localization, autonomous navigation,” etc. In the context of paragraph 0144 and the entire disclosure, this reasonably means that the “problem” the neural network is being trained to solve is autonomous driving. According to paragraph 0146, there is “specialized software that can be used to train a neural network before deployment”. See paragraph 0173 for the neural network being “trained using a training dataset 1102”. This dataset, according to paragraph 0175, includes an input paired with a desired output.
Regarding claim 19, the claim is substantially similar to claim 3. Please see the rejection for that claim.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Lakshamanan et al. (US2018/0300964) in view of Redding et al. (US2018/0089563) in further view of Myers et al. (US2018/0074495).
Regarding claim 20, the claim is substantially similar to claim 8. Please see the rejection for that claim.
Additional Art
The prior art made of record here, though not relied upon, is considered pertinent to the present disclosure.
Chen et al. (US2019/0049980), a TuSimple disclosure, teaches training the neural network or an autonomous vehicle using a “vehicle simulation environment that can be changed and adapted to a new simulation environment” using “a new training dataset”. See also paragraph 0018, which teaches that “a neural network or other machine learning system is used to predict accurate vehicle accelerations based on recorded or otherwise captured historical vehicle driving data. In an example embodiment, vehicle driving data corresponding to real world vehicle operations or simulated vehicle movements is captured over time for a large number of vehicles in a large number of operating environments…this historical vehicle driving data for a plurality of particular vehicle simulation environments can be represented as various sets of data in training datasets 135….This enables the autonomous vehicle simulation system 140 to adapt to a particular and desired autonomous vehicle simulation environment.
Nagel et al. (US2013/0335318). See paragraph 0029 for using a GPU and a hardware accelerator. The application is for “gesture recognition,” which is also a focus of the present application.
Sun et al. (US 20190129436 A1), a TuSimple disclosure. Teaches collecting real world driving data for using in training a machine learning system.
Heesche et al. (US2021/0278810) teaches training neural networks for vehicle control.
Kentley (U.S. 9,606,539) a Zoox disclosure. Appears not to teach a neural network, but does teach a Nvidia GPU.
Kislovskiy (US2018/0339712), an Uber disclosure. Does not teach a GPU but teaches a “neural net” at least in Fig. 18.
Ghafarianzadeh et al. (U.S. 10,955,851), a Zoox disclosure. See col. 8, lines 49-65 for: “At operation 212, the example process 200 may include determining, using the BV ML [machine learning] model 214” in which “the feature values 210 may be input into the BV MOL model 214” which uses “deep learning…such as an artificial neural network” to determine if the host vehicle is blocked by another vehicle. See Fig. 1B and col. 5, lines 50 – col. 6, line 7 for sensor data including lidar data and camera data. See col. 7, lines 46-48 for features values 210 being based on sensor data. See col. 14, lines 22-36 for the processor 404 including a GPU. See Fig. 4 for the host vehicle having processors 404.
Levinson et al. (US2017/0132334). Levinson teaches a lot about simulation, but not so much about training.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL M. ROBERT whose telephone number is (571)270-5841. The examiner can normally be reached M-F 7:30-4:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hunter Lonsberry can be reached at 571-272-7298. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL M. ROBERT/Primary Examiner, Art Unit 3665
Prosecution Insights