DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant’s claim for domestic benefit under 35 U.S.C. 119(e) is acknowledged.
Information Disclosure Statement
The information disclosure statement submitted has been considered by the Examiner and made of record in the application file.
Claim Interpretation
Applicant is cautioned that a recitation “capable of” or any variations thereof as claimed is not functional and is not a positive recitation. Therefore, under BRI would only require a “capability” or “ability” and not necessarily perform the function or action based on the applied prior art.
Claim Objections
Claims 8, 13, 14, and 18 are objected to because of the following informalities:
Claim 8 recites "whether a current a safety goal corresponds to the first set of safety goals or the second set of safety goals." The second occurrence of "a" is a typographical error. The examiner suggests "whether a current safety goal corresponds to …".
Claims 13 and 14 each recite "a respective sensor of the first processing pipeline or the second processing pipeline," whereas claim 9 recites "a first sensor processing pipeline" and "a second sensor processing pipeline." Although the intended reference is reasonably clear, the terminology should be made consistent: "a respective sensor of the first sensor processing pipeline or the second sensor processing pipeline." The claims have been examined with that reading.
Claim 18, last clause, recites "determine, using an arbiter, a final output based at least the first fused output and the second fused output." The word "on" appears to have been omitted. The examiner suggests "based at least on the first fused output and the second fused output" (cf. claim 1).
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 4 and 11 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 4 recites "the first safety integrity level" and "the second safety integrity level." There is insufficient antecedent basis for these limitations in the claim. Claim 4 depends from claim 3, which recites only "a higher safety integrity level"; neither claim 3 nor claim 1 introduces "a first safety integrity level" or "a second safety integrity level." It is therefore unclear which of the several levels implicated by claim 3 (the level of the sensor data, of the intermediate outputs, of the first fused output, or the "higher" level of the second fused output and final output) is the "first" and which is the "second." For purposes of examination, "the second safety integrity level" is interpreted as the "higher safety integrity level" of claim 3, and "the first safety integrity level" is interpreted as the lower level associated with the first fused output.
Claim 11 recites "the first processing component" and "the second processing component." There is insufficient antecedent basis for these limitations in the claim. Claim 11 depends from claim 9, which does not recite any processing component; the first and second processing components are introduced in claim 10. For purposes of examination, claim 11 is treated as depending from claim 10.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 2, 5, 7, 18, 19, and 20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Sakamoto et al., US 2022/0089179 A1 (published March 24, 2022; issued as US 12,162,511 B2), hereinafter "Sakamoto."
Consider claim 1, Sakamoto discloses a method ([0025]–[0039], [0066]–[0070], describing the operation of the vehicle control device 11) comprising:
generating, using a first sensor processing pipeline that includes a first processing component to process first sensor data obtained using a first sensor, a first intermediate output — Fig. 1; [0026], [0028]: "a camera 1 is a sensor module incorporating a sensor 101 and the information processing unit 102 … The information processing unit 102 generates the object data based on the raw data acquired from the sensor 101." The camera 1 is a first sensor processing pipeline; the information processing unit 102 (a microcomputer or FPGA, [0030]) is a first processing component that processes the first sensor data (raw data 104 of sensor 101); and the object data 103 is a first intermediate output.
generating, using a second sensor processing pipeline that includes a second processing component to process second sensor data obtained using a second sensor, a second intermediate output — Fig. 1; [0029]: "A radar 2 is a sensor module incorporating a sensor 201 and the information processing unit 202 … The information processing unit 202 generates the object data 203 based on the raw data acquired from the sensor 201."
processing first data representative of the first intermediate output and the second intermediate output to generate a first fused output — [0027], [0034], [0044]: "an arithmetic block 212 of the microcomputer 112 generates an object data fusion result 702 by performing data fusion processing on the pieces of object data 103, 203, and 303 obtained from the sensors." The object data 103, 203 is first data representative of the first and second intermediate outputs, and the object data fusion result 702 is a first fused output.
processing at least one of the first data or second data from the first sensor processing pipeline and the second sensor processing pipeline to generate a second fused output — [0025], [0028]–[0029], [0031]: "an arithmetic block 211 of the microcomputer 111 generates a raw data fusion result 701 by performing data fusion processing on pieces of raw data 104, 204, and 304 obtained from the sensors"; the raw data acquired from each sensor "is transmitted to the information processing unit 102 and the arithmetic block 211 …." The raw data 104, 204 output by the sensors of the two pipelines is second data from the first and second sensor processing pipelines, and the raw data fusion result 701 is a second fused output.
determining, using an arbiter, a final output based at least on processing the first fused output and the second fused output — Figs. 1 and 3; [0037]: "The raw data fusion result 701 obtained by the sensor fusion in the arithmetic block 211 and the object data fusion result 702 obtained by the sensor fusion in the arithmetic block 212 are diagnosed by a comparison diagnosis function of the arithmetic block 213. Trajectory tracking control is performed based on the diagnosis result of the arithmetic block 213"; [0052]: "when the object fusion result 702 output by the arithmetic block 212 is included in the raw data fusion result 701 output by the arithmetic block 211, the determination of diagnosis result: normal is performed" (see also [0056] and Sakamoto's claims 2–3); [0038]: "When the arithmetic block 211 is normal as the result of the comparison and diagnosis, the trajectory tracking control unit 214 generates and transmits a control command to actuator control ECUs 13, 14, and 15 such that the host vehicle is tracked based on the trajectory data 711 generated by the microcomputer 111"; [0039]: "When the arithmetic block 211 is diagnosed as abnormal, the control tracking control unit 214 generates and transmits a control command … such that the host vehicle tracks the trajectory data 712 generated by the microcomputer 112"; [0055]: in that case "it is determined that the object fusion result 702 output from the arithmetic block 212 with high reliability is correct"; [0058]: the output of the raw data fusion result 701 is prohibited; [0070]: "it is possible to switch between the trajectories." The arithmetic block 213, together with the trajectory tracking control unit 214 that acts on its diagnosis result, is an arbiter, and the fusion result and trajectory adopted for control — the raw data fusion result 701/trajectory 711 or the object data fusion result 702/trajectory 712 — is a final output determined by processing the two fused outputs.
Consider claim 2, Sakamoto discloses that the arbiter uses a rule-based processing component to determine the final output - the arithmetic block 213 applies a fixed, predetermined rule — the diagnosis is "normal" when the object group of the object data fusion result 702 is included in the object group of the raw data fusion result 701, and "abnormal" otherwise ([0052], [0056]; Sakamoto's claims 2-3) — and the trajectory tracking control unit 214 selects the trajectory according to that diagnosis ([0038]–[0039]). Sakamoto expressly distinguishes this diagnosis path, which relies on "the system independent of the fusion processing by the machine learning," from the machine-learning fusion of arithmetic block 211 ([0061]).
Consider claim 5, Sakamoto discloses that the second data from the first sensor processing pipeline and the second sensor processing pipeline includes at least the first sensor data and the second sensor data ([0025], [0028]–[0029], [0031]: the raw data 104 of the camera's sensor 101 and the raw data 204 of the radar's sensor 201 are transmitted to the arithmetic block 211, which fuses them to generate the raw data fusion result 701).
Consider claim 7, Sakamoto discloses that the first fused output and the second fused output correspond to a same safety goal: both the object data fusion result 702 and the raw data fusion result 701 recognize the object group — vehicles and pedestrians — around the host vehicle so that a trajectory avoiding contact with surrounding objects can be generated ([0002], [0041]–[0042], [0047]–[0048]), and the diagnosis specifically tests whether both results recognize the vehicles traveling near the host vehicle ([0051], [0054]–[0055]).
Consider claim 18, Sakamoto discloses one or more processors — the information processing units 102, 202 of the sensor modules (each a microcomputer or FPGA, [0030]) together with the microcomputers 111, 112, 113 of the vehicle control device 11 ([0036]) — comprising processing circuitry (the information processing units; the arithmetic blocks 211, 212, 213; the lockstep microcomputer 112 "includes a CPU and a memory," [0032]) to:
generate, using a first sensor processing pipeline that includes a first processing component to process first sensor data obtained using a first sensor, a first intermediate output (camera 1: sensor 101, information processing unit 102, object data 103; [0026], [0028]);
generate, using a second sensor processing pipeline that includes a second processing component to process second sensor data obtained using a second sensor, a second intermediate output (radar 2: sensor 201, information processing unit 202, object data 203; [0029]);
process first data from the first sensor processing pipeline and the second sensor processing pipeline to generate a first fused output (arithmetic block 211 fuses the raw data 104, 204 from the sensors of the two pipelines to generate the raw data fusion result 701; [0025], [0031]);
process at least one of the first data or second data representative of the first intermediate output and the second intermediate output to generate a second fused output (arithmetic block 212 fuses the object data 103, 203 to generate the object data fusion result 702; [0027], [0034], [0044]); and
determine, using an arbiter, a final output based at least [on] the first fused output and the second fused output (arithmetic block 213 compares the fusion results 701 and 702 and, based on the comparison, one of the fusion results and its trajectory is adopted for trajectory tracking control; [0037]–[0039], [0052], [0055], [0070], as applied to claim 1). The examiner notes that claim 18 recites the two fused outputs in the opposite order from claim 1 — its first fused output is generated from "first data from the … pipeline[s]" and its second fused output from data "representative of the first intermediate output and the second intermediate output" — so that Sakamoto's raw data fusion result 701 is the first fused output and its object data fusion result 702 is the second fused output for purposes of claims 18–20.
Consider claim 19, Sakamoto discloses that the first data includes at least one of the first sensor data or the second sensor data ([0025], [0028]–[0029], [0031]: the arithmetic block 211 fuses the raw data 104, 204 output by the sensors 101, 201).
Consider claim 20, Sakamoto discloses that the one or more processors are comprised in a control system and a perception system for an autonomous or semi-autonomous machine ([0002], [0017]–[0018]: a self-driving electronic control unit that recognizes the surrounding environment and controls the vehicle's brake, engine, and power steering actuators).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 9, 10, 11, 12, 13, 15, 16, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Sakamoto in view of Pfeifle et al., US 2019/0279000 A1, hereinafter "Pfeifle."
Consider claim 9, Sakamoto discloses a system (vehicle control device 11) comprising one or more processors (microcomputers 111, 112, 113; [0036]) to:
determine, using learned sensor fusion and based at least on first data generated using a first sensor processing pipeline and a second sensor processing pipeline, a first output capable of satisfying a — [0025], [0028]–[0029], [0031]: the arithmetic block 211 of microcomputer 111 generates the raw data fusion result 701 (a first output) by inputting the raw data 104, 204 generated by the camera 1 and radar 2 pipelines "to a neural network (NN) or a deep neural network (DNN)," i.e., using learned sensor fusion; the machine-learning result is the one Sakamoto treats as the less reliable of the two and subjects to diagnosis ([0060]–[0062]), and it is computed on "an electronic control processor different from" the lockstep processor of the rule-based path ( [0064]).
determine, using rule-based sensor fusion and based at least on at least one of the first data or second data generated using the first sensor processing pipeline and the second sensor processing pipeline, a second output capable of satisfying — [0027], [0034], [0044]: the arithmetic block 212 generates the object data fusion result 702 (a second output) from the object data 103, 203 generated by the information processing units of the two pipelines (second data); [0061]: "the fusion by the arithmetic block 212 is desirably generated by a rule-based algorithm"; [0032], [0045], [0062]–[0064]: the arithmetic block 212 runs on the lockstep microcomputer 112, whose duplex, clock-synchronous CPU subsystems give it "high reliability," and adopts the sensors' object data by majority decision so that "a highly reliable output can still be obtained even though one sensor fails" ([0058]); [0055], [0060]: the object data fusion result 702 is treated as "correct" and is the reference against which the machine-learning result is diagnosed.
determine, based at least on an arbitration between the first output and the second output, a final output capable of satisfying the — [0037]–[0039], [0052], [0055]–[0056], [0070]: the arithmetic block 213 compares the two fusion results and, depending on the outcome, either the raw data fusion result 701/trajectory 711 (validated by its agreement with the high-reliability result 702 as to the nearby vehicles) or the object data fusion result 702/trajectory 712 is adopted for trajectory tracking control.
perform one or more operations based at least on the final output — [0038]–[0039]: the trajectory tracking control unit 214 generates and transmits control commands to the actuator control ECUs 13, 14, 15 so that the host vehicle tracks the adopted trajectory; [0069]: a degenerate (safe-stop) trajectory may be executed.
However, Sakamoto does not explicitly disclose that the first output, the second output, and the final output are capable of satisfying, respectively, a first safety integrity level, a second safety integrity level greater than the first, and the second safety integrity level.
In the related field of endeavor, Pfeifle discloses assigning ISO 26262 safety integrity levels to an analogous decomposed architecture: the complex neural-network processing components carry a lower ASIL (ASIL B); the comparison/validation stage, which votes across independently processed results (2-out-of-3), carries a higher ASIL (ASIL D); and, by decomposition, the validated output of the system as a whole satisfies the higher ASIL ([0005], [0009], [0053], [0055], [0063], [0070]; Figs. 4–5). Sakamoto's rule-based path likewise adopts the independently processed sensor outputs by majority decision on a lockstep processor ([0045], [0058], [0063]–[0064]), which is the same voting principle Pfeifle rates at ASIL D.
Therefore, it would have been obvious to a person of ordinary skill in the art at a time before the effective filing date of the claimed subject matter to design and rate Sakamoto's machine-learning fusion path (microcomputer 111) to a first ASIL such as ASIL B, and Sakamoto's rule-based lockstep fusion path (microcomputer 112) and diagnosis/selection stage (arithmetic block 213) to a second, higher ASIL such as ASIL D — so that the rule-based second output and the adopted final output are capable of satisfying the higher level by applying Pfeifle's ASIL decomposition to Sakamoto's architecture is the use of a known technique to improve a similar device in the same way, with the predictable result of a self-driving controller that satisfies ISO 26262 at the higher level (MPEP 2143(I)(C)). A person of ordinary skill in automotive functional safety would, moreover, have been motivated since Pfeifle identifies ISO 26262 as the governing standard for production automobiles ([0005]).
Consider claim 10, Sakamoto discloses that the first sensor processing pipeline includes at least a first sensor 101 and a first processing component (information processing unit 102) that processes first sensor data (raw data 104) obtained using the first sensor to determine a first intermediate output (object data 103), and that the second sensor processing pipeline includes at least a second sensor 201 and a second processing component (information processing unit 202) that processes second sensor data (raw data 204) obtained using the second sensor to determine a second intermediate output (object data 203) (Fig. 1; [0026], [0028]–[0030]).
Consider claim 11 (examined as depending from claim 10; see the rejection under 35 U.S.C. 112(b)), Sakamoto discloses that the information processing units 102, 202 are microcomputers or FPGAs that preprocess each sensor's raw data into object data ([0024], [0030], [0033]), but does not state whether a given information processing unit implements its detection with a rule-based or a learned algorithm. Claim 11 itself recites the two known alternatives — a rule-based processing component or a learned processing component — for each pipeline.
Pfeifle discloses that object detection from camera images is performed by a (region-based) convolutional neural network in the camera's object identification controller, which Pfeifle describes as state of the art for detecting, classifying, and localizing road objects from camera images ( [0004], [0053]), while acknowledging that traditional computer-vision and deterministic processing techniques remain the conventional approach in ADAS applications ([0003]–[0004]). The examiner further takes official notice (MPEP 2144.03) that, before the effective filing date, object detection in automotive radar sensor modules was conventionally performed by deterministic signal processing — e.g., constant-false-alarm-rate detection, clustering, and Kalman-filter tracking — i.e., by a rule-based processing component.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to implement the information processing unit 102 of Sakamoto's camera 1 with a learned processing component (a CNN/RCNN) as taught by Pfeifle, to obtain state-of-the-art detection and classification from the camera images, while implementing the information processing unit 202 of Sakamoto's radar 2 with the conventional rule-based radar detection and tracking algorithm, because selecting, for each sensor module, one of the two known per-sensor implementations that the art (and claim 11 itself) identifies is a choice from a finite number of predictable options (MPEP 2143(I)(E)), and because keeping the radar and laser modules and the lockstep fusion non-learned preserves the diversity of "principle, … method, and … algorithm" between the rule-based path and the machine-learning raw-data fusion of arithmetic block 211 on which Sakamoto's majority decision and diagnosis rely ([0061], [0063]–[0064]); a per-sensor image classifier is, in any event, different in principle from a multi-sensor raw-data fusion network, so the diversity between Sakamoto's two paths is maintained.
Consider claim 12, Sakamoto discloses that the final output is determined based at least on an arbiter (arithmetic block 213, with the trajectory tracking control unit 214) processing the first output and the second output ([0037]–[0039], [0052], [0055]–[0056]).
Consider claim 13 (as best understood; see the claim objection above), Sakamoto discloses that the second data corresponds to one or more outputs (object data 103, 203) of one or more processing components (information processing units 102, 202) of the first and second sensor processing pipelines, the processing components processing at least sensor data (raw data 104, 204) obtained using the respective sensors 101, 201 of those pipelines ([0026], [0028]–[0029], [0034]).
Consider claim 15, Sakamoto discloses that the determination of the first output includes using early learned sensor fusion: the arithmetic block 211 fuses the raw data 104, 204, 304 of the sensors — data that has not been subjected to the sensors' preprocessing and object-data conversion ([0024]–[0025]) — by inputting the raw data to a neural network or deep neural network ([0031]), which is early learned sensor fusion as that term is used in applicant's specification ([0035], [0039]).
Consider claim 16, Pfeifle discloses that the first safety integrity level corresponds to a first automotive safety integrity level (ASIL B) and the second safety integrity level corresponds to a second ASIL higher than the first ASIL (ASIL D) (Figs. 4–5; [0053], [0055], [0070]). The motivation to combine is as set forth for claim 9.
Consider claim 17, Sakamoto discloses that the system is comprised in a control system and a perception system for an autonomous or semi-autonomous machine ([0002], [0017]–[0018]: a self-driving electronic control unit that recognizes the surrounding environment and drives the vehicle's actuators).
Allowable Subject Matter
Claims 3, 6, 8, and 14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: See respective claimed recitation.
Claim 4 would allowable over prior art once other outstanding rejection has been overcome.
Conclusion
Any response to this Office Action should be faxed to (571) 273-8300 or mailed to:
Commissioner for Patents
P.O. Box 1450
Alexandria, VA 22313-1450
Hand-delivered responses should be brought to
Customer Service Window
Randolph Building
401 Dulany Street
Alexandria, VA 22314
Any inquiry concerning this communication or earlier communications from the
Examiner should be directed to Fayyaz Alam whose telephone number is (571) 270-1102. The Examiner can normally be reached on Monday-Friday from 9:30am to 7:00pm.
If attempts to reach the Examiner by telephone are unsuccessful, the Examiner’s supervisor, Jeanette Parker can be reached on (571) 270-3647. The fax phone number for the organization where this application or proceeding is assigned is (571) 273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free) or 703-305-3028.
Any inquiry of a general nature or relating to the status of this application or proceeding should be directed to the receptionist/customer service whose telephone number is (571) 272-2600.
Fayyaz Alam
September 19, 2026
/FAYYAZ ALAM/
Primary Examiner, Art Unit 2646
Prosecution Insights