DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 01/26/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because Fig. 8, reference character “804” has been used to designate both [0058] “Image-based 3D bounding box 802 includes a center 804.” and [0058] “Radar returns 806 (represented by circles in Fig. 8)”. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1 and 11 require the generation of radar data through the projection of radar data by the following claim element: “generate radar data by projecting radar returns of objects within a scene onto an image plane of camera data of the scene”. The language is not clear in general, and the specification is not found to clarify the meaning. Rather, the claim limitation is mentioned only by repeating the language of the claim at [0018], [0020]. Is the indicated “generation” simply mapping and registering the radar data to the image data, if so, those acts are encompassed with the phrase “projecting radar returns”. If that is the Applicant’s intent, “generating radar data” does not contribute to limiting the metes and bounds of the claims and therefore cannot be given patentable weight. Clarification is required, with reference to the disclosure to clarify the intended limitation to be imposed on the invention. For the purpose of examination, the examiner is not applying patentable weight to the phrase “generate radar data”.
Claims 2-10 and 12-20 are also rejected based on their dependency of the defected parent claims.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-8 and 11-18 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more. Claims 1-8 and 11-18 recite a system and a method for combining radar and camera image data through the use of a neural network to detect objects. This judicial exception is not integrated into a practical application because the claim requires no more than a generic computer to perform generic computer functions that are well-understood, routine, and conventional activities. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because all claims elements, both individually and in combination, are directed to the manipulation of data by a general purpose computer and/or performing by a person. Thus, it does not integrate the abstract idea into a practical application.
An invention is patent-eligible if it claims a “new and useful process, machine, manufacture, or composition of matter.” 35 U.S.C. § 101. However, the Supreme Court has long interpreted 35 U.S.C. § 101 to include implicit exceptions: “[l]aws of nature, natural phenomena, and abstract ideas” are not patentable. E.g., Alice Corp. v. CLS Bank Int’l, 573 U.S. 208, 216(2014).
In determining whether a claim falls within an excluded category, we are guided by the Supreme Court’s two-step framework, described in Mayo and Alice. Id. at 217-18 (citing Mayo Collaborative Servs. v. Prometheus Labs., Inc., 566 U.S. 66, 75-77 (2012)). In accordance with that framework, we first determine what concept the claim is “directed to.” See Alice, 573 U.S. at 219 (“On their face, the claims before us are drawn to the concept of intermediated settlement, i.e., the use of a third party to mitigate settlement risk.”); see also Bilski v. Kappos, 561 U.S. 593, 611 (2010) (“Claims 1 and 4 in petitioners’ application explain the basic concept of hedging, or protecting against risk.”).
Concepts determined to be abstract ideas, and thus patent ineligible, include certain methods of organizing human activity, such as fundamental economic practices {Alice, 573 U.S. at 219-20, Bilski, 561 U.S. at 611); mathematical formulas {Parker v. Flook, 437 U.S. 584, 594-95 (1978)); and mental processes {Gottschalk v. Benson, 409 U.S. 63, 69 (1972)). Concepts determined to be patent eligible include physical and chemical processes, such as “molding rubber products” {Diamond v. Diehr, 450 U.S. 175, 192 (1981)); “tanning, dyeing, making waterproof cloth, vulcanizing India rubber, smelting ores” {id. at 184 n.7 (quoting Corning v. Burden, 56 U.S. 252, 267-68 (1854))); and manufacturing flour {Benson, 409 U.S. at 69 (citing Cochrane v. Deener, 94 U.S. 780, 785 (1876))).
In Diehr, the claim at issue recited a mathematical formula, but the Supreme Court held that “[a] claim drawn to subject matter otherwise statutory does not become nonstatutory simply because it uses a mathematical formula.” Diehr, 450 U.S. at 176; see also id. at 192 (“We view respondents’ claims as nothing more than a process for molding rubber products and not as an attempt to patent a mathematical formula.”). Having said that, the Supreme Court also indicated that a claim “seeking patent protection for that formula in the abstract... is not accorded the protection of our patent laws, . . . and this principle cannot be circumvented by attempting to limit the use of the formula to a particular technological environment.” Id. (citing Benson and Flook); see, e.g., id. at 187 (“It is now commonplace that an application of a law of nature or mathematical formula to a known structure or process may well be deserving of patent protection.”).
If the claim is “directed to” an abstract idea, we turn to the second step of the Alice and Mayo framework, where “we must examine the elements of the claim to determine whether it contains an ‘inventive concept’ sufficient to ‘transform’ the claimed abstract idea into a patent- eligible application.” , 573 U.S. at 221 (quotation marks omitted). “A claim that recites an abstract idea must include ‘additional features’ to ensure ‘that the [claim] is more than a drafting effort designed to monopolize the [abstract idea].”” Id. ((alteration in the original) quoting Mayo, 566 U.S. at 77). “[M]erely requiring] generic computer implementation” fail[s] to transform that abstract idea into a patent-eligible invention.” Id.
The PTO recently published revised guidance on the application of § 101. USPTO’s January 7, 2019 Memorandum, 2019 Revised Patent Subject Matter Eligibility Guidance (“Memorandum”). Under Step 2A of that guidance, we first look to whether the claim recites:
(1) any judicial exceptions, including certain groupings of abstract ideas (i.e., mathematical concepts, certain methods of organizing human activity such as a fundamental economic practice, or mental processes); and
(2) additional elements that integrate the judicial exception into a practical application (see MPEP § 2106.05(a)-(c), (e)-(h)).
Only if a claim (1) recites a judicial exception and (2) does not integrate that exception into a practical application, do we then look to whether the claim:
(3) adds a specific limitation beyond the judicial exception that is not “well- understood, routine, conventional” in the field (see MPEP § 2106.05(d)); or
(4) simply appends well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception.
Analysis
Step 1 – Statutory Category
Claim 1 (and its dependents) recites a system. Thus, the claim is a machine and/or manufacture and falls within one of the statutory categories of invention.
Claim 11 (and its dependents) recites a method. Thus, the claim is to a process, which is one of the statutory categories of invention.
Step 2A, Prong One – Recitation of Judicial Exception
Step 2A of the 2019 Guidance is a two-prong inquiry. In Prong One, we evaluate whether the claim recites a judicial exception. For abstract ideas, Prong One represents a change as compared to prior guidance because we here determine whether the claim recites mathematical concepts, certain methods of organizing human activity, or mental processes.
As set forth above, claims 1-8 and similarly claims 11-18, recite a judicial exception since the claims set forth a plurality of mathematical concepts and mental process as defined at least by the claimed steps of:
generate radar data by projecting radar returns of objects within a scene onto an image plane of camera data of the scene based on extrinsic and intrinsic parameters of a camera and extrinsic parameters of a radar sensor to generate the radar data;
determine image object features and image confidence scores by the image channel, and radar object features and radar confidences by the radar channel; and
combine the image object features with the radar object features using a weighted sum based on image confidence scores and radar confidence scores.
The step of “generate radar data by projecting radar returns of objects within a scene onto an image plane of camera data of the scene” may be accomplished through a series of mathematical operations performed by a generic processor or computer. The claim element amounts to transforming received radar data rather than producing it.
The step of “determine … features” may be performed by observing and evaluating the data received (i.e. radar and image data) which may be practically performed in the human mind using observation, evaluation, judgment, and opinion. The claim does not limit how the determination is performed, and there is nothing about an object feature or confidence score itself that would limit how it can be determined.
The step of “combine the image object features with the radar object features using a weighted sum based on image confidence scores and radar confidence scores.” may be accomplished through a series of mathematical operations performed by a generic processor or computer.
In addition, dependent claims 2-8 and 12-18 further claiming information gleaned from mathematical operations (i.e. claim 2/12 – “determine an image class score”, “determine one or more of object offset, object depth, object size, object rotation, object velocity, object direction, and object center-ness based”; claim 3/13- “combines image object features and radar object features”) or recites specific sets of mathematical operations to be utilized (claim 4/14 – “optical flow”, claim 5/15 – “a feature pyramid network”; claims 7-8/17-18 – utilization of loss functions), and therefore, also falls within the “mathematical ” grouping of abstract ideas. When given their broadest reasonable interpretation in light of the background, the optical flow, a feature pyramid network, loss function, and focal loss are mathematical calculations. The plain meaning of these terms are optimization algorithms, which compute neural network parameters using a series of mathematical calculations.
Since the claims recite an abstract idea, the analysis proceeds to Prong Two to determine whether the claim is “directed to” the judicial exception.
Step 2A, Prong Two – Practical Application
If a claim recites a judicial exception, in Prong Two, we next determine whether the recited judicial exception is integrated into a practical application of that exception by: (a) identifying whether there are any additional elements recited in the claim beyond the judicial exception(s); and (b) evaluating those additional elements individually and in combination to determine whether they integrate the exception into a practical application.
If the recited judicial exception is integrated into a practical application, the claim is not directed to the judicial exception. This evaluation requires an additional element or a combination of additional elements in the claim to apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the exception. If the recited judicial exception is integrated into a practical application, the claim is not directed to the judicial exception.
One way to determine integration into a practical application is when the claimed invention improves the functioning of a computer or improves another technology or technical field. To evaluate an improvement to a computer or technical field, the specification must set forth an improvement in technology and the claim itself must reflect the disclosed improvement. See MPEP 2106.04(d)(1) and 2106.05(a).
The only additional element of claims 1 and 11 is to “receive image data at an image channel of an image/radar convolutional neural network (CNN) and receive the radar data at a radar channel of the image/radar CNN, wherein features are transferred from the image channel to the radar channel at multiple stages”.
The limitations of “receive image data at an image channel of an image/radar convolutional neural network (CNN) and receive the radar data at a radar channel of the image/radar CNN, wherein features are transferred from the image channel to the radar channel at multiple stages” are mere data gathering and output recited at a high level of generality, and thus are insignificant extra-solution activity. See MPEP 2106.05(g) (“whether the limitation is significant”). In addition, all uses of the recited judicial exceptions require such data gathering and output, and, as such, these limitations do not impose any meaningful limits on the claim. These limitations amount to necessary data gathering and outputting. See MPEP 2106.05.
The remaining claim, claim 6/16 discloses “wherein projecting the radar returns onto the image plane of the camera is bounded by a depth range based on resolution levels of the image data and radar data.”. These additional steps are all extraneous pre-solution activity. Viewed as a whole, these additional claim elements do not provide meaningful limitations to transform the abstract idea into a patent eligible application of the abstract idea such that the claims amount to significantly more than the abstract idea itself. Viewed as a whole, these additional claim elements do not provide meaningful limitations to transform the abstract idea into a patent eligible application of the abstract idea such that the claims amount to significantly more than the abstract idea itself. Additionally, the totality of mathematical operations and mental processes are anchored in a specific application, the acts of information processing do not link to or result in affecting an additional system or result in any stated output.
Step 2B – Inventive Concept
For Step 2B of the analysis, it is determined whether the claim adds a specific limitation beyond the judicial exception that is not “well-understood, routine, conventional” in the field.
As stated above, claims 1-8 and 11-18 do not include additional elements that are sufficient to amount to significantly more than the judicial exception. Apart from the judicial exceptions of mental process and mathematical operations, the additional claim elements amount to receiving or transmitting data over a network and are well-understood, routine, conventional activity. See MPEP 2106.05(d), subsection II.
Since the judicial exceptions are not integrated into a practical application because the claim requires no more than data gathering steps that collect necessary data for estimating, analyzing, and evaluating and requires no more than a generic computer to perform operations and generic computer functions that are well-understood, routine, and conventional activities.
The courts have considered the following examples to be well-understood, routine, and conventional when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity: i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network).
As explained by the Supreme Court, the addition of insignificant extra-solution activity does not amount to an inventive concept, particularly when the activity is well-understood or conventional. Viewed as a whole, these additional claim elements do not provide meaningful limitations to transform the abstract idea into a patent eligible application of the abstract idea such that the claims amount to significantly more than the abstract idea itself. Therefore, the claims are patent ineligible under 35 USC 101.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-6, 9-16 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Sless et al. (US Pat. 11270170) in view of Popov et al. (US PG Pub. 20210156963).
Regarding claims 1 and 11, Sless discloses a system and method (Pg. 14, col. 5, lines 15-17; FIG. 2 shows a schematic diagram of an object detection system 200 operable on the processor 44 of the vehicle 10 of FIG. 1, in an embodiment.) comprising:
a computer that includes a processor and a memory, the memory including instructions executable by the processor to (Pg. 13, col. 4, lines 42-44; The at least one controller 34 includes at least one processor 44 and a computer readable storage device or media 46.):
generate radar data by projecting radar returns of objects within a scene onto an image plane of camera data of the scene based on extrinsic and intrinsic parameters of a camera and extrinsic parameters of a radar sensor to generate the radar data (Pg. 14, col. 6, lines 14-20; The radar data neural network 308 receives radar point cloud data from the radar backbone 208, the boundary box from the RPN 306 and the selected feature at the ROI alignment module 304. The radar data neural network 308 generates a fused data set including image data and radar data and also determines a foreground/background score for the received feature.);
receive image data at an image channel of an image/radar convolutional neural network (CNN) and receive the radar data at a radar channel of the image/radar CNN (Fig. 2, Pg. 14, col 5, lines 17-21; The object detection system 200 includes a neural network system 202 and a plurality of heads 204. The neural network system 202 receives both image data related to the object from a camera backbone 206 and radar data related to the object from a radar backbone 208.), wherein features are transferred from the image channel to the radar channel at multiple stages (Page 14, col. 6, lines 1-20; Data can be pulled from an intermediate convolutional layer for separate processing. In the illustrative embodiment, network layer data is pulled from the fourth convolutional network layer (Conv4) and sent to the RPN 306....The ROI alignment module 304 compares the proposed boundary box to image data from the last convolutional network layer (e.g., Conv6) of the image neural network 302 and adjusts a size of the proposed boundary box in order to isolate a feature of the object within the final network layer data... The radar data neural network 308 receives radar point cloud data from the radar backbone 208, the boundary box from the RPN 306 and the selected feature at the ROI alignment module 304. The radar data neural network 308 generates a fused data set including image data and radar data and also determines a foreground/background score for the received feature.);
determine image object features and image confidence scores by the image channel, and radar object features and radar confidences by the radar channel (Pg. 14, col 5, lines 21-25; The neural network system 202 processes the image data and the radar data in order to identify features and regions within the image data that can be proposed to the plurality of heads 204 in order to detect the object.); and
combine the image object features with the radar object features based on image confidence scores and radar confidence scores (Pg. 15, col. 7, lines 32-39; The image feature layer data f.sub.C6 is sent through a first image convolution branch 504 and a second image convolution branch 506. The output from the first image convolution branch 504 produces a classification score. The classification score from the radar convolutional neural network 502 and the classification score from the first image convolution branch 504 are combined in order to generate a final classification score (CS) for the object.). Sless fails to explicitly teach using a weighted sum based on the image and radar confidence scores to combine the image and radar object features.
However, Popov teaches a system and method for neural network detection of objects ([0005] Embodiments of the present disclosure relate to object detection for autonomous machines using deep neural networks (DNNs). Systems and methods are disclosed that use object detection techniques to identify or detect instances of moving or stationary obstacles (e.g., cars, trucks, pedestrians, cyclists, etc.) and other objects within environments for use by autonomous vehicles, semi-autonomous vehicles, robots, and/or other object types.) where weighted sum based on the image and radar confidence scores to combine the image and radar object features ([0163] The DLA (Deep Learning Accelerator) may be used to run any type of network to enhance control and driving safety, including for example, a neural network that outputs a measure of confidence for each object detection. Such a confidence value may be interpreted as a probability, or as providing a relative “weight” of each detection compared to other detections.).
Sless and Popov are both considered to be analogous to the claimed invention because they are in the same field of endeavor of image processing with neural networks technology. A person of ordinary skill in the art would have had the technological capabilities before the effective filing date of the claimed invention to modify the system and method of Sless by incorporating the weighted sum technique for combining of Popov to yield a predictable result of fused object features that represent the accuracy of identification feature identification, this results in improved object detection and understanding the quality of the object detection.
Regarding claims 2 and 12, Sless as modified by Popov teaches the system of claim 1 and the method of claim 11. Sless further teaches wherein a classification feature block of the image/radar CNN includes one or more of: the classification feature block that receives one or more classification features to determine an image class score based on the image object features and the radar object features; and a regression feature block that receives one or more regression features that determine one or more of object offset, object depth, object size, object rotation, object velocity, object direction, and object center-ness based on the image object features and the radar object features (Pg. 14, col. 5, lines 22-32; The neural network system 202 processes the image data and the radar data in order to identify features and regions within the image data that can be proposed to the plurality of heads 204 in order to detect the object. Object proposals created at the neural network system 202 are sent to the plurality of heads 204. The plurality of heads 204 determines various parameters of the object 50 from the object proposals, such parameters including a classification of the object, a location and size of the object, an observation angle of the object, a range of the object and a two-dimensional velocity vector for the object.).
Regarding claims 3 and 13, Sless as modified by Popov teaches the system of claim 1 and the method of claim 11. Sless further teaches wherein output of the image/radar CNN includes one or more of: a depth fusion block that combines image object features and radar object features output by a head network including depth, location, orientation, size (Page 14, col. 6, lines 35-47; The heads 204 include various heads for determining parameters of an object from the data from the neural network system 202. These heads include, but are not limited to, a two-dimensional (2D) head 320, three-dimensional (3D) head 322, observation angle head 324, depth head 326, and a velocity head 328. The two-dimensional head 320 receives a two-dimensional boundary box from the ROI alignment module 304 as well as the fused set of data f.sub.fused.sup.glb from the radar data neural network 308. The 2D head 320 outputs a location and dimensions of the boundary box within an image. The two-dimensional head 320 also generates a classification score (CS) for the feature within the two-dimensional boundary box.), and confidence for 3D bounding boxes surrounding objects included in image object features and radar object features (Page 14, col. 6, lines 48-57; The 3D head 322 receives a 3D boundary box and determines a width, height and length dimension for the 3D boundary box. The observation angle head 324 receives the 3D boundary box and determines an orientation of the 3D boundary box (and thus, of the object within the 3D boundary box) with respect to a line of sight between the vehicle and the object. The depth head 326 receives the 3D boundary box as well as the fused data set from the radar data neural network 308 and outputs a range from the vehicle to the object based on this data.).
Regarding claims 4 and 14, Sless as modified by Popov teaches the system of claim 1 and the method of claim 11. Sless fails to explicitly teach wherein the radar channel includes an optical flow channel.
However, Popov teaches a system and method for neural network detection of objects wherein the radar channel includes an optical flow channel ([0162] In some examples, the programmable vision accelerator (PVA) may be used to perform dense optical flow. According to process raw RADAR data (e.g., using a 4D Fast Fourier Transform) to provide Processed RADAR.). It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention to incorporate that the radar channel includes an optical flow channel as in Popov with the system and method of Sless with a reasonable expectation of success, as both inventions are directed to the same field of endeavor – object detection through the use of neural networks on radar and image data. The combination would improve the quality of object detection.
Regarding claims 5 and 15, Sless as modified by Popov teaches the system of claim 1 and the method of claim 11. Sless fails to explicitly teach wherein one or more of the radar channels and the image channels include a feature pyramid network.
However, Popov teaches a system and method for neural network detection of objects wherein one or more of the radar channels and the image channels include a feature pyramid network ([0029] For example, the DNN may include a common trunk (or stream of layers) with several heads (or at least partially discrete streams of layers) for predicting different outputs based on the input data. The common trunk may be implemented using encoder and decoder components with skip connections, in embodiments (e.g., similar to a Feature Pyramid Network, U-Net, etc.).). A person of ordinary skill in the art would have had the technological capabilities before the effective filing date of the claimed invention to incorporate that one or more of the radar channels and the image channels include a feature pyramid network of Popov with the system and method of Sless with a reasonable expectation of success, as both inventions are directed to the same field of endeavor – object detection through the use of neural networks on radar and image data. The combination would improve the quality of object detection.
Regarding claims 9 and 19, Sless as modified by Popov teaches the system of claim 1 and the method of claim 11. Sless further teaches wherein the camera, the radar sensor, and the image/radar CNN are included in a vehicle and predictions output from the image/radar CNN are used to operate the vehicle (Pg. 14, col. 5, lines 1-22; The instructions, when executed by the processor 44, receive and process signals from the sensor system 28, perform logic, calculations, methods and/or algorithms for automatically controlling the components of the autonomous vehicle 10, and generate control signals to the actuator system 30 to automatically control the components of the vehicle 10 based on the logic, calculations, methods, and/or algorithms. In one embodiment, data from sensing devices 40a-40n are communicated to processor 44. Processor 44 operates various methods disclosed herein to detect an object and determine various parameters of the object. The detection and parameters can be used to navigate the vehicle with respect to the object. FIG. 2 shows a schematic diagram of an object detection system 200 operable on the processor 44 of the vehicle 10 of FIG. 1, in an embodiment. The object detection system 200 includes a neural network system 202 and a plurality of heads 204. The neural network system 202 receives both image data related to the object from a camera backbone 206 and radar data related to the object from a radar backbone 208.).
Regarding claims 10 and 20, Sless as modified by Popov teaches the system of claim 9 and the method of claim 19. Sless further teaches wherein the vehicle is operated by controlling one or more of vehicle propulsion, vehicle steering, or vehicle brakes based on the predictions output by the image/radar CNN (Page 13 col. 3 line 63 - col. 4 line 2; As shown, the vehicle 10 generally includes a propulsion system 20, a transmission system 22, a steering system 24, a brake system 26, a sensor system 28, an actuator system 30, and at least one controller 34. The propulsion system 20 may, in various embodiments, include an internal combustion engine, an electric machine such as a traction motor, and/or a fuel cell propulsion system.).
Claim(s) 6 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Sless as modified by Popov as applied to claims 1 and 11 above, and further in view of Jiang (US PG Pub. 20210012165).
Regarding claims 6 and 16, Sless as modified by Popov teaches the system of claim 1 and the method of claim 11. Sless as modified by Popov fails to explicitly teach wherein projecting the radar returns onto the image plane of the camera is bounded by a depth range based on resolution levels of the image data and radar data.
However, Jiang teaches a method for multi-sensor fusion ([0006] The invention provides a data processing method, a device and a multi-sensor fusion method for multi-sensor fusion, so as to solve the problem that the data detected by the sensor lacks deep fusion.) wherein projecting the radar returns onto the image plane of the camera is bounded by a depth range based on resolution levels of the image data and radar data ([0217] If the radar data is directly matched with the camera pixel data, the radar data is too sparse. If we want to match the camera image data point by point according to the pixel relationship, we need to do some processing firstly and convert the radar data into intensive class image data with tensor structure… Its 2D spatial resolution is equal to the pixel resolution of the matched camera in the system, and the one-to-one mapping relationship between radar data and camera data is established; The target (image) detected by radar is mapped to the 2D mapping surface of radar target to generate “radar perception matrix”; On the matrix layer (depth), the data (layer) of radar perception matrix is composed of the following “radar raw data input”).
Sless, Popov and Jiang are all considered to be analogous to the claimed invention because they are in the same field of endeavor of image processing with neural networks technology. A person of ordinary skill in the art would have had the technological capabilities before the effective filing date of the claimed invention to incorporate the wherein projecting the radar returns onto the image plane of the camera is bounded by a depth range based on resolution levels of the image data and radar data of Jiang with the system and method of Sless as modified by to gain the advantage of a means of scaling the radar data to the image data to improve the data fusion as noted by Jiang ([0005] In the existing related technologies, the data detected by each sensor is independent of each other and lacks deep fusion. Furthermore, when using the detected data for feature extraction and data mining, the ability of environment perception and target detection is weak.).
Claim(s) 7-8 and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Sless as modified by Popov as applied to claims 1 and 11 above, and further in view of Laddah et al. (US PG Pub. 20220035376).
Regarding claims 7-8 and 17-18, Sless as modified by Popov teaches the system of claim 1 and the method of claim 11 as explained above and incorporated herein. Sless as modified by Popov fails to explicitly teach wherein the image/radar CNN is trained using a loss function based on summing classification loss, regression loss, attribute classification loss, direction loss, center-ness loss for respective object features and wherein the classification loss is based on a focal loss.
However, Laddah teaches a [method/system] wherein the image/radar CNN is trained using a loss function ([0006] The method can further include training the network model based at least in part on the loss function.) based on summing classification loss, regression loss, attribute classification loss, direction loss, center-ness loss for respective object features ([0045] For example, the detection loss can be a multi-task loss comprising a weighted sum of a classification and regression loss… In some implementations, each object i can be parameterized by its center (x.sub.i, y.sub.i), orientation (Θ.sub.I), and size (w.sub.i, h.sub.i). In some implementations, smooth L1 loss can be used to train regression parameters of each object.) and wherein the classification loss is based on a focal loss ([0045] The classification loss can be trained using a focal loss for each coordinate frame cell for being at the center of a particular object class.)
Sless, Popov and Laddah are all considered to be analogous to the claimed invention because they are in the same field of endeavor of radar/ image data processing and fusing using neural networks for vehicular applications technology. A person of ordinary skill in the art would have had the technological capabilities before the effective filing date of the claimed invention to incorporate the system and method of Sless as modified by Popov with the image/radar CNN training using the loss function teachings of Laddah with a reasonable expectation of success, as both inventions are directed to the same field of endeavor – object detection through the use of neural networks on radar and image data. The combination would improve the quality of object detection.
For applicant’s benefit portions of the cited reference(s) have been cited to aid in the review of the rejection(s). While every attempt has been made to be thorough and consistent within the rejection it is noted that the PRIOR ART MUST BE CONSIDERED IN ITS ENTIRETY, INCLUDING DISCLOSURES THAT TEACH AWAY FROM THE CLAIMS. See MPEP 2141.02 VI.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20200249316 discloses a radar system in an autonomous vehicle for object detection and classification. The radar system has a radar module having a dynamically controllable beam steering antenna and a perception module. The perception module includes a machine learning module trained on a first set of data and retrained on a second set of data to generate a set of object locations and classifications, and a classifier to use velocity information combined with the set of object locations and classifications to output a set of classified data.
US 20210397907 discloses a computer implemented method for object detection comprises the following steps carried out by computer hardware components: acquiring a plurality of lidar data sets from a lidar sensor; acquiring a plurality of radar data sets from a radar sensor; acquiring at least one image from a camera; determining concatenated data based on casting the plurality of lidar data sets and the plurality of radar data sets to the at least one image; and detecting an object based on the concatenated data.
US 20190258878 discloses techniques for how to determine object data representative of locations of detected objects in a field of view. One or more clusters of the detected objects may be generated based at least in part on the locations and features of the cluster may be determined for use as inputs to a machine learning model(s). A confidence score, computed by the machine learning model(s) based at least in part on the inputs, may be received, where the confidence score may be representative of a probability that the cluster corresponds to an object depicted at least partially in the field of view. Further examples provide approaches for determining ground truth data for training object detectors, such as for determining coverage values for ground truth objects using associated shapes, and for determining soft coverage values for ground truth objects.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHN BS ABRAHAM whose telephone number is (571)272-4145. The examiner can normally be reached Monday - Friday 9:00 am - 5:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jack Keith can be reached at (571)272-6878. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JBSA/Examiner, Art Unit 3646
/JACK W KEITH/Supervisory Patent Examiner, Art Unit 3646