DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on 01/02/2025 have been considered and placed in the applicant file.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier.
Claims 1-9 and 12 recites limitations that use words like “means” (or “step”) or similar terms with functional language and do invoke 35 U.S.C. 112(f):
Claim 1, 8 and 12; recites the limitation, “an object recognition unit configured to…” [Line 2; Line 1-2; Line 3].
Claim 1 and 12; recites the limitation, “a contribution ratio calculation unit configured to…” [Line 5; Line 5].
Claim 1-9 and 12; recites the limitation, “a recognition processing control unit configured to...” [Claim 1: Line 7; Claim 2-9: Line 1-2; Claim 12: Line 7].
Because these claim limitation(s) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
After a careful analysis, as disclosed above, and a careful review of the specification the following limitations in claims 1-9 and 12:
(i) “an object recognition unit” (Fig. 3-4, #241. Paragraph [0122-0124 and 0129]- the object recognition unit 241 is described as performing object recognition processing on the basis of image data, radar image data for recognition, and the point cloud data and supplying the data indicating the results of the object recognition to the vehicle control unit 251. Fig. 3 illustrates the object recognition unit 241 as a black box. Fig. 3 also illustrates the recognizer 212 as containing a recognition processing unit 234, and the recognition processing unit 234 as containing an object recognition unit 241, a contribution ratio calculation unit 242, and a recognition processing control unit 243. The recognizer 212 is further described as executing recognition processing for objects based on image, radar and LIDAR data and the recognition processing unit 234 is further described as performing object recognition processing. Additionally, FIG. 4 shows an exemplary configuration of an object recognition model 301 used in the object recognition unit 241 in FIG. 3 (wherein the object recognition unit 241 has sufficient structure associated with it and is a machine learning model executed by a processor)).
(ii) “a contribution ratio calculation unit” (Fig. 3, #242. Paragraph [0125-0126]-the contribution ratio calculation unit 242 is described as calculating the contribution ratio, which indicates the degree of contribution of each sensing data piece from each sensor of the sensing unit 211 to recognition processing by the object recognition unit 241. Fig. 3 illustrates the contribution ratio calculation unit 242 as a black box. Fig. 3 also illustrates the recognizer 212 as containing a recognition processing unit 234, and the recognition processing unit 234 as containing an object recognition unit 241, a contribution ratio calculation unit 242, and a recognition processing control unit 243. The recognizer 212 is further described as executing recognition processing for objects based on image, radar and LIDAR data and the recognition processing unit 234 is further described as performing object recognition processing. (wherein the contribution ratio calculation unit 242 does not have sufficient structure associated with it and is a processor)).
(iii) “a recognition processing control unit” (Fig. 3, #243. Paragraph [0126 and 0156-0161]-the recognition processing control unit 243 is described as controlling the sensors of the sensing unit 211, the image processing unit 231, the signal processing unit 232, the signal processing unit 233, and the object recognition unit 241 on the basis of the contribution ratio of each sensing data piece to the recognition processing, thereby restricting the sensing data to be used for recognition processing. The recognition processing control unit 243 is further described as executing one or more types of processing. Fig. 3 illustrates the recognition processing control unit 243 as a black box. Fig. 3 also illustrates the recognizer 212 as containing a recognition processing unit 234, and the recognition processing unit 234 as containing an object recognition unit 241, a contribution ratio calculation unit 242, and a recognition processing control unit 243. The recognizer 212 is further described as executing recognition processing for objects based on image, radar and LIDAR data and the recognition processing unit 234 is further described as performing object recognition processing. (wherein the recognition processing control unit 243 has sufficient structure associated with it and is a processor)).
If applicant does not intend to have these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claims 1 and 10-12 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by SHARMA et al. (US 20190050692 A1), hereinafter referenced as SHARMA.
Regarding claim 1, SHARMA explicitly teaches an information processing device (Fig. 6, #600 called a computer system. Paragraph [0066]-SHARMA discloses FIG. 6 is a block diagram illustrating a machine in the example form of a computer system 600, within which a set or sequence of instructions may be executed to cause the machine to perform any one of the methodologies. Please also see Fig. 1), comprising:
an object recognition unit (Fig. 6, #602 called a processor. Paragraph [0067]-SHARMA discloses computer system 600 includes at least one processor 602 (e.g., a central processing unit (CPU), a graphics processing unit (GPU) or both, processor cores, compute nodes, etc.). Please also see Fig. 1 and read paragraph [0038-0040]) configured to combine sensing data pieces (Fig. 2, #210, #212, #214, and #216 called video signals, lidar signals, acoustic signals and radar signals, respectively. Paragraph [0031]) from multiple types of sensors (Fig. 2 #200, #202, #204, and #206 called a camera, LIDAR, acoustic sensor and radar. Paragraph [0031]-SHARMA discloses FIG. 2 is a diagram illustrating data and control flow for context-based digital signal processing for efficient object detection. A number of sensors may be used, including a camera 200, LiDAR 202, acoustic sensor 204, and radar 206 (wherein additional sensors may be used such as vibration sensors, olfactory sensors, a GPS unit, an IMU, time of the day, weather sensors, etc.). In paragraph [0067]-SHARMA discloses the computer system 600 may additionally include a one or more sensors (not shown)) that perform sensing around a vehicle (Fig. 1, #104 called a vehicle. Paragraph [0022]-SHARMA discloses the host vehicle may be equipped with sensors to detect objects around the host vehicle. In paragraph [0024]-SHARMA discloses the vehicle 104 may be of any type of vehicle, such as a commercial vehicle, a consumer vehicle, a recreation vehicle, a car, a truck, a motorcycle, a boat, a drone, a robot, an airplane, a hovercraft, or any mobile craft. In paragraph [0025]-SHARMA discloses the vehicle 104 includes a sensor array, which may include various forward, side, and rearward facing cameras, radar, LiDAR, ultrasonic, or similar sensors) so as to perform object recognition processing (Fig. 2. Paragraph [0031]-SHARMA discloses object detection algorithms 210, 212, 214, and 216 are used for each respective sensor. In paragraph [0032]-SHARMA discloses contextual information is gathered and a context is output (operation 220). The contextual information may be gathered, at least in part, from the sensors 200, 202, 204, or 206 (wherein contextual information may also be obtained from additional sensors (e.g. vibration sensors, olfactory sensors, a GPS unit, etc.). In paragraph [0042]-SHARMA discloses one or more sensors, detectors, detection algorithms, context evaluations, or the like may be operated in parallel. LiDAR, radar, and camera signals may be processed in parallel and the resulting information may be combined after weighting as per the context. Please also see Fig. 4);
PNG
media_image1.png
1085
1047
media_image1.png
Greyscale
FIGURE 2 illustrates a multi-modal vehicle sensor system that fuses and adaptively weights sensor output based on sensor contribution and functionality.
a contribution ratio calculation unit (Fig. 6, #602 called a processor. Paragraph [0067]) configured to calculate a contribution ratio of each of the sensing data pieces (Fig. 2, #210, #212, #214, and #216 called video signals, lidar signals, acoustic signals and radar signals, respectively. Paragraph [0031]) in the recognition processing (Fig. 3. Paragraph [0022]-SHARMA discloses using context-based sensor fusion, sensor signals are selectively and dynamically weighted with reference to the context in which the sensors are operating. In paragraph [0034]-SHARMA discloses relative weighting is applied to each of the object detection algorithm outputs (operation 230) based on the context identified by operation 220 (wherein each sensor type may have different effective operating ranges, and contextual information may indicate whether vehicle #104 is presented with various situational challenges that may impact sensor effectiveness, such as tunnels, glare, shadow, or smoke, which can either obstruct image sensors or result in false readings from acoustic sensors). In paragraph [0038]-SHARMA discloses the weights for the relevant sensors under specific context are assigned values between 0 to 1, when computed automatically by a machine learning framework. The combined output is a linear combination of the output of the different object detection modules (e.g., object detection algorithms 210, 212, 214, and 216 of FIG. 2, which may be referred to as sensor detection output S.sub.1, S.sub.2, S.sub.3, etc.) as shown in FIG. 2, as per their respective weights (W.sub.1, W.sub.2, W.sub.3, etc.) (wherein the combined output may be based, for example, on the following equation: S.sub.1×W.sub.1+S.sub.2×W.sub.2+S.sub.3×W.sub.3+ . . . +S.sub.n×W.sub.n); and
a recognition processing control unit (Fig. 6, #602 called a processor. Paragraph [0067]) configured to restrict the sensing data pieces to be used for the recognition processing on the basis of the contribution ratio (Fig. 3. Paragraph [0033]-SHARMA discloses using a continuous learning system, the context-based digital signal processing may adapt to when sensors are not working properly and modify weights. Sensors that are overdue on maintenance, out of specification, uncalibrated, or otherwise malfunctioning, may have their corresponding detection algorithm weighted less, indicating less confidence in the result. Please also read paragraph [0034-0039]).
Regarding claim 10, SHARMA explicitly teaches the information processing device according to claim 1, SHARMA further teaches wherein the multiple types of sensors (Fig. 2 #200, #202, #204, and #206 called a camera, LIDAR, acoustic sensor and radar. Paragraph [0031]) include at least two of a camera, a LiDAR, a radar, and an ultrasonic sensor (Fig. Paragraph [0031]-SHARMA discloses a number of sensors may be used, including a camera 200, LiDAR 202, acoustic sensor 204, and radar 206 (wherein additional sensors may be used such as vibration sensors, olfactory sensors, a GPS unit, an IMU, time of the day, weather sensors, etc.). Please also see Fig. 6 and read paragraph [0067]).
Regarding claim 11, SHARMA explicitly teaches an image processing method comprising:
combining sensing data pieces (Fig. 2, #210, #212, #214, and #216 called video signals, lidar signals, acoustic signals and radar signals, respectively. Paragraph [0031]) from multiple types of sensors (Fig. 2 #200, #202, #204, and #206 called a camera, LIDAR, acoustic sensor and radar. Paragraph [0031]-SHARMA discloses a number of sensors may be used, including a camera 200, LiDAR 202, acoustic sensor 204, and radar 206 (wherein additional sensors may be used such as vibration sensors, olfactory sensors, a GPS unit, an IMU, time of the day, weather sensors, etc.). Please also see Fig. 6 and read paragraph [0067]) that perform sensing around a vehicle (Fig. 1, #104 called a vehicle. Paragraph [0022] [0022]-SHARMA discloses the host vehicle may be equipped with sensors to detect objects around the host vehicle. In paragraph [0025]-SHARMA discloses the vehicle 104 includes a sensor array, which may include various forward, side, and rearward facing cameras, radar, LiDAR, ultrasonic, or similar sensors (wherein the vehicle 104 may be of any type of vehicle, such as a commercial vehicle, a consumer vehicle, a recreation vehicle, a car, a truck, a motorcycle, a boat, a drone, a robot, an airplane, a hovercraft, or any mobile craft). Please also read paragraph [0024]), thereby performing object recognition processing (Fig. 2. Paragraph [0031]-SHARMA discloses object detection algorithms 210, 212, 214, and 216 are used for each respective sensor. In paragraph [0032]-SHARMA discloses contextual information is gathered and a context is output (operation 220). The contextual information may be gathered, at least in part, from the sensors 200, 202, 204, or 206 (wherein contextual information may also be obtained from additional sensors (e.g. vibration sensors, olfactory sensors, a GPS unit, etc.). In paragraph [0042]-SHARMA discloses one or more sensors, detectors, detection algorithms, context evaluations, or the like may be operated in parallel. LiDAR, radar, and camera signals may be processed in parallel and the resulting information may be combined after weighting as per the context. Please also see Fig. 4);
calculating a contribution ratio of each of the sensing data pieces (Fig. 2, #210, #212, #214, and #216 called video signals, lidar signals, acoustic signals and radar signals, respectively. Paragraph [0031]) in the recognition processing (Fig. 3. Paragraph [0022]-SHARMA discloses using context-based sensor fusion, sensor signals are selectively and dynamically weighted with reference to the context in which the sensors are operating. In paragraph [0034]-SHARMA discloses relative weighting is applied to each of the object detection algorithm outputs (operation 230) based on the context identified by operation 220 (wherein each sensor type may have different effective operating ranges, and contextual information may indicate whether vehicle #104 is presented with various situational challenges that may impact sensor effectiveness, such as tunnels, glare, shadow, or smoke, which can either obstruct image sensors or result in false readings from acoustic sensors). In paragraph [0038]-SHARMA discloses the weights for the relevant sensors under specific context are assigned values between 0 to 1, when computed automatically by a machine learning framework. The combined output is a linear combination of the output of the different object detection modules (e.g., object detection algorithms 210, 212, 214, and 216 of FIG. 2, which may be referred to as sensor detection output S.sub.1, S.sub.2, S.sub.3, etc.) as shown in FIG. 2, as per their respective weights (W.sub.1, W.sub.2, W.sub.3, etc.)); and
restricting the sensing data pieces to be used for the recognition processing on the basis of the contribution ratio (Fig. 3. Paragraph [0033]-SHARMA discloses using a continuous learning system, the context-based digital signal processing may adapt to when sensors are not working properly and modify weights. Sensors that are overdue on maintenance, out of specification, uncalibrated, or otherwise malfunctioning, may have their corresponding detection algorithm weighted less, indicating less confidence in the result. Please also read paragraph [0034-0039]).
Regarding claim 12, SHARMA explicitly teaches an information processing system (Fig. 6, #600 called a computer system. Paragraph [0066]-SHARMA discloses FIG. 6 is a block diagram illustrating a machine in the example form of a computer system 600, within which a set or sequence of instructions may be executed to cause the machine to perform any one of the methodologies. Please also see Fig. 1), comprising:
multiple types of sensors (Fig. 2, #200, #202, #204, and #206 called a camera, LIDAR, acoustic sensor and radar. Paragraph [0031]-SHARMA discloses a number of sensors may be used, including a camera 200, LiDAR 202, acoustic sensor 204, and radar 206 (wherein additional sensors may be used such as vibration sensors, olfactory sensors, a GPS unit, an IMU, time of the day, weather sensors, etc.). Please also see Fig. 6 and read paragraph [0067]) configured to perform sensing around a vehicle (Fig. 1, #104 called a vehicle. Paragraph [0022] [0022]-SHARMA discloses the host vehicle may be equipped with sensors to detect objects around the host vehicle. In paragraph [0025]-SHARMA discloses the vehicle 104 includes a sensor array, which may include various forward, side, and rearward facing cameras, radar, LiDAR, ultrasonic, or similar sensors (wherein the vehicle 104 may be of any type of vehicle, such as a commercial vehicle, a consumer vehicle, a recreation vehicle, a car, a truck, a motorcycle, a boat, a drone, a robot, an airplane, a hovercraft, or any mobile craft). Please also read paragraph [0024]);
an object recognition unit (Fig. 6, #602 called a processor. Paragraph [0067]. Please also read paragraph [0038-0040]) configured to combine sensing data pieces (Fig. 2, #210, #212, #214, and #216 called video signals, lidar signals, acoustic signals and radar signals, respectively. Paragraph [0031]) from the respective sensors (Fig. 2, #200, #202, #204, and #206 called a camera, LIDAR, acoustic sensor and radar. Paragraph [0031]) and perform object recognition processing (Fig. 2. Paragraph [0031]-SHARMA discloses object detection algorithms 210, 212, 214, and 216 are used for each respective sensor. In paragraph [0032]-SHARMA discloses contextual information is gathered and a context is output (operation 220). The contextual information may be gathered, at least in part, from the sensors 200, 202, 204, or 206 (wherein contextual information may also be obtained from additional sensors (e.g. vibration sensors, olfactory sensors, a GPS unit, etc.). In paragraph [0042]-SHARMA discloses one or more sensors, detectors, detection algorithms, context evaluations, or the like may be operated in parallel. LiDAR, radar, and camera signals may be processed in parallel and the resulting information may be combined after weighting as per the context. Please also see Fig. 4);
a contribution ratio calculation unit (Fig. 6, #602 called a processor. Paragraph [0067]) configured to calculate a contribution ratio of each of the sensing data pieces (Fig. 2, #210, #212, #214, and #216 called video signals, lidar signals, acoustic signals and radar signals, respectively. Paragraph [0031]) in the recognition processing (Fig. 3. Paragraph [0022]-SHARMA discloses using context-based sensor fusion, sensor signals are selectively and dynamically weighted with reference to the context in which the sensors are operating. In paragraph [0034]-SHARMA discloses relative weighting is applied to each of the object detection algorithm outputs (operation 230) based on the context identified by operation 220 (wherein each sensor type may have different effective operating ranges, and contextual information may indicate whether vehicle #104 is presented with various situational challenges that may impact sensor effectiveness, such as tunnels, glare, shadow, or smoke, which can either obstruct image sensors or result in false readings from acoustic sensors). In paragraph [0038]-SHARMA discloses the weights for the relevant sensors under specific context are assigned values between 0 to 1, when computed automatically by a machine learning framework. The combined output is a linear combination of the output of the different object detection modules (e.g., object detection algorithms 210, 212, 214, and 216 of FIG. 2, which may be referred to as sensor detection output S.sub.1, S.sub.2, S.sub.3, etc.) as shown in FIG. 2, as per their respective weights (W.sub.1, W.sub.2, W.sub.3, etc.)); and
a recognition processing control unit (Fig. 6, #602 called a processor. Paragraph [0067]) configured to restrict the sensing data pieces to be used for the recognition processing on the basis of the contribution ratio (Fig. 3. Paragraph [0033]-SHARMA discloses using a continuous learning system, the context-based digital signal processing may adapt to when sensors are not working properly and modify weights. Sensors that are overdue on maintenance, out of specification, uncalibrated, or otherwise malfunctioning, may have their corresponding detection algorithm weighted less, indicating less confidence in the result (wherein low contribution sensors are, for example, sensors that not working properly, or unable to properly gather information due to situational factors are low contribution sensors). Please also read paragraph [0034-0039]).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2-4 are rejected under 35 U.S.C. 103 as being unpatentable over SHARMA et al. (US 20190050692 A1), hereinafter referenced as SHARMA in view of SOLTANI et al. (US 20210097783 A1), hereinafter referenced as SOLTANI.
Regarding claim 2, SHARMA explicitly teaches the information processing device according to claim 1, although SHARMA explicitly teaches wherein the recognition processing control unit (Fig. 6, #602 called a processor. Paragraph [0067]) is configured to restrict use of low contribution ratio sensing data (Fig. 2. Paragraph [0033]-SHARMA discloses using a continuous learning system, the context-based digital signal processing may adapt to when sensors are not working properly and modify weights. Sensors that are overdue on maintenance, out of specification, uncalibrated, or otherwise malfunctioning, may have their corresponding detection algorithm weighted less, indicating less confidence in the result. In paragraph [0038]-SHARMA discloses the combined output is a linear combination of the output of the different object detection modules (e.g., object detection algorithms 210, 212, 214, and 216 of FIG. 2, which may be referred to as sensor detection output S.sub.1, S.sub.2, S.sub.3, etc.) as shown in FIG. 2, as per their respective weights (W.sub.1, W.sub.2, W.sub.3, etc.). The combined output may be derived from the following equation: Combined Output=S.sub.1×W.sub.1+S.sub.2×W.sub.2+S.sub.3×W.sub.3+ . . . +S.sub.n×W.sub.n). Please also read paragraph [0032 and 0035-0037]).
SHARMA fails to explicitly teach low contribution ratio sensing data which is the sensing data with the contribution ratio equal to or less than a prescribed threshold value in the recognition processing.
However, SOLTANI explicitly teaches low contribution ratio sensing data (Fig. 3. Paragraph [0029]-SOLTANI discloses FIG. 3 is a diagram of an adaptive sensor fusion system 300. Adaptive sensor fusion system 300 is a system based on reinforcement learning (REINFORCEMENT LEARNING) 310 that determines confidence levels applied to a plurality of vehicle sensors 116 using fusion weights 312 assigned to various data sources (SENSOR DATA) 306 for sensor fusion (SENSOR FUSION) 314. An adaptive sensor fusion system 300 is used to change or adapt the manner in which data from multiple sensors is combined in a multi-sensor, multi-modality vehicle system as the measured reliability or accuracy of the sensors change over time (wherein multi-modality data sources include video cameras, lidar, radar and sonar). Adaptive sensor fusion system 300 uses reinforcement learning 310 to determine fusion weights 312 input to sensor fusion 314 that determine how multi-modality data sources 306 will be combined or fused by sensor fusion 314 to be output as output data 316. Please also read paragraph [0035] Reinforcement learning 310 includes multiple deep neural networks that output multiple fusion weights 312, for example w.sub.L , w.sub.R, w.sub.S, and w.sub.C corresponding to distance measurements D.sub.L, D.sub.R, D.sub.S and D.sub.C respectively output by data sources 306 corresponding to object locations for lidar, radar, sonar and video camera, respectively.) which is the sensing data with the contribution ratio (Fig. 3, #312 called fusion weights. Paragraph [0029]) equal to or less than a prescribed threshold value in the recognition processing (Fig. 3. Paragraph [0036]-SOLTANI discloses fusion weights 312 are also passed to a system health monitor (HEALTH MONITOR) 318 that compares the fusion weights 312 to previously acquired fusion weights 312 to determine changes in data sources 306, where a decrease in a weight corresponding to a particular sensor can indicate that the sensor is no longer operating reliably. Each fusion weight w can be compared to a pre-set threshold values T. When the weighting value corresponding to a sensor drops below a threshold, i.e. w<T, sensor performance can be determined to be below the expected level. If a camera is obscured by dirt or ice and no longer producing reliable data, reinforcement learning 310 would output fusion weights 312 that reflect the lower reliability of the camera).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SHARMA of having an information processing device, with the teachings of SOLTANI of having low contribution ratio sensing data which is the sensing data with the contribution ratio equal to or less than a prescribed threshold value in the recognition processing.
Wherein SHARMA’s device having wherein the recognition processing control unit is configured to restrict use of low contribution ratio sensing data which is the sensing data with the contribution ratio equal to or less than a prescribed threshold value in the recognition processing.
The motivation behind the modification would have been to obtain a method that improves object tracking and environment monitoring, since both SHARMA and SOLTANI concern multi-modality sensor fusion. Wherein SHARMA’s provides improves object detection and tracking by selectively weighting sensor data, while SOLTANI’s systems and methods improve automated tasks including vehicle operation and robot guidance by providing an adaptive scheme based on reinforcement learning that can automatically tune the degree of confidence assigned to various data sources to guide fusion of multiple data sources for input to an automated task. Please see SHARMA et al. (US 20190050692 A1), Paragraph [0032, 0040, and 0056] and SOLTANI et al. (US 20210097783 A1), Abstract and Paragraph [0028].
Regarding claim 3, SHARMA in view of SOLTANI explicitly teaches the information processing device according to claim 2, SHARMA further teaches wherein the recognition processing control unit (Fig. 6, #602 called a processor. Paragraph [0067]) is configured to restrict processing by a low contribution ratio sensor which is the sensor corresponding to the low contribution ratio sensing data (Fig. 2. Paragraph [0033]-SHARMA discloses using a continuous learning system, the context-based digital signal processing may adapt to when sensors are not working properly and modify weights. Sensors that are overdue on maintenance, out of specification, uncalibrated, or otherwise malfunctioning, may have their corresponding detection algorithm weighted less, indicating less confidence in the result. Please also read paragraph [0037-0040]).
Regarding claim 4, SHARMA in view of SOLTANI explicitly teaches the information processing device according to claim 3, SHARMA further teaches wherein the recognition processing control unit (Fig. 6, #602 called a processor. Paragraph [0067]) stops sensing by the low contribution ratio sensor (Fig. 2. Paragraph [0033]-SHARMA discloses using a continuous learning system, the context-based digital signal processing may adapt to when sensors are not working properly and modify weights. Sensors that are overdue on maintenance, out of specification, uncalibrated, or otherwise malfunctioning, may have their corresponding detection algorithm weighted less, indicating less confidence in the result. In paragraph [0037]-SHARMA discloses different sensors may be weighted differently based on the operating context of the vehicle. A lower weight, one that is not marked with an “H” in the chart, may have a value of 0. Further in paragraph [0038]-SHARMA discloses the combined output is a linear combination of the output of the different object detection modules (e.g., object detection algorithms 210, 212, 214, and 216 of FIG. 2, which may be referred to as sensor detection output S.sub.1, S.sub.2, S.sub.3, etc.) as shown in FIG. 2, as per their respective weights (W.sub.1, W.sub.2, W.sub.3, etc.). The combined output may be derived from the following equation: Combined Output=S.sub.1×W.sub.1+S.sub.2×W.sub.2+S.sub.3×W.sub.3+ . . . +S.sub.n×W.sub.n. Please also read paragraph [0039-0040]).
Claims 5-9 are rejected under 35 U.S.C. 103 as being unpatentable over SHARMA et al. (US 20190050692 A1), hereinafter referenced as SHARMA in view of SOLTANI et al. (US 20210097783 A1), hereinafter referenced as SOLTANI and in further view of BURES et al. (US 20200322703 A1), hereinafter referenced as BURES.
Regarding claim 5, SHARMA in view of SOLTANI explicitly teaches the information processing device according to claim 3, SHARMA in view of SOLTANI fails to explicitly teach wherein the recognition processing control unit lowers at least one of a frame rate and resolution of the low contribution ratio sensor.
However, BURES explicitly teaches wherein the recognition processing control unit (Fig. 1, #140 called a monitoring data analysis system. Paragraph [0016]-BURES disclose FIG. 1 illustrates a monitoring system 100. The monitoring system 100 can include a plurality of multi-sensor units 120, at least one gateway device 130, a monitoring data analysis system 140 (wherein monitoring system #100 may be installed on a vehicle, and the monitoring data analysis system #140 may be implemented by processing modules, such as processing module #540, which include one or more processors that execute stored instructions for controlling and weighting multi-sensor units). In paragraph [0303]-BURES discloses the monitoring data analysis system 140 to automatically generate weights for different measurement sources. In paragraph [0314]-BURES discloses where monitoring worth values indicates how valuable a particular measurement type is for a particular location, this information can be utilized to weight particular sensor devices on one or more particular multi-sensor units (wherein monitoring worth values can be based on sensor proximity to a feature and/or other sensors, how accurately sensors monitor a feature, relative importance or redundancy for detection, the use or statistical significance of types of sensors for detection, etc.). Please also read paragraph [0308-0316]) lowers at least one of a frame rate and resolution of the low contribution ratio sensor (Fig. 1. Paragraph [0326]-BURES discloses particular measurement sources that are determined to be statistically significant in determining conditions of interest in particular locations can be assigned heavier weights and/or particular measurement sources that are determined to be unimportant can be assigned lower weights and/or turned off. This can be utilized to control which proper subset of sensors of each multi-sensor units are turned on and/or how their respective collection rates and/or quality is allocated, based on their determined significance in detection of conditions of interest in their respective locations. In paragraph [0316]-BURES discloses higher resolution measurements and/or uncompressed measurements can be included in outgoing data packets for sensor devices with higher weights, and lower resolution measurements and/or compressed measurement data that is compressed via a lossy compression function can be included in outgoing data packets for sensor devices with lower weights. The monitoring data analysis system 140 can dictate that sensor devices with lower weights have their collection rate and/or quality be decreased before higher weighted sensor devices when network constraints decrease the transmission rate; and/or be turned off first when network constraints decrease the transmission rate and/or power constraint data allocates less power to sensor devices).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SHARMA in view of SOLTANI of having an information processing device, with the teachings of BURES of having wherein the recognition processing control unit lowers at least one of a frame rate and resolution of the low contribution ratio sensor..
Wherein SHARMA’s device having wherein the recognition processing control unit lowers at least one of a frame rate and resolution of the low contribution ratio sensor.
The motivation behind the modification would have been to obtain a method that improves object tracking and environment monitoring, since both SHARMA and BURES concern multi-modality sensor fusion. Wherein SHARMA’s provides improves object detection and tracking by selectively weighting sensor data, while BURES’s systems and methods improve environmental monitoring technologies and their efficiency by dynamically weighting sensors. Please see SHARMA et al. (US 20190050692 A1), Paragraph [0032, 0040, and 0056] and BURES et al. (US 20200322703 A1), Abstract and Paragraph [0303, 0311-0316 and 0420-0423].
Regarding claim 6, SHARMA in view of SOLTANI explicitly teaches the information processing device according to claim 2, SHARMA in view of SOLTANI fails to explicitly teach wherein the recognition processing control unit lowers resolution of the low contribution ratio sensing data.
However, BURES explicitly teaches wherein the recognition processing control unit (Fig. 1, #140 called a monitoring data analysis system. Paragraph [0016]-BURES disclose FIG. 1 illustrates a monitoring system 100. The monitoring system 100 can include a plurality of multi-sensor units 120, at least one gateway device 130, a monitoring data analysis system 140 (wherein monitoring system #100 may be installed on a vehicle, and the monitoring data analysis system #140 may be implemented by processing modules, such as processing module #540, which include one or more processors that execute stored instructions for controlling and weighting multi-sensor units). In paragraph [0303]-BURES discloses the monitoring data analysis system 140 to automatically generate weights for different measurement sources. In paragraph [0314]-BURES discloses where monitoring worth values indicates how valuable a particular measurement type is for a particular location, this information can be utilized to weight particular sensor devices on one or more particular multi-sensor units (wherein monitoring worth values can be based on sensor proximity to a feature and/or other sensors, how accurately sensors monitor a feature, relative importance or redundancy for detection, the use or statistical significance of types of sensors for detection, etc.). Please also read paragraph [0308-0316]) lowers resolution of the low contribution ratio sensing data (Fig. 1. Paragraph [0326]-BURES discloses particular measurement sources that are determined to be statistically significant in determining conditions of interest in particular locations can be assigned heavier weights and/or particular measurement sources that are determined to be unimportant can be assigned lower weights and/or turned off. This can be utilized to control which proper subset of sensors of each multi-sensor units are turned on and/or how their respective collection rates and/or quality is allocated, based on their determined significance in detection of conditions of interest in their respective locations. In paragraph [0316]-BURES discloses higher resolution measurements and/or uncompressed measurements can be included in outgoing data packets for sensor devices with higher weights, and lower resolution measurements and/or compressed measurement data that is compressed via a lossy compression function can be included in outgoing data packets for sensor devices with lower weights. The monitoring data analysis system 140 can dictate that sensor devices with lower weights have their collection rate and/or quality be decreased before higher weighted sensor devices when network constraints decrease the transmission rate; and/or be turned off first when network constraints decrease the transmission rate and/or power constraint data allocates less power to sensor devices).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SHARMA in view of SOLTANI of having an information processing device, with the teachings of BURES of having wherein the recognition processing control unit lowers resolution of the low contribution ratio sensing data.
Wherein SHARMA’s device having wherein the recognition processing control unit lowers resolution of the low contribution ratio sensing data.
The motivation behind the modification would have been to obtain a method that improves object tracking and environment monitoring, since both SHARMA and BURES concern multi-modality sensor fusion. Wherein SHARMA’s provides improves object detection and tracking by selectively weighting sensor data, while BURES’s systems and methods improve environmental monitoring technologies and their efficiency by dynamically weighting sensors. Please see SHARMA et al. (US 20190050692 A1), Paragraph [0032, 0040, and 0056] and BURES et al. (US 20200322703 A1), Abstract and Paragraph [0303, 0311-0316 and 0420-0423].
Regarding claim 7, SHARMA in view of SOLTANI explicitly teaches the information processing device according to claim 2, SHARMA in view of SOLTANI fails to explicitly teach wherein the recognition processing control unit restricts an area to be subjected to the recognition processing in the low contribution ratio sensing data.
However, BURES explicitly teaches wherein the recognition processing control unit (Fig. 1, #140 called a monitoring data analysis system. Paragraph [0016]-BURES disclose FIG. 1 illustrates a monitoring system 100. The monitoring system 100 can include a plurality of multi-sensor units 120, at least one gateway device 130, a monitoring data analysis system 140 (wherein monitoring system #100 may be installed on a vehicle, and the monitoring data analysis system #140 may be implemented by processing modules, such as processing module #540, which include one or more processors that execute stored instructions for controlling and weighting multi-sensor units). In paragraph [0303]-BURES discloses the monitoring data analysis system 140 to automatically generate weights for different measurement sources. In paragraph [0314]-BURES discloses where monitoring worth values indicates how valuable a particular measurement type is for a particular location, this information can be utilized to weight particular sensor devices on one or more particular multi-sensor units (wherein monitoring worth values can be based on sensor proximity to a feature and/or other sensors, how accurately sensors monitor a feature, relative importance or redundancy for detection, the use or statistical significance of types of sensors for detection, etc.). Please also read paragraph [0308-0316]) restricts an area to be subjected to the recognition processing in the low contribution ratio sensing data (Fig. 1. Paragraph [0311]-BURES discloses performing the multi-sensor unit weighting function can cause a weight of a first multi-sensor unit 120 to be higher than the weight of a second multi-sensor unit 120, in response to the first location having a monitoring worth value that indicates a greater worth or is otherwise more favorable than a monitoring worth value of the second location. Furthermore, the monitoring worth values can indicate monitoring worth of different types of measurements collected by different types of sensors on the same or different multi-sensor unit 120. Please also read paragraph [0306-0310 and 0312-0316]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SHARMA in view of SOLTANI of having an information processing device, with the teachings of BURES of having wherein the recognition processing control unit restricts an area to be subjected to the recognition processing in the low contribution ratio sensing data.
Wherein SHARMA’s device having wherein the recognition processing control unit restricts an area to be subjected to the recognition processing in the low contribution ratio sensing data.
The motivation behind the modification would have been to obtain a method that improves object tracking and environment monitoring, since both SHARMA and BURES concern multi-modality sensor fusion. Wherein SHARMA’s provides improves object detection and tracking by selectively weighting sensor data, while BURES’s systems and methods improve environmental monitoring technologies and their efficiency by dynamically weighting sensors. Please see SHARMA et al. (US 20190050692 A1), Paragraph [0032, 0040, and 0056] and BURES et al. (US 20200322703 A1), Abstract and Paragraph [0303, 0311-0316 and 0420-0423].
Regarding claim 8, SHARMA in view of SOLTANI explicitly teaches the information processing device according to claim 2, SHARMA fails to explicitly teach wherein the object recognition unit performs the recognition processing using an object recognition model using a convolutional neural network, and the recognition processing control unit stops convolution operation corresponding to the low contribution ratio sensing data.
SHARMA fails to explicitly teach wherein the object recognition unit performs the recognition processing using an object recognition model using a convolutional neural network, and the recognition processing control unit stops convolution operation corresponding to the low contribution ratio sensing data.
However, BURES explicitly teaches wherein the object recognition unit (Fig. 1, #140 called a monitoring data analysis system. Paragraph [0016]-BURES disclose FIG. 1 illustrates a monitoring system 100. The monitoring system 100 can include a plurality of multi-sensor units 120, at least one gateway device 130, a monitoring data analysis system 140 (wherein monitoring system #100 may be installed on a vehicle, and the monitoring data analysis system #140 may be implemented by processing modules, such as processing module #540, which include one or more processors that execute stored instructions for controlling and weighting multi-sensor units). In paragraph [0303]-BURES discloses the monitoring data analysis system 140 to automatically generate weights for different measurement sources. In paragraph [0314]-BURES discloses where monitoring worth values indicates how valuable a particular measurement type is for a particular location, this information can be utilized to weight particular sensor devices on one or more particular multi-sensor units (wherein monitoring worth values can be based on sensor proximity to a feature and/or other sensors, how accurately sensors monitor a feature, relative importance or redundancy for detection, the use or statistical significance of types of sensors for detection, etc.). Please also read paragraph [0308-0316]) performs the recognition processing using an object recognition model using a convolutional neural network (Fig. 1. Paragraph [0294]-BURES discloses inference functions can be generated by the monitoring data analysis system 140 by training a model. The model can correspond to a convolutional neural network), and the recognition processing control unit stops convolution operation corresponding to the low contribution ratio sensing data (Fig. 1. Paragraph [0024]-BURES discloses FIG. 2 illustrates a multi-sensor unit 120. A multi-sensor unit 120 can include a memory module 210, processing module 240, and set of sensor devices 1-W (wherein sensor devices 1-W can include one or more light/optical/imaging sensors (e.g. cameras), acoustic sensors, electrical sensors, accelerometers, temperature sensors, humidity sensors, pressure sensors, etc.). In paragraph [0311]-BURES discloses performing the multi-sensor unit weighting function can cause a weight of a first multi-sensor unit 120 to be higher than the weight of a second multi-sensor unit 120, in response to the first location having a monitoring worth value that indicates a greater worth or is otherwise more favorable than a monitoring worth value of the second location. In paragraph [0316]-BURES discloses the monitoring data analysis system 140 can dictate that sensor devices with higher weights be turned on and that sensor devices with lower weights be turned off (wherein power to lower weight sensors may be constrained or stopped). Please also read paragraph [0057-0062, 0141-0149, 0159-0163]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SHARMA in view of SOLTANI of having an information processing device, with the teachings of BURES of having wherein the object recognition unit performs the recognition processing using an object recognition model using a convolutional neural network, and the recognition processing control unit stops convolution operation corresponding to the low contribution ratio sensing data.
Wherein SHARMA’s device having wherein the object recognition unit performs the recognition processing using an object recognition model using a convolutional neural network, and the recognition processing control unit stops convolution operation corresponding to the low contribution ratio sensing data.
The motivation behind the modification would have been to obtain a method that improves object tracking and environment monitoring, since both SHARMA and BURES concern multi-modality sensor fusion. Wherein SHARMA’s provides improves object detection and tracking by selectively weighting sensor data, while BURES’s systems and methods improve environmental monitoring technologies and their efficiency by dynamically weighting sensors. Please see SHARMA et al. (US 20190050692 A1), Paragraph [0032, 0040, and 0056] and BURES et al. (US 20200322703 A1), Abstract and Paragraph [0303, 0311-0316 and 0420-0423].
Regarding claim 9, SHARMA in view of SOLTANI explicitly teaches the information processing device according to claim 2, SHARMA in view of SOLTANI fails to explicitly teach wherein the recognition processing control unit lifts restriction on use of the low contribution ratio sensing data for the recognition processing at prescribed time intervals.
However, BURES explicitly teaches wherein the recognition processing control unit (Fig. 1, #140 called a monitoring data analysis system. Paragraph [0016]-BURES disclose FIG. 1 illustrates a monitoring system 100. The monitoring system 100 can include a plurality of multi-sensor units 120, at least one gateway device 130, a monitoring data analysis system 140 (wherein monitoring system #100 may be installed on a vehicle, and the monitoring data analysis system #140 may be implemented by processing modules, such as processing module #540, which include one or more processors that execute stored instructions for controlling and weighting multi-sensor units). In paragraph [0303]-BURES discloses the monitoring data analysis system 140 to automatically generate weights for different measurement sources. In paragraph [0314]-BURES discloses where monitoring worth values indicates how valuable a particular measurement type is for a particular location, this information can be utilized to weight particular sensor devices on one or more particular multi-sensor units (wherein monitoring worth values can be based on sensor proximity to a feature and/or other sensors, how accurately sensors monitor a feature, relative importance or redundancy for detection, the use or statistical significance of types of sensors for detection, etc.). Please also read paragraph [0308-0316]) lifts restriction on use of the low contribution ratio sensing data for the recognition processing at prescribed time intervals (Fig. 1. Paragraph [0316]-BURES discloses the monitoring data analysis system 140 can dictate that sensor devices with higher weights be allocated greater amounts of outgoing data packets sent by the multi-sensor units 120 than sensor devices with lower weights. A greater fraction of measurements captured by the sensor devices with higher weights within the timeframe since the most recent set of outgoing data packets were transmitted can be included in the next set of outgoing data packets, and a smaller fraction of measurements captured by the sensor devices with lower weights within the timeframe can be included in the next set of outgoing data packets. The monitoring data analysis system 140 can dictate that sensor devices with lower weights have their respective packets skipped before skipping higher weighted sensor devices when network constraints decrease the transmission rate; have their collection rate and/or quality be decreased before higher weighted sensor devices when network constraints decrease the transmission rate; and/or be turned off first when network constraints decrease the transmission rate and/or power constraint data allocates less power to sensor devices).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SHARMA in view of SOLTANI of having an information processing device, with the teachings of BURES of having wherein the recognition processing control unit lifts restriction on use of the low contribution ratio sensing data for the recognition processing at prescribed time intervals.
Wherein SHARMA’s device having wherein the recognition processing control unit lifts restriction on use of the low contribution ratio sensing data for the recognition processing at prescribed time intervals.
The motivation behind the modification would have been to obtain a method that improves object tracking and environment monitoring, since both SHARMA and BURES concern multi-modality sensor fusion. Wherein SHARMA’s provides improves object detection and tracking by selectively weighting sensor data, while BURES’s systems and methods improve environmental monitoring technologies and their efficiency by dynamically weighting sensors. Please see SHARMA et al. (US 20190050692 A1), Paragraph [0032, 0040, and 0056] and BURES et al. (US 20200322703 A1), Abstract and Paragraph [0303, 0311-0316 and 0420-0423].
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant`s disclosure.
XIANG et al. (US 20220032955 A1)- An automatic drive device includes a fusion unit recognizing travel environment of a vehicle based on an output signal of a surrounding monitoring sensor and a control planning unit generating a control plan based on a recognition result of the fusion unit. A diagnostic device includes an abnormality detection unit that detects an abnormality of a field recognition system from the surrounding monitoring sensor to the fusion unit by monitoring the output signal of the surrounding monitoring sensor and the recognition result of the fusion unit over time. The diagnostic device requests the control planning unit to perform MRM based on a detection of the abnormality of the recognition system by the abnormality detection unit.......................... Please see Fig. 4-6 and read para. [0071, 0085-0093 and 0111] (wherein Fig. 4-6 illustrate a diagnostic result and degree/ratio of usage for four vehicle sensors). Abstract.
AHUJA et al. (US 20200326667 A1)- Techniques are disclosed for using neural network architectures to estimate predictive uncertainty measures, which quantify how much trust should be placed in the deep neural network (DNN) results. The techniques include measuring reliable uncertainty scores for a neural network, which are widely used in perception and decision-making tasks in automated driving. The uncertainty measurements are made with respect to both model uncertainty and data uncertainty, and may implement Bayesian neural networks or other types of neural networks.…....................... Please see Fig. 1-2 and 4, and para. [0041-0043]. Abstract.
CONNOR et al. (US 20190023413 A1)- Vehicle navigation methods, systems and computer program products are provided that create a composite video image of a scene from video images of the scene generated by a plurality of video image sensors. The video image from each image sensor includes a respective array of pixels. The composite video image is created by selecting for a composite image pixel at a given position in an array of composite image pixels a pixel at the given position from one of the respective pixel arrays having the highest signal level or highest signal-to-noise ratio. The selecting is repeatedly performed for a plurality of given positions in the array of composite image pixels. The composite video image can be displayed via a display, such as a head-up display utilized in a vehicle........................ Please see Fig. 3 and para. [0067] (wherein the weighted contribution of sensors may be dynamically adjusted). Abstract.
Nehmadi et al. (US 20220398851 A1)- A process for sensing a scene. The process includes receiving sensor data from a plurality of sensor modalities, where each sensor modality observes at least a portion of the scene containing at least one of the objects of interest and generates sensor data conveying information on the scene and of the object of interest. The process further includes processing the sensor data from each sensor modality to detect objects of interest and produce a plurality of primary detection results, each detection result being associated with a respective sensor modality. The process also includes fusing sensor data from a first sensor modality with sensor data from a second sensor modality to generate a fused 3D map of the scene, processing the fused 3D map to detect objects of interest and produce secondary detection results and performing object level fusion on the primary and the secondary detection results.......................... Please see Fig. 1A and para. [0056-0063 and 0140]. Abstract.
JOHNANDER et al. (US 20260116398 A1)- A method for generating prediction output for an Automated Driving System (ADS) of a vehicle is disclosed. The method includes obtaining, by one or more processors, a first sensor dataset from a first sensor and a second sensor dataset from a different sensor, each including information about a portion of a surrounding environment of the vehicle. The method further includes processing the first sensor dataset using a first ensemble of two or more encoder networks, each trained to output a first set of encoded features, and processing the second dataset using a second ensemble of two or more encoder networks, each trained to output a second set of encoded features. Then, one or more sets of the encoded features from the first and second datasets are fused using a fusion algorithm to output fused encoded features. A decoder network then generates a prediction output based on the fused encoded features.......................... Please see Fig. 1-3, and para. [0061-0062 and 0077-0078]. Abstract.
BACCHUS et al. (US 20190106085 A1)- Methods and apparatus are provided for cleaning a sensor lens cover for an optical vehicle sensor. The method includes monitoring the sensor lens cover for a contaminant obstructing at least a portion of the sensor lens cover and determining the presence of the commandant and a contaminant type using information provided by one or more vehicle sensors. A cleaning modality selected based the contaminant type is activated and it is determined whether the cleaning modality has removed the contaminant from the sensor lens cover......................... Please see Fig. 2 and para. [0040]. Abstract.
SANO et al. (US 20170345182 A1)- An information processing apparatus according to one embodiment includes a processing circuit. The processing circuit calculates a first presence probability of an object present around a moving body with positional information measured by each of a plurality of sensors having different characteristics, acquires non-measurement information indicating that the positional information on the object has not been obtained for each of the sensors, and determines a second presence probability of the object based on the first presence probability and the non-measurement information.......................... Please see para. [0085-0091]. Abstract.
Tokizaki et al. (US 10880498 B2)- An image processing apparatus includes a first combination unit that receives a far-infrared image and multiple reference images obtained by capturing the same object as that of the far-infrared image and generates a first composite signal which is a composite signal of the multiple reference images and a second combination unit that combines the far-infrared image and the first composite signal to generate a quality-improved image of the far-infrared image. The first combination unit generates the first composite signal on the basis of a visible image-far-infrared image correlation amount and a near-infrared image-far-infrared image correlation amount. The first combination unit sets a contribution ratio of a reference image, which has a larger amount of correlation with the far-infrared image, of the two reference images, that is, the visible image and the near-infrared image to a large value and generates the first composite signal..…....................... Please see Fig. 4-5 and 9. Abstract.
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to Aaron Bonansinga whose telephone number is (703) 756-5380 The examiner can normally be reached on Monday-Friday, 9:00 a.m. - 6:00 p.m. ET.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s
supervisor, Chineyere Wills-Burns can be reached by phone at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is (571) 273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AARON TIMOTHY BONANSINGA/Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673
Prosecution Insights