DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 11 recites the limitation "the RADAR data" in line 7. There is insufficient antecedent basis for this limitation in the claim.
Claim 20 recites the limitation “the RADAR data” in line 9. There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1 is/are rejected under 35 U.S.C. 103 as being unpatentable over Posselius (U.S. PG-PUB NO. 2017/0064278) in view of Lee (WO 2020171281 A1).
-Regarding claim 1, Posselius discloses an infrared (IR) assisted object segmentation and range perception system for a vehicle (FIG. 13, 14), the IR assisted object segmentation and range perception system comprising: a camera system configured to capture image data of an environment external to the vehicle (LWIR camera 18, [0032]); and a control system configured to: generate an environmental model for the environment external to the vehicle by: (i) performing object segmentation of one or more objects in the captured image data (segmentation and dense computation 38, [0036]), and (ii) based on the object segmentation, perform range perception of the one or more objects (segmentation/interpolation algorithm is used to fill in depth regions between sparse points, [0035]); and utilize the generated environmental model during operation of the vehicle (vehicle 10, [0031]).
Posselius is silent to teaching that the image data including unfiltered IR data; and based on the unfiltered IR data. However, the claimed limitation is well known in the art as evidenced by Lee.
In the same field of endeavor, Lee teaches the image data including unfiltered IR data (infrared image data is raw data, page 5 of translation); and based on the unfiltered IR data (classify an object using RGB data of the visible light image and infrared image data acquired through an infrared camera as an input value of a pre-learned object detection network, page 3 of translation).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Posselius with the teaching of Lee in order to improve night/low-visibility perception.
Claim(s) 2 and 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Posselius (U.S. PG-PUB NO. 2017/0064278) in view of Lee (WO 2020171281 A1) and further in view of Hu (U.S. PG-PUB NO. 2026/0203854).
-Regarding claim 2, the combination is silent to teaching that the camera system is further configured to apply a color filter array (CFA) to at least a portion of the image data to obtain filtered color data that is part of the captured image data. However, the claimed limitation is well known in the art as evidenced by Hu.
In the same field of endeavor, Hu teaches the camera system is further configured to apply a color filter array (CFA) to at least a portion of the image data to obtain filtered color data that is part of the captured image data (one or more color filter arrays (CFAs), [0062]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of the combination with the teaching of Hu in order to improve driving assistance or for autonomous driving.
-Regarding claim 3, the combination further discloses the control system is configured to perform the object segmentation of the one or more objects based further on a combination of the unfiltered IR data and the filtered color data (Lee, derives a feature map as described above by using information on visible light image data and infrared image data as input values to an object detection network, page 7 of translation).
Claim(s) 7-9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Posselius (U.S. PG-PUB NO. 2017/0064278) in view of Lee (WO 2020171281 A1), Hu (U.S. PG-PUB NO. 2026/0203854) and further in view of Lu (U.S. PG-PUB NO. 2015/0092059).
-Regarding claim 7, the combination is silent to teaching that the camera system is a color type camera system that does not include an IR filter. However, the claimed limitation is well known in the art as evidenced by Lu.
In the same field of endeavor, Lu teaches the camera system is a color type camera system that does not include an IR filter (with the IR cut-off in the sensor pixel level, there is no need to add IR cut-off filter in a lens, [0023]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of the combination with the teaching of Lu in order to avoid a separate physical IR filter.
-Regarding claim 8, the combination further discloses the camera system is a color-IR type camera system where the CFA and a light sensor are associated with only a portion of a plurality of pixels and at least some of the plurality of pixels are associated with an IR sensor (Lu, The arrangement of pixels has a pixel pattern that is derived from the Bayer pattern, but has one of the green filters or pixels replaced by an infrared (IR) filter or pixel 12d, such as shown in FIGS. 3A and 3B, [0021]).
-Regarding claim 9, the combination further discloses the CFA is a red/green/blue (RGB) type CFA having red, green, and blue pixels associated with every three pixels of the plurality of pixels, and wherein the IR sensor is a passive IR sensor and every fourth pixel of the plurality of pixels are associated with the IR sensor and not the CFA (Lu, the R-G-B-IR filter arrangement may be any suitable arrangement of pixels/filters in a repeating pattern, such as the 2X2 arrangements of pixels shown in FIGS. 3A and 3B. As shown in FIG. 3A, a 2X2 pixel matrix or sub-array may have the IR pixel 12d positioned diagonal to the red pixel 12a, [0022]).
Claim(s) 10, 11 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Posselius (U.S. PG-PUB NO. 2017/0064278) in view of Lee (WO 2020171281 A1) and further in view of Burina (U.S. PG-PUB NO. 2025/0076486).
-Regarding claim 10, the combination is silent to teaching that a radio detection and ranging (RADAR) system of the vehicle, wherein the RADAR system is configured to capture RADAR data of the environment external to the vehicle, and wherein the control system configured to generate the environmental model further by (iii) fusing the range perception of the one or more objects based on the object segmentation with range perception based on the RADAR data. However, the claimed limitation is well known in the art as evidenced by Burina.
In the same field of endeavor, Burina teaches a radio detection and ranging (RADAR) system of the vehicle, wherein the RADAR system is configured to capture RADAR data of the environment external to the vehicle, and wherein the control system configured to generate the environmental model further by (iii) fusing the range perception of the one or more objects based on the object segmentation with range perception based on the RADAR data (The object detection component 118 may be configured to detect objects in the environment based on a combination of the infrared data 112 and the radar data 114, [0034]; the infrared-radar fusion system may execute one or more trained models configured to detect, segment, classify, and/or track objects within the combined sensor data view, [0018]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of the combination with the teaching of Burina in order to provide range redundancy/robustness in low-visibility.
-Regarding claim 11, Posselius discloses an infrared (IR) assisted object segmentation and range perception method for a vehicle (FIG. 13, 14), the IR assisted object segmentation and range perception method comprising: capturing, by a camera system of the vehicle, image data of an environment external to the vehicle (LWIR camera 18, [0032]); receiving, by a control system of the vehicle and from the camera system, and the image data (A controller 24 can receive data from stereo LWIR camera 18, NIR sensor 20 and stereo RGB camera 22 to undertake the processing of image data received therefrom using algorithms of the present invention, [0032]); generating, by the control system, an environmental model for the environment external to the vehicle by: (i) performing object segmentation of one or more objects in the captured image data (segmentation and dense computation 38, [0036]), and (ii) based on the object segmentation, perform range perception of the one or more objects (segmentation/interpolation algorithm is used to fill in depth regions between sparse points, [0035]); and utilizing, by the control system, the generated environmental model during operation of the vehicle (vehicle 10, [0031]).
Posselius is silent to teaching that the image data including unfiltered IR data; and based on the unfiltered IR data. However, the claimed limitation is well known in the art as evidenced by Lee.
In the same field of endeavor, Lee teaches the image data including unfiltered IR data (infrared image data is raw data, page 5 of translation); and based on the unfiltered IR data (classify an object using RGB data of the visible light image and infrared image data acquired through an infrared camera as an input value of a pre-learned object detection network, page 3 of translation).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Posselius with the teaching of Lee in order to improve night/low-visibility perception.
The combination is silent to teaching that the RADAR data. However, the claimed limitation is well known in the art as evidenced by Burina.
In the same field of endeavor, Burina teaches the RADAR data (the radar data 114, [0034]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of the combination with the teaching of Burina in order to provide range redundancy/robustness in low-visibility.
-Regarding claim 20, the combination further discloses capturing, by a radio detection and ranging (RADAR) system of the vehicle, RADAR data of an environment external to the vehicle; and receiving, by the control system and from the RADAR system, the RADAR data, wherein the generating of the environmental model further comprises (iii) fusing the range perception of the one or more objects based on the object segmentation with range perception based on the RADAR data (Burina, The object detection component 118 may be configured to detect objects in the environment based on a combination of the infrared data 112 and the radar data 114, [0034]; the infrared-radar fusion system may execute one or more trained models configured to detect, segment, classify, and/or track objects within the combined sensor data view, [0018]).
Claim(s) 12 and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Posselius (U.S. PG-PUB NO. 2017/0064278) in view of Lee (WO 2020171281 A1), Burina (U.S. PG-PUB NO. 2025/0076486) and further in view of Hu (U.S. PG-PUB NO. 2026/0203854).
-Regarding claim 12, the combination is silent to teaching that applying, by the camera system, a color filter array (CFA) to at least a portion of the image data to obtain filtered color data that is part of the captured image data. However, the claimed limitation is well known in the art as evidenced by Hu.
In the same field of endeavor, Hu teaches applying, by the camera system, a color filter array (CFA) to at least a portion of the image data to obtain filtered color data that is part of the captured image data (one or more color filter arrays (CFAs), [0062]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of the combination with the teaching of Hu in order to improve driving assistance or for autonomous driving.
-Regarding claim 13, the combination further discloses the performing of the object segmentation of the one or more objects is based further on a combination of the unfiltered IR data and the filtered color data (Lee, derives a feature map as described above by using information on visible light image data and infrared image data as input values to an object detection network, page 7 of translation).
Claim(s) 17-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Posselius (U.S. PG-PUB NO. 2017/0064278) in view of Lee (WO 2020171281 A1), Burina (U.S. PG-PUB NO. 2025/0076486), Hu (U.S. PG-PUB NO. 2026/0203854) and further in view of Lu (U.S. PG-PUB NO. 2015/0092059).
-Regarding claim 17, the combination is silent to teaching that the camera system is a color type camera system that does not include an IR filter. However, the claimed limitation is well known in the art as evidenced by Lu.
In the same field of endeavor, Lu teaches the camera system is a color type camera system that does not include an IR filter (with the IR cut-off in the sensor pixel level, there is no need to add IR cut-off filter in a lens, [0023]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of the combination with the teaching of Lu in order to avoid a separate physical IR filter.
-Regarding claim 18, the combination further discloses the camera system is a color-IR type camera system where the CFA and a light sensor are associated with only a portion of a plurality of pixels and at least some of the plurality of pixels are associated with an IR sensor (Lu, The arrangement of pixels has a pixel pattern that is derived from the Bayer pattern, but has one of the green filters or pixels replaced by an infrared (IR) filter or pixel 12d, such as shown in FIGS. 3A and 3B, [0021]).
-Regarding claim 19, the combination further discloses the CFA is a red/green/blue (RGB) type CFA having red, green, and blue pixels associated with every three pixels of the plurality of pixels, and wherein the IR sensor is a passive IR sensor and every fourth pixel of the plurality of pixels are associated with the IR sensor and not the CFA (Lu, the R-G-B-IR filter arrangement may be any suitable arrangement of pixels/filters in a repeating pattern, such as the 2X2 arrangements of pixels shown in FIGS. 3A and 3B. As shown in FIG. 3A, a 2X2 pixel matrix or sub-array may have the IR pixel 12d positioned diagonal to the red pixel 12a, [0022]).
Allowable Subject Matter
Claims 4-6 and 14-16 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PING Y HSIEH whose telephone number is (571)270-3011. The examiner can normally be reached Monday-Friday, 9am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PING Y HSIEH/ Primary Examiner, Art Unit 2664
Prosecution Insights