DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 6/12/2026 has been entered.
Response to Amendment
The following addresses applicant’s remarks/amendments dated 12 June 2026.
Claims 1 was amended. Claims 12-13 and 20-21 were cancelled. New claims 24 was added. Therefore, claims 1-2, 4-5, 7-8, 10-11, 14-19, and 22-24 are currently pending in the current application and are addressed below.
Response to Arguments
Applicant’s arguments, see pages 7-8 of the Remarks, filed 12 June 2026, with respect to the rejection(s) of claim(s) 1 under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Bamji et al., US 20110285910 A1 in view of Kempf et al., US 20200209398 A1.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 4-5, 7-8, 10, 14-19, and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Bamji et al., US 20110285910 A1 (“Bamji”) in view of Kempf et al., US 20200209398 A1 (“Kempf”).
Regarding claim 1, Bamji discloses an information processing system to be applied for an image sensor having a plurality of first pixels with sensitivity for visible light (Fig. 5, RGB array 240', Paragraph [0048]) and a plurality of second pixels with sensitivity for infrared light (Fig. 5, array 130 of Z pixels 140, Paragraph [0048]), the information processing system comprising:
a processor configured to perform operations comprising (Fig. 5, processor 160, Paragraph [0048]):
acquiring first brightness information relating to pixel values of the plurality of first pixels from the plurality of first pixels (Fig. 6A, step 400, RBG image, Paragraph [0055]), wherein the first brightness information constitutes a brightness image that is a set of outputs of the plurality of first pixels (Fig. 6A, step 400, RBG image, Paragraph [0055], Fig. 5, RGB array 240', Paragraph [0048]);
acquiring second brightness information relating to pixel values of the plurality of second pixels from the plurality of second pixels (Fig. 6A, step 430, confidence map, Paragraph [0056]), wherein the second brightness information constitutes a brightness image that is set of outputs of the plurality of second pixels (Fig. 6A, step 430, confidence map, Paragraph [0056]; Fig. 5, array 130 of Z pixels 140, Paragraph [0048]);
acquiring distance information from at least one second pixel of the plurality of second pixels, the distance information relating to a distance between the image sensor and an object by which the infrared light is reflected (Fig. 6A, step 400, Z depth data, Paragraph [0055]; Fig. 5, array 130 of Z pixels 140, Paragraph [0048]);
detecting, as a two-dimensional detection result for the object, the object based on reference brightness information selected from the group consisting of the first brightness information and the second brightness information (Fig. 6A, step 460, edge map of foreground object, Paragraph [0063]), […];
detecting, as a three-dimensional detection result for the object, the object based on the distance information (Fig. 6B, step 470, refined depth image, Paragraph [0065]); and
composing:
the two-dimensional detection result for the object (Fig. 6B, step 490, final image, Paragraph [0065]); and
the three-dimensional detection result for the object (Fig. 6B, step 470, refined depth image, Paragraph [0065]).
Bamji does not teach: the two-dimensional detection result indicating a type of the object.
However, Kempf teaches a 2D processing system that includes an object identification system that identifies objects in an image and classifies the type of object with a label (Fig. 2, 2D image processing system 200, object identification system 202, classification 204, Paragraph [0027]).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bamji’s signal processing method by adding a 2D processor which identifies and classifies objects in a 2D image, which is disclosed by Kempf. One of ordinary skill in the art would have been motivated to make this modification in order to obtain high resolution depth data for the objects while maintaining lower depth resolution for the background scene, as suggested by Kempf (Paragraph [0026]).
Regarding claim 2, Bamji, as modified in view of Kempf, discloses the information processing system of claim 1, wherein the first brightness information includes light and darkness information representing intensity of light input to the first pixel (Bamji, Fig. 6A, step 400, RBG image, Paragraph [0055]).
Regarding claim 4, Bamji, as modified in view of Kempf, discloses the information processing system of claim 1, wherein the detecting the object based on the distance information comprises detecting the object based on not only the distance information but also one or more pieces of information selected from the group consisting of the first brightness information and the second brightness information (Bamji, Fig. 6A, step 460, edge map based on RGB and depth image, Paragraph [0063]).
Regarding claim 5, Bamji, as modified in view of Kempf, discloses the information processing system of claim 1, wherein the detecting the object based on the distance information comprises detecting the object based on not only the distance information but also the first brightness information corrected so as to match a timing of the second brightness information (Bamji, Fig. 6A, step 460, edge map based on RGB and depth image, Paragraph [0063]).
Regarding claim 7, Bamji, as modified in view of Kempf, discloses the information processing system of claim 1, wherein the distance information includes information obtained by a Time-of-Flight method (Bamji, Fig. 5, array 130 of Z pixels 140, Paragraph [0048]; See also Paragraph [0076]).
Regarding claim 8, Bamji, as modified in view of Kempf, discloses the information processing system of claim 1, wherein the operations further comprise correcting the distance information based on the distance information and one or more pieces of information selected from the group consisting of the first brightness information and the second brightness information (Bamji, Fig. 5, processor 160, Fig. 6A, depth map, confidence map, Step 430, Paragraph [0008], Paragraph [0056]).
Regarding claim 10, Bamji, as modified in view of Kempf, discloses the information processing system of claim 1,wherein the composing comprises composing the two dimensional detection result and the three dimensional detection result by making a correction of the three dimensional detection result based on the two dimensional detection result (Bamji, Fig. 6B, step 470, refined depth image, Paragraph [0065]).
Regarding claim 14, Bamji, as modified in view of Kempf, discloses the information processing system of claim 1, wherein the operations further comprise separating the object from a peripheral area located around the object (Bamji, Fig. 5, processor 160, Fig. 6A-B, alpha-matting image, step 480, Paragraph [0008], Paragraph [0052], Paragraphs [0071]-[0072]).
Regarding claim 15, Bamji, as modified in view of Kempf, discloses the information processing system of claim 14, wherein the detecting the object based on the distance information comprises detecting the object based on information in which the peripheral area is removed from the distance information (Bamji, Fig. 6B, alpha-matting image, step 480, Paragraph [0008], Paragraph [0052], Paragraphs [0071]-[0072]).
Regarding claim 16, Bamji, as modified in view of Kempf, discloses the information processing system of claim 1, wherein the operations further comprise correcting a time difference between the first brightness information and the second brightness information (Bamji, Fig. 6A, steps 410-430, Paragraph [0055]-[0056]).
Regarding claim 17, Bamji, as modified in view of Kempf, discloses the information processing system of claim 1,
wherein the operations further comprise outputting an information processing result, obtained based on the first brightness information, the second brightness information and the distance information (Bamji, Fig. 6B, step 490, final image, Paragraph [0065]), and
the information processing result relates to a state of a monitoring area within an angle of view of the image sensor (Bamji, Fig. 5, target object 20, Paragraph [0008]).
Regarding claim 18, Bamji, as modified in view of Kempf, discloses the information processing system of claim 17, wherein the information processing result includes one or more pieces of information selected from the group consisting of:
information about whether or not the object is present in the monitoring area (Bamji, step 460, Paragraph [0063]);
information about a position in the monitoring area, of the object present in the monitoring area (Bamji, step 440 460, Paragraph [0057], [0063]); and
information about an attribute of the object (Kempf, Fig. 2, 2D image processing system 200, object identification system 202, classification 204, Paragraph [0027]).
Regarding claim 19, Bamji, as modified in view of Kempf, discloses a sensor system, comprising the information processing system of claim 1 and the image sensor (Bamji, Fig. 5, RGB array 240', Paragraph [0048]).
Regarding claim 22, Bamji, as modified in view of Kempf, discloses the information processing system of claim 1, wherein the composing comprises correcting the two-dimensional detection result based on the three-dimensional detection result (Bamji, Fig. 6B, step 480-490, final image, Paragraph [0071]-[0073]).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Bamji, as modified in view of Kempf, in further view of Oder et at., US 20180067966 A1 ("Oder").
Regarding claim 11, Bamji, as modified in view of Kempf, discloses the information processing system of claim 1.
Bamji, as modified in view of Kempf, does not teach: wherein the operations further comprise outputting a feedback signal to a sensor system including the image sensor, and the image sensor is configured to output an electrical signal in which one or more parameters selected from the group consisting of an exposure time and a frame rate are changed in response to the feedback signal.
However, Oder teaches a sensor fusion system that receives raw measurement data from multiple sensor systems, including a LIDAR device and an image capture device. The sensor fusion system can generate feedback signals to provide to the sensor system. The feedback signals can reposition sensors, expand the field of view of the sensors, change the exposure time, or alter a mode of operation. (Fig. 1, sensor fusion system 300, feedback signals 116, Paragraph [0026]).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have combined the imaging system disclosed by Bamji, as modified in view of Kempf, with the functionality to generate feedback signals to change the sensor’s operation through a sensor fusion system, which is disclosed by Oder. One of ordinary skill in the art could have combined these elements and yielded the predictable result of updating the sensor exposure time.
Claim 23 is rejected under 35 U.S.C. 103 as being unpatentable over Bamji, as modified in view of Kempf, in further view of Banerjee et al., EP 3438777 A1 (“Banerjee”).
Regarding claim 23, Bamji, as modified in view of Kempf, discloses the information processing system of claim 1.
Bamji, as modified in view of Kempf, does not teach: wherein the two-dimensional detection result includes a marker indicating a position of the object, and the composing comprises adjusting a position of the marker based on the three-dimensional detection result.
However, Banerjee teaches a method for combining camera sensor data and LIDAR sensor data based on information related to one or more edges. The determining of a combined image may comprise of reducing the mismatch of an overlay of the edges over the camera sensor data and point cloud data. A calibration may be performed where the camera sensor data or the LIDAR sensor data is transformed to a common coordinate system (Fig. 1a, determining a combined image 160, Paragraph [0034]-[0035]).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method identifying objects disclosed by Bamji, as modified in view of Kempf, by adjusting the coordinates of the object, which is disclosed by Banerjee. One of ordinary skill in the art would have been motivated to make this modification in order to “enable a more precise determination of a location of these objects”, as suggested by Banerjee (Paragraph [0003]).
Claim 24 is rejected under 35 U.S.C. 103 as being unpatentable over Bamji, as modified in view of Kempf, in further view of Xu et al., US 20190096086 A1 (“Xu”).
Regarding claim 24, Bamji, as modified in view of Kempf, discloses The information processing system of claim 1.
Bamji, as modified in view of Kempf, does not teach: wherein the detecting the object based on the reference brightness information comprises detecting the object based on a Convolutional Neural Network (CNN).
However, Xu teaches an image being captured by a camera and each pixel on the image being represented by a two-dimensional coordinate. Objects in the image may be identified with various convolutional neural network algorithms including Fast-CNN and Faster-F CNN (Fig. 1, image 110, bounding box 114, Paragraph [0019]).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of identifying objects disclosed by Bamji, as modified in view of Kempf, by using a CNN algorithm, which is disclosed by Xu. One of ordinary skill in the art would have been motivated to make this modification in order to only identify certain object classes, as suggested by Xu (Paragraph [0019]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RACHEL N NGUYEN whose telephone number is (571)270-5405. The examiner can normally be reached Monday - Friday 8 am - 5:30 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Yuqing Xiao can be reached at (571) 270-3603. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RACHEL NGUYEN/Examiner, Art Unit 3645
/YUQING XIAO/Supervisory Patent Examiner, Art Unit 3645
Prosecution Insights