DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-14 are pending.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 14 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
Claim 14 recites limitation “A program for …”. A computer program per se, is not directed to one of the statutory categories, Gottschalk v. Benson, 409 U.S. at 72, 175 USPQ at 676-77. See MPEP § 2106(I).
Claim Rejections - 35 USC § 103
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claim(s) 1, 5-6, 8 and 11-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Richards (US2017/0302910) in view of Hayasaka et al (US2015/0248744).
Regarding claims 1 and 13-14, Richards teaches an image processing device comprising:
a generation unit that generates a reference image in which information indicating ambiguity of a pixel value of each pixel of a depth image acquired from a sensor that measures a distance is set as a pixel value of each pixel; and
(Richards, Figs. 4 and 6; s630, “a center camera, such as the first camera 410, can be selected as a reference camera. A first depth map, such as a horizontal parallax depth map, can be generated from a first image from the first camera 410 and a second image from the second camera 420 based on the image coordinates of the first camera 410”, [0020]; “depth Z can be calculated per pixel of an image”, [0017]; the horizontal depth map may be considered as a reference depth map; “At 660, a confidence score map can be generated for each depth map”, [0029], including the confidence score map for the horizontal depth map (the “reference depth map”); a confidence score map of the measured depths represents the degree of uncertainty or ambiguity of the measured depth)
an integration unit that integrates a plurality of the depth images on a basis of the reference image corresponding to each of the plurality of the depth images.
(Richards, Fig. 5, “The cameras 510, 520, and 530 of the device 500 can operate similarly to the cameras 410, 420, and 430, respectively, of the device 400 for the generation of the first two depth maps”, [0025]; similar to Fig. 4, [0019-0023], the images from cameras 510 and 520 may be used for generating the horizontal depth map; the images from cameras 510 and 530 may be used for generating the vertical depth map; the images from cameras 510 and 540 may be used for generating the diagonal depth map; s680 and s690, “At 680, a depth value of a pixel on a fusion depth map can be selected based on the confidence score maps and the depth maps. ... Selecting the depth value can also include averaging the depth values between corresponding pixels on the x-axis depth map and y-axis depth map”, [0031]; obviously, the depth values of pixels on a fusion depth map, in a four-camera system (Fig. 5), may be selected by averaging (integrating) the horizontal depth map, the vertical depth map and the diagonal depth map based on the confidence score maps of these three depth maps; generation of the fusion depth map may also be considered as an averaging processing based on the horizontal depth map coordinates (the “reference depth map” coordinates) since the depth averaging is performed per pixel)
Regarding claim 5, Richards teaches its/their respective base claim(s).
Richards further teaches the image processing device according to claim 1, wherein the integration unit integrates pixels of the plurality of the depth images by using weight corresponding to the information indicating the ambiguity of the pixel value.
(Richards, “Selecting the depth value can also include averaging the depth values between corresponding pixels on the x-axis depth map and y-axis depth map when the confidence score of each corresponding pixel on the confidence scores maps of two depth maps is within a threshold difference from each other. For example, the depth values can be averaged when the difference between confidence scores is within a value of zero, five, or ten on a 0-100 scale or any other threshold difference useful for determining that the confidence scores are the same or close to each other”, [0031]; averaging the depth values of a pixel between two depth maps is only performed if the difference (delta) between the confidence scores corresponding to the depth values are within a predetermined threshold)
Regarding claim 6, Richards teaches its/their respective base claim(s).
Richards further teaches the image processing device according to claim 1, wherein the integration unit further integrates a plurality of the reference images.
(Richards, see comments on claim 1; for a multi-camera system, one or more depth maps from one or more stereo image pairs may be considered as reference pairs which may be averaged for generating a fusion depth map)
Regarding claim 8, Richards teaches its/their respective base claim(s).
Richards further teaches the image processing device according to claim 1, wherein the reference image is an image having same resolution as resolution of the depth image.
(Richards, Figs. 4 and 5; image resolution of the multiple cameras may be same, and resulting depth maps may have same resolution, “depth Z can be calculated per pixel of an image”, [0018])
Regarding claim 12, Richards teaches its/their respective base claim(s).
Richards further teaches the image processing device according to claim 1, wherein the generation unit estimates information indicating ambiguity of a pixel value of a depth image generated by a stereo camera as the sensor on a basis of two images having parallax generated by the stereo camera.
(Richards, “Selecting the depth value can include determining the depth value of a pixel on the fusion depth map based on the confidence score maps using a decision rule. The decision rule can include selecting the depth value of a pixel on the fusion depth map as the depth value of the pixel with the higher confidence score between the confidence score of the pixel on the confidence score map of the x-axis depth map and the confidence score of the corresponding pixel on the confidence score map of the y-axis depth map”, [0031]; stereo cameras can have uneven depth confidences, meaning the stereo cameras can generate uncertain depths)
Claim(s) 2-4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Richards (US2017/0302910) in view of Hayasaka et al (US2015/0248744).
Regarding claim 2, Richards teaches its/their respective base claim(s).
Richards does not expressly disclose but Hayasaka teaches the image processing device according to claim 1, further comprising
an alignment unit that performs alignment that is processing of matching a viewpoint of the depth image and a viewpoint of the reference image with a reference viewpoint,
(Hayasaka, Fig. 8; after horizontally aligning/matching C1 and V1, C1 may be considered as a viewpoint of the horizontal stereo image pair G22 and G23 (i.e., a viewpoint of the horizontal depth map) or a reference viewpoint of the reference depth map, [0083, 0084]; viewpoint V1 of the vertical stereo image pair G22 and G12 (i.e., a viewpoint of the horizontal depth map) should be aligned/matched with the reference viewpoint C1 for determining the depth of viewpoint V1)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the teachings of Hayasaka into the system or method of Richards in order to accurately determine a depth map from a pair of stereo images by aligning/matching the viewpoints of the stereo image pair. The combination of Richards and Hayasaka also teaches other enhanced capabilities.
The combination of Richards and Hayasaka further teaches:
wherein the integration unit integrates the plurality of the depth images obtained by the alignment.
(Richards, see comments on claim 1; it’s well known that a correct depth map can be generated only after the viewpoints of a stereo image pair are aligned/matched; the generated depth maps such as horizontal depth map, vertical depth map and diagonal depth map may be fused together using averaging)
Regarding claim 3, the combination of Richards and Hayasaka teaches its/their respective base claim(s).
The combination further teaches the image processing device according to claim 2, wherein the alignment unit matches viewpoints of the plurality of the depth images with a viewpoint of one of the plurality of the depth images.
(Richards, see comments on claim 1; Fig. 5, in a four camera system, since the fused depth map may be obtained by averaging the depth maps of 510/520, 510/530 and 510/540, the viewpoints of these three stereo image pairs (or three depth maps) must be aligned/matched (Hayasaka, see comments on claim 2) in order to obtain accurate depth maps)
Regarding claim 4, the combination of Richards and Hayasaka teaches its/their respective base claim(s).
The combination further teaches the image processing device according to claim 2, wherein the alignment unit matches viewpoints of the plurality of the depth images with a viewpoint of a color image.
(Hayasaka, “the color image”, [0262])
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Richards (US2017/0302910) in view of Rowell et al (US2019/0208181).
Regarding claim 7, Richards teaches its/their respective base claim(s).
Richards does not expressly disclose but Rowell teaches the image processing device according to claim 1, wherein the information indicating the ambiguity of the pixel value is a standard deviation.
(Rowell, “Depth information includes disparity data, depth data (e.g., a depth value for each pixel or a depth map), and a depth confidence metric (e.g., a standard deviation, confidence score, confidence threshold, confidence level, percent error, variance, skewness, kurtosis, or correlation coefficient)”, [0047]
map) should be aligned/matched with the reference viewpoint C1 for determining the depth of viewpoint V1)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the teachings of Rowell into the system or method of Richards in order to provide a quantitative measure of the reliability or confidence in the depth estimation at each pixel or region using the well-known standard deviation. The combination of Richards and Rowell also teaches other enhanced capabilities.
Claim(s) 9-10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Richards (US2017/0302910) in view of Waitz (US2022/0035226).
Regarding claim 9, Richards teaches its/their respective base claim(s).
Richards does not expressly disclose but Waitz teaches the image processing device according to claim 1, further comprising:
a plurality of the sensors that measures a distance by different distance measuring methods.
(Waitz, “The 3D sensor consists of a stereoscopic camera array, TOF camera, laser scanner, lidar sensor, radar sensor or combination of different 3D sensors to improve the measurement quality, range and resolution”, [0053])
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the teachings of Waitz into the system or method of Richards in order to improve measurement quality, range and resolution using different technologies for depth measurement. The combination of Richards and Waitz also teaches other enhanced capabilities.
Regarding claim 10, The combination of Richards and Waitz teaches its/their respective base claim(s).
The combination further teaches the image processing device according to claim 9, wherein the sensor includes a ToF camera, a stereo camera, a LIDAR, and a RADAR.
(Waitz, see comments on claim 9)
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Richards (US2017/0302910) in view of Kim et al (US2021/0149086).
Regarding claim 11, Richards teaches its/their respective base claim(s).
Richards does not expressly disclose but Kim teaches the image processing device according to claim 1, wherein the generation unit estimates information indicating ambiguity of a pixel value of a depth image generated by a ToF camera as the sensor on a basis of an image indicating light reception intensity at a time of distance measurement generated by the ToF camera.
(Kim, “since intensity of the light is relatively weak and a reach distance is short, the ToF camera may be disadvantageous in obtaining the depth information of a subject located at a far distance”, [0098]; obviously, low receive signal of the ToF camera => low SNR => high uncertainty in ToF measurement => high ambiguity in depth measurement)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the teachings of Kim into the system or method of Richards in order to recognize that low receive signal in a ToF camera can cause high depth ambiguity in depth measurement. The combination of Richards and Kim also teaches other enhanced capabilities.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIANXUN YANG whose telephone number is (571)272-9874. The examiner can normally be reached on MON-FRI: 8AM-5PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, Applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JIANXUN YANG/
Primary Examiner, Art Unit 2662 9/4/2025