DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Allowable Subject Matter
Claim 8 is objected to as being dependent upon a rejected base claim but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The prior art teaches RGB and depth cameras along a row, but they are not interleaved and there is only a single RGB camera [see Grunnet-Jepsen, US 2017/0186166, Fig. 7B]. The claim requires a plurality of both types of camera and for them to be interleaved, i.e. to spatially alternate.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 11-15, 38 and 40 are rejected under 35 U.S.C. 103 as being unpatentable over Eldar, US 2017/0270654 in view of Aguilar et al., US 2020/0106960.
Claims 1 and 15 and 40. Eldar teaches a composite camera device for determining a composite image of a scene, comprising:
a memory storing instructions to be executed by a processor [paras. 18-20];
a plurality of visible light cameras, wherein each of the plurality of visible light cameras is configured to determine an image of the scene [multiple cameras, Figs. 1, 3, paras. 22-24, 35, 36];
a plurality of distance sensors, covering a plurality of distance sections of the view of the scene, wherein each of the plurality of distance sensors is configured to determine a distance information to a nearest object from the respective distance sensor for at least one of the plurality of distance sections [each image capture mechanism 118 includes RGB camera and associated (respective) depth sensor, Fig. 1, paras. 22, 23, 33, 34, 37, 42]; and
a processing device, wherein the processing device is configured to:
calibrate, for each of the plurality of distance sections [distance section is simply a generic description of a distance measurement; e.g. a 1 m distance is in a “close” section and 100m is a “far” section; Eldar teaches measurement of varying distances], a camera model of one of the plurality of visible light cameras using the respective determined distance information [cameras are calibrated using depth data; calibration is used to obtain correct model parameters, Fig. 1, 3, 4A, 4B, paras. 23, 24, 35-38, 44-47, 50, 58, 59]; and
determining the composite image of the scene using the calibrated camera models [depth/calibration data are used to capture stereoscopic image pairs, paras. 22, 40, 56; also see paras. 23, 24, 35-38, 44-47, 50, 58, 59].
Eldar is but is silent on covering a 360 degree view of the scene with images of the scene from each of the plurality of visible light cameras. Aguilar teaches a camera system that obtains a 360 degree view of the scene with images of the scene from each of the plurality of visible light cameras [Figs. 28, 36, paras. 54, 166, 167, 241].
Before the effective filing date of the claimed invention, it would have been obvious to one skilled in the art to combine the references in order to create a 360 degree image that is suitable for virtual reality where a user can look in all directions [see Aguilar, para. 52]. This also simply ensures that interesting content in any direction is captured by the camera system.
11. Aguilar teaches the device of claim 1, wherein the plurality of visible light cameras are large field-of-view cameras [e.g. wide-angle or fish-eye, para. 155].
12. Aguilar teaches the device of claim 11, wherein each of the large field-of-view cameras comprise a field-of-view that is at least 160 degrees [a fisheye lens is conventionally understood to cover a FOV of 180 degrees or more; para. 155].
13. Aguilar teaches the device of claim 11, wherein the large field-of- view cameras comprise a camera with a fish-eye lens [para. 155].
14. Eldar teaches the device of claim 1, wherein the processing device comprises a hardware processing device [Fig. 1, para. 18].
38. Eldar teaches the method of claim 15, wherein the method is a computer-implemented method [Fig. 1, para. 18].
Claims 2-4, 36 and 37 are rejected under 35 U.S.C. 103 as being unpatentable over Eldar and Aguilar as cited above in view of Aagaard et al., US 2003/0210329.
2 and 36. Eldar is silent on static pre-calibration. Aagaard teaches a camera system wherein the processing device is configured to calibrate the camera model by:
obtaining a plurality of camera models, being statically pre-calibrated for different distances [calibration table holds predefined camera settings (models) for different distances, Figs. 1, 4, 13, 14, paras. 11, 12, 36, 38, 43, 44, 95-97, 112-114, 127]; and
selecting, for each of the plurality of distance sections, a selected camera model of the plurality of camera models based on the determined distance information of the respective distance section [multiple cameras are calibrated based on distance to subject and calibration table, Figs. 1, 4, 13, 14, paras. 11, 12, 36, 38, 43, 44, 95-97, 112-114, 127].
Before the effective filing date of the claimed invention, it would have been obvious to one skilled in the art to combine the references to ensure that the different camera images are all in focus even when observed objects are different distances from the camera. Calibration improves quality, and using a lookup table with predefined modes is an efficient method compared to generating new modes on the fly.
3 and 37. Aagaard teaches the device of claim 1, wherein the processing device is configured to calibrate the camera model by:
obtaining a parameterizable calibrated model [calibration table holds predefined camera settings (models) for different distances, Figs. 1, 4, 13, 14, paras. 11, 12, 36, 38, 43, 44, 95-97, 19-114, 127]; and
parametrizing, for each of the plurality of distance section, a parametrizable calibrated model based on the determined distance information of the respective distance section [calibration settings (models) are based on a created geometric transform functions (i.e. a parametrized equation to describe the settings, paras. 44, 50, 53, 54, 110, 111, 117, 119; also see Figs. 1, 4, 13, 14, paras. 11, 12, 36, 38, 43, 95-97, 112-114, 127].
4. Aargaard teaches the device of claim 1, wherein the camera model comprises an intrinsic compensation model [focus and/or zoom calibration is intrinsic, within the camera, para. 12, 44, 96, 97, 127] and an extrinsic compensation model [pan/tilt calibration is extrinsic since the whole camera moves relative to the environment, paras. 44, 95, 104, 108, 110].
Claims 5, 6, and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Eldar and Aguilar as cited above in view of Grundhoefer, US 11,979,657.
5. The above references are silent on a low resolution depth sensor. Grundhoefer teaches a camera system wherein at least one distance sensor of the plurality of distance sensors is a low resolution distance sensor [low resolution depth sensor, col. 6, 19-38].
Before the effective filing date of the claimed invention, it would have been obvious to one skilled in the art to combine the references, using a relatively low resolution depth sensor in order to save power and lower the cost of the depth sensor.
6. Grundhoefer teaches the device of claim 5, wherein the low resolution distance sensor has a resolution of one of 4x4, 2x2, or 1x1 [e.g. one pixel, col. 6, 19-38].
9. Grundhoefer teaches the device of claim 1, wherein the plurality of distance sensors comprise one of a Time of Flight sensor and/or a LIDAR sensor [col. 5, 44-52].
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Eldar and Aguilar as cited above in view of Xu et al., US 2023/0368495.
7. The above references are silent on more distance sensors than light cameras. Xu teaches a depth/RGB camera system wherein an amount of the plurality of visible light cameras is smaller than an amount of the plurality of distance sensors [e.g. one RGB camera and multiple depth cameras, para. 87].
Before the effective filing date of the claimed invention, it would have been obvious to one skilled in the art to combine the references, allowing the use of multiple depth cameras to capture a larger field of view and improve depth resolution, without duplicating the more expensive wide-lens RGB camera.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Eldar and Aguilar as cited above in view of Cullen et al., US 2023/0015889.
10. The above references are silent on a calibration time cycle. Cullen teaches a device wherein the processing device is configured to calibrate the camera model and determine the composite image in a predetermined time cycle [e.g. sequentially, paras. 3, 169].
Before the effective filing date of the claimed invention, it would have been obvious to one skilled in the art to combine the references, calibrating periodically in order to ensure that the camera calibration changes along with varying lighting and depth conditions, maintaining a quality image even when the scene is not static.
Claim 39 is rejected under 35 U.S.C. 103 as being unpatentable over Eldar and Aguilar as cited above in view of Souiai et al., US 2024/0371114.
39. The above references do not teach a certain time period for cyclic calibration. Souiai teaches a camera calibration method comprising calibrating, by the processing device, the camera model and determining the composite image in a predetermined time cycle, the predetermined time cycle being at least every 100ms [1-second (and/or 200 ms) calibration period, para. 98].
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine the references, calibrating with a one-second period in order to conserve power (by limiting frequency) while updating the calibration parameters frequently enough to maintain quality when motion is present in a scene.
Relevant Prior Art
Other prior art considered relevant but not cited in a rejection includes Hung et al., US 9,811,946, Fig. 3A, 3B, which shows a ring of RBG cameras but no interleaved depth sensors.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Timothy R Newlin whose telephone number is (571)270-3015. The examiner can normally be reached M-F 8-5 Mountain Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Benjamin Bruckart can be reached at 571-272-3982. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TIMOTHY R NEWLIN/ Examiner, Art Unit 2424
