DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Objections
Claims 10, 17, and 19 are objected to because of the following informalities: Claim 10 recites “intrinsic calibration data and extrinsic calibration” instead of “intrinsic calibration data and extrinsic calibration data.” In claims 17 and 19, the comma after “receiving” is unnecessary. Appropriate correction is required.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-2, 5-6, 14, 16, and 18-19 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Gupta (US 11856297).
Regarding claim 1, Gupta teaches: A multi-camera system for generating a panoramic image, the multi-camera system comprising:
a plurality of camera channels, individual of the plurality of camera channels being configured to capture image data in a respective field of view (Gupta col. 2 ll. 50-62 “The camera comprises three sensors indicated by reference numerals 102-106 ... The reference numeral 108 indicates the field-of-view (FOV) for the sensor 102, reference numeral 110 indicates the field-of-view (FOV) for the sensor 104, and reference numeral 112 indicates the field-of-view (FOV) for the sensor 106”);
memory; a processor; and computer-executable instructions stored in the memory and executable by the processor (Gupta Claim 12 “a non-transitory computer readable storage medium, having stored thereon, a set of computer-executable instructions”) to perform operations comprising:
receiving information specifying a panoramic image to be generated (Gupta col. 3 ll. 32-38 “The panoramic image streams are created using stitching techniques” col. 3 line 63 – col. 4 line 9 “The device is designed with known FOV of each camera and the amount of overlapping FOV between the multiple cameras”);
for a pixel location in the panoramic image, determining, based on the information and camera configuration data associated with the plurality of camera channels, at least a first camera channel associated with a first field of view and a second camera channel associated with a second field of view, wherein the first field of view and the second field of view include the pixel location (Gupta col. 3 line 63 – col. 4 line 9 “The device is designed with known FOV of each camera and the amount of overlapping FOV between the multiple cameras … the horizontal overlap in pixels between two adjacent imagers”);
determining, based on the camera configuration data, an overlap region between a first image captured by the first camera channel and a second image captured by the second camera channel (Gupta col. 3 line 63 – col. 4 line 9 “The device is designed with known FOV of each camera and the amount of overlapping FOV between the multiple cameras”);
determining, based on a first portion of the first image in the overlap region and a second portion of the second image in the overlap region, a pixel value associated with the pixel location (Gupta col. 5 ll. 41-50 “the original color values of the co-sited pixels in the left/right overlap regions are blended”); and
generating the panoramic image including the pixel value at the pixel location (Gupta col. 3 ll. 32-38 “The panoramic image streams are created using stitching techniques” col. 5 ll. 41-50 “the original color values of the co-sited pixels in the left/right overlap regions are blended”).
Regarding claim 2, Gupta teaches: The multi-camera system of claim 1, wherein determining the pixel value comprises:
determining a weighted average of a first value of a first pixel in the first portion of the first image and a second value of a second pixel in the second portion of the second image, wherein the pixel value associated with the pixel location is based on the weighted average (Gupta col. 5 ll. 41-50 “the original color values of the co-sited pixels in the left/right overlap regions are blended and the linear average of left and right overlap regions is computed, by using a weighting strategy”).
Regarding claim 5, Gupta teaches: The multi-camera system of claim 1, wherein determining the pixel value comprises:
determining a first weight corresponding to the first image and a second weight corresponding to the second image (Gupta col. 5 ll. 41-50 “applies a higher weight to left/right overlap pixels when there is a large difference between the colors of co-sited pixels in the left and right overlap region”); and
sampling pixel values from the first image and the second image based on the first weight and the second weight, wherein the pixel value is based on the sampled pixel values (Gupta col. 5 ll. 41-50 “the original color values of the co-sited pixels in the left/right overlap regions are blended and the linear average of left and right overlap regions is computed, by using a weighting strategy” [The co-sited pixels meet the sampled pixel values.]).
Regarding claim 6, Gupta teaches: The multi-camera system of claim 1, wherein determining the pixel value is based on content of the first image and the second image in the overlap region (Gupta col. 5 ll. 41-50 “the original color values of the co-sited pixels in the left/right overlap regions are blended”).
Regarding claim 14, Gupta teaches: A method for generating a panoramic image, comprising:
receiving a plurality of images of a scene captured by a respective plurality of camera channels (Gupta col. 2 line 63 – col. 3 line 13 “The images captured by the image sensors 102-106 are processed”);
determining, based on camera configuration data associated with the plurality of camera channels, an overlap region between a first image of the plurality of images captured by a first camera channel and a second image of the plurality of images captured by a second camera channel, wherein the overlap region includes a representation of content in a portion of the panoramic image (Gupta col. 3 line 63 – col. 4 line 9 “The device is designed with known FOV of each camera and the amount of overlapping FOV between the multiple cameras … the horizontal overlap in pixels between two adjacent imagers”);
determining, based on first pixel values of the first image in the overlap region and second pixel values of the second image in the overlap region, a pixel value associated with a pixel location in the portion of the panoramic image (Gupta col. 5 ll. 41-50 “the original color values of the co-sited pixels in the left/right overlap regions are blended”); and
generating the panoramic image including the pixel value at the pixel location (Gupta col. 3 ll. 32-38 “The panoramic image streams are created using stitching techniques” col. 5 ll. 41-50 “the original color values of the co-sited pixels in the left/right overlap regions are blended”).
Regarding claim 16, Gupta teaches: The method of claim 14, further comprising:
determining, based on a first location of the first pixel values and a second location of the second pixel values, a first weight corresponding to the first image and a second weight corresponding to the second image (Gupta col. 5 ll. 41-50 “applies a higher weight to left/right overlap pixels when there is a large difference between the colors of co-sited pixels in the left and right overlap region”),
wherein determining the pixel value comprises one of:
determining, based on the first weight and the second weight, a weighted average of a portion of the first pixel values and the second pixel values (Gupta col. 5 ll. 41-50 “the original color values of the co-sited pixels in the left/right overlap regions are blended and the linear average of left and right overlap regions is computed, by using a weighting strategy”), or
determining, based on the first weight and the second weight, a stochastic sampling of the first pixel values and the second pixel values [This is yet to be considered because of the “one of” recitation.].
Regarding claim 18, Gupta teaches: The method of claim 14, wherein determining the pixel value is based on content of the first image and the second image in the overlap region (Gupta col. 5 ll. 41-50 “the original color values of the co-sited pixels in the left/right overlap regions are blended”).
Regarding claim 19, Gupta teaches: The method of claim 14, further comprising:
receiving, first calibration data associated with the first camera channel and second calibration data associated with the second camera channel (Gupta col. 3 line 63 – col. 4 line 9 “The device is designed with known FOV of each camera and the amount of overlapping FOV between the multiple cameras”); and
adjusting, based on the first calibration data and the second calibration data, the first pixel values and the second pixel values (Gupta col. 4 ll. 48-63 “The overlap regions are dynamically adjusted by the camera system … They can be made smaller or larger in value as compared to the factory calibrated value”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gupta (US 11856297) in view of Cutler (US 2003/0234866).
Regarding claim 3, Gupta does not teach/suggest: The multi-camera system of claim 2, wherein weights of the weighted average are based on a first distance between the first pixel and an edge of the overlap region and a second distance between the second pixel and the edge of the overlap region. Cutler, however, teaches/suggests weights of the weighted average are based on a first distance between the first pixel and an edge of the overlap region and a second distance between the second pixel and the edge of the overlap region (Cutler [0098] “the luminance value of the pixels in each image are weighted proportionally to their distance to the edge of the overlapping region”). Before the effective filing date of the claimed invention, the substitution of one known element (the weighting of Cutler) for another (the weighting of Gupta) would have been obvious to one of ordinary skill in the art because such substitutions would have yielded predictable results, namely for the blending.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gupta (US 11856297) in view of Jiang et al. (US 2024/0177300).
Regarding claim 4, Gupta does not teach/suggest: The multi-camera system of claim 2, wherein weights of the weighted average are based on a first distance between the first pixel and a center pixel of the first image and a second distance between the second pixel and a center pixel of the second image. Jiang, however, teaches/suggests weights of the weighted average are based on a first distance between the first pixel and a center pixel of the first image and a second distance between the second pixel and a center pixel of the second image (Jiang [0016] “merges the same pixel from multiple patches by weighing the pixels relatively to its distance to the center”). Before the effective filing date of the claimed invention, the substitution of one known element (the weighting of Jiang) for another (the weighting of Gupta) would have been obvious to one of ordinary skill in the art because such substitutions would have yielded predictable results, namely for the blending.
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gupta (US 11856297) in view of Frisken et al. (US 2004/0189644).
Regarding claim 7, Gupta teaches/suggests weighted average of pixel values of the first image and the second image (Gupta col. 5 ll. 41-50 “the original color values of the co-sited pixels in the left/right overlap regions are blended and the linear average of left and right overlap regions is computed, by using a weighting strategy”). Gupta does not teach/suggest: The multi-camera system of claim 6, the operations further comprising:
determining a frequency signature of the content;
based on the frequency signature, determining the pixel value using stochastic sampling of pixel values of the first image and the second image.
Frisken, however, teaches/suggests:
determining a frequency signature of the content (Frisken [0101] “Stochastic sampling tends to replace moir pattern aliasing with high frequency noise and has been shown to be particularly effective in reducing temporal aliasing”);
Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the co-sited pixels of Gupta to be sampled as taught/suggested by Frisken to reduce temporal aliasing. As such, Gupta as modified by Frisken teaches/suggests:
based on the frequency signature, determining the pixel value using stochastic sampling of pixel values of the first image and the second image (Gupta col. 5 ll. 41-50 “the original color values of the co-sited pixels in the left/right overlap regions are blended” Frisken [0101] “Stochastic sampling tends to replace moir pattern aliasing with high frequency noise and has been shown to be particularly effective in reducing temporal aliasing”).
Claim(s) 8 and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gupta (US 11856297) in view of Battles et al. (US 2007/0097254).
Regarding claim 8, Gupta does not teach/suggest: The multi-camera system of claim 6, wherein the content comprises one of: a flare or a veiling glare. Battles, however, teaches/suggests a veiling glare (Battles [0003] “bright spots in the image, such as glare, may cause the pixels imaging the bright spots to become clipped” [0016] “the number of clipped and/or dark pixels is determined based on the changed exposure settings”). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the co-sited pixels of Gupta to be adjusted as taught/suggested by Battles to reduce flaring.
Regarding claim 12, Gupta does not teach/suggest: The multi-camera system of claim 1, the operations further comprising:
determining respective exposure levels associated with the first camera channel and the second camera channel;
adjusting, based on the respective exposure levels, pixel values in the overlap region of the first image and the second image.
Battles, in view of Gupta, teaches/suggests:
determining respective exposure levels associated with the first camera channel and the second camera channel (Gupta col. 2 ll. 50-62 “The camera comprises three sensors indicated by reference numerals 102-106” Battles [0016] “the processor 106 of FIG. 1 may receive the data indicative of the changed exposure setting”);
adjusting, based on the respective exposure levels, pixel values in the overlap region of the first image and the second image (Gupta col. 5 ll. 41-50 “the original color values of the co-sited pixels in the left/right overlap regions are blended” Battles [0016] “the processor 106 may change the values of the previously generated image data so as to reflect values based on the new exposure settings”).
The same rationale to combine as set forth in the rejection of claim 8 above is incorporated herein.
Claim(s) 9, 11, and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gupta (US 11856297) in view of Kurtz et al. (WO 2020/263867).
Regarding claim 9, Gupta teaches/suggests: The multi-camera system of claim 1, wherein:
the plurality of camera channels comprise at least three camera channels (Gupta col. 2 ll. 50-62 “The camera comprises three sensors indicated by reference numerals 102-106”),
Gupta does not teach/suggest:
the field of view comprises a polygon of more than four sides, and
the panoramic image comprises an equirectangular panorama.
Kurtz, however, teaches/suggests:
the field of view comprises a polygon of more than four sides (Kurtz p. 10 ll. 14-35 “compared to facets with pentagonal and or hexagonal facets, as they have fewer edges to cut to provide polygonal edges on the outermost lens element, so as to define a captured polygonal FOV”), and
the panoramic image comprises an equirectangular panorama (Kurtz p. 10 ll. 14-35 “produce high quality low-parallax panoramic images from an improved multi-camera panoramic capture device … a 360-degree panorama equirectangular image”).
Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the camera of Gupta to be that of Kurtz for low-parallax panoramic images.
Regarding claim 11, Gupta teaches/suggests: The multi-camera system of claim 1, wherein determining the first camera channel comprises:
determining, based on the camera configuration data, a location on an imaging the pixel location in the panoramic image (Gupta col. 3 ll. 21-27 “FIG. 4 shows a projection of the image planes of each the sensors of the camera of FIG. 1 onto a cylindrical plane” col. 3 line 63 – col. 4 line 9 “The device is designed with known FOV of each camera and the amount of overlapping FOV between the multiple cameras … the horizontal overlap in pixels between two adjacent imagers”); and
determining that the first field of view includes the location on the imaging
Gupta does not teach/suggest an imaging sphere. Kurtz, however, teaches/suggests an imaging sphere (Kurtz p. 10 ll. 14-35 “a plurality of cameras arranged around a circumference of a sphere to capture a 360-degree annular FOV”). The same rationale to combine as set forth in the rejection of claim 9 above is incorporated herein.
Regarding claim 15, Gupta teaches/suggests: The method of claim 14, wherein the plurality of camera channels comprise at least three cameras (Gupta col. 2 ll. 50-62 “The camera comprises three sensors indicated by reference numerals 102-106”). Gupta does not teach/suggest low-parallax cameras, wherein at least one edge of a first camera adjoins an edge of a second camera. Kurtz, however, teaches/suggests low-parallax cameras, wherein at least one edge of a first camera adjoins an edge of a second camera (Kurtz p. 1 ll. 13-17 “panoramic low-parallax multi-camera capture devices having a plurality of adjacent and abutting polygonal cameras”). The same rationale to combine as set forth in the rejection of claim 9 above is incorporated herein.
Claim(s) 10 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gupta (US 11856297).
Regarding claim 10, Gupta teaches/suggests: The multi-camera system of claim 1, wherein the camera configuration data includes
determining a first mathematical model corresponding to
determining a second mathematical model corresponding to mathematical model is an inherent and/or implicit feature of the FOV of the second camera.]);
determining, based on
Gupta is silent regarding intrinsic calibration data and extrinsic calibration data. However, the concept and advantages of intrinsic and extrinsic calibration data are well known and expected in the art (Official Notice). It would have been obvious for the calibration of Gupta to include such data to determine the overlapping.
Regarding claim 20, Gupta does not teach/suggest: The method of claim 14, wherein determining the pixel value is based on inputting, to a machine-learned model, the first pixel values and the second pixel values. However, the concept and advantages of a machine-learned model are well known and expected in the art (Official Notice). It would have been obvious for the co-sited pixels of Gupta to be determined using such model for machine learning.
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gupta (US 11856297) in view of Ben-David et al. (US 2013/0009950).
Regarding claim 10, Gupta teaches/suggests: The multi-camera system of claim 1, wherein the panoramic image is a first panoramic image of a scene and the first image and the second image are captured from a first position of the multi-camera system (Gupta col. 2 line 63 – col. 3 line 13 “The images captured by the image sensors 102-106 are processed”). Gupta does not teach/suggest the operations further comprising:
receiving a set of images of the scene captured from a second position of the multi-camera system;
determining, based on the set of images, a second panoramic image; and
determining, based on the first panoramic image and the second panoramic image, a 3D model of a portion of the scene.
Ben-David, however, teaches/suggests a 3D model of a portion of the scene (Ben-David [0083] “real-time images 300A, 300B are optionally sent to an image preprocessing module 301 and then sent to a construction module 302 that generates one or more 3D models … A multi-view rendering module 336 uses the view selection criteria in combination with the 3D model and maps to select images (real-time images or stored images as appropriate) and renders a multi-view”). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the images of Gupta to be captured at different positions as taught/suggested by Ben-David for 3D modeling. As such, Gupta as modified by Ben-David teaches/suggests:
receiving a set of images of the scene captured from a second position of the multi-camera system (Gupta col. 2 line 63 – col. 3 line 13 “The images captured by the image sensors 102-106 are processed” Ben-David [0083] “real-time images 300A, 300B are optionally sent to an image preprocessing module 301 and then sent to a construction module 302 that generates one or more 3D models”);
determining, based on the set of images, a second panoramic image (Gupta col. 3 ll. 32-38 “The panoramic image streams are created using stitching techniques”); and
determining, based on the first panoramic image and the second panoramic image, a 3D model of a portion of the scene (Gupta col. 3 ll. 32-38 “The panoramic image streams are created using stitching techniques” Ben-David [0083] “real-time images 300A, 300B are optionally sent to an image preprocessing module 301 and then sent to a construction module 302 that generates one or more 3D models”).
Claim(s) 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gupta (US 11856297) in view of Mandelbaum et al. (US 2004/0100443).
Regarding claim 17, Gupta does not teach/suggest: The method of claim 14, further comprising:
receiving, an object track associated with two or more camera channels of the plurality of camera channels, wherein determining the first image and the second image is based on the object track.
Mandelbaum, however, teaches/suggests:
receiving, an object track associated with two or more camera channels of the plurality of camera channels, wherein determining the first image and the second image is based on the object track (Mandelbaum [0024] “track moving objects within the panoramic viewing area” [0026] “the fields of view 14 overlap 16 so as to enable smooth blending of neighboring fields of view 14” [Tracking the object in the overlapping FOV meets the determining.]).
Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the overlapping FOV of Gupta to include the objects of Mandelbaum for tracking.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
US 2019/0058811 – image stitching
US 2020/0104977 – image stitching
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANH-TUAN V NGUYEN whose telephone number is 571-270-7513. The examiner can normally be reached on M-F 9AM-5PM ET. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JASON CHAN can be reached on 571-272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANH-TUAN V NGUYEN/
Primary Examiner, Art Unit 2619