DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Applicant’s submission filed on 06/25/2026 has been entered. Claims 1, 10, 14, 17, and 19 were amended. Claims 1-20 are pending in the application.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 5-6, 10, 14, 16, and 18-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gupta (US 11856297) in view of Matsumoto et al. (US 2012/0257007).
Regarding claim 1, Gupta teaches/suggests: A multi-camera system for generating a panoramic image, the multi-camera system comprising:
a plurality of camera channels, individual of the plurality of camera channels being configured to capture image data in a respective field of view (Gupta col. 2 ll. 50-62 “The camera comprises three sensors indicated by reference numerals 102-106 ... The reference numeral 108 indicates the field-of-view (FOV) for the sensor 102, reference numeral 110 indicates the field-of-view (FOV) for the sensor 104, and reference numeral 112 indicates the field-of-view (FOV) for the sensor 106”);
memory; a processor; and computer-executable instructions stored in the memory and executable by the processor to perform operations (Gupta Claim 12 “a non-transitory computer readable storage medium, having stored thereon, a set of computer-executable instructions”) comprising:
for a pixel location in the panoramic image, determining, based on camera configuration data associated with the plurality of camera channels, at least a first camera channel associated with a first field of view and a second camera channel associated with a second field of view, wherein the first field of view and the second field of view include the pixel location (Gupta col. 3 line 63 – col. 4 line 9 “The device is designed with known FOV of each camera and the amount of overlapping FOV between the multiple cameras … the horizontal overlap in pixels between two adjacent imagers”);
determining, based on the camera configuration data, an overlap region between a first image captured by the first camera channel and a second image captured by the second camera channel (Gupta col. 3 line 63 – col. 4 line 9 “The device is designed with known FOV of each camera and the amount of overlapping FOV between the multiple cameras”);
determining, based on a first portion of the first image in the overlap region and a second portion of the second image in the overlap region, a pixel value associated with the pixel location (Gupta col. 5 ll. 41-50 “the original color values of the co-sited pixels in the left/right overlap regions are blended”); and
generating the panoramic image including the pixel value at the pixel location (Gupta col. 3 ll. 32-38 “The panoramic image streams are created using stitching techniques” col. 5 ll. 41-50 “the original color values of the co-sited pixels in the left/right overlap regions are blended”).
Gupta does not teach/suggest:
receiving information specifying a panoramic image to be generated, the information indicating a spatial extent of the panoramic image;
determining, based on the spatial extent and camera configuration data associated with the plurality of camera channels, at least a first camera channel associated with a first field of view and a second camera channel associated with a second field of view;
Matsumoto, however, teaches/suggests:
receiving information specifying a panoramic image to be generated, the information indicating a spatial extent of the panoramic image (Matsumoto [0054] “when the imaging range of a wide-range image is specified in advance using the key inputting section 16, a panoramic image of a size based on the specified imaging range is generated”);
Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the generating of Gupta include the imaging range (the spatial extent) of Matsumoto so that the size of the panoramic images is based on the imaging range. As such, Gupta as modified by Matsumoto teaches/suggests:
determining, based on the spatial extent and camera configuration data associated with the plurality of camera channels, at least a first camera channel associated with a first field of view and a second camera channel associated with a second field of view (Gupta col. 3 line 63 – col. 4 line 9 “The device is designed with known FOV of each camera and the amount of overlapping FOV between the multiple cameras … the horizontal overlap in pixels between two adjacent imagers” Matsumoto [0054] “when the imaging range of a wide-range image is specified in advance using the key inputting section 16, a panoramic image of a size based on the specified imaging range is generated”);
Regarding claim 2, Gupta as modified by Matsumoto teaches/suggests: The multi-camera system of claim 1, wherein determining the pixel value comprises:
determining a weighted average of a first value of a first pixel in the first portion of the first image and a second value of a second pixel in the second portion of the second image, wherein the pixel value associated with the pixel location is based on the weighted average (Gupta col. 5 ll. 41-50 “the original color values of the co-sited pixels in the left/right overlap regions are blended and the linear average of left and right overlap regions is computed, by using a weighting strategy”).
Regarding claim 5, Gupta as modified by Matsumoto teaches/suggests: The multi-camera system of claim 1, wherein determining the pixel value comprises:
determining a first weight corresponding to the first image and a second weight corresponding to the second image (Gupta col. 5 ll. 41-50 “applies a higher weight to left/right overlap pixels when there is a large difference between the colors of co-sited pixels in the left and right overlap region”); and
sampling pixel values from the first image and the second image based on the first weight and the second weight, wherein the pixel value is based on the sampled pixel values (Gupta col. 5 ll. 41-50 “the original color values of the co-sited pixels in the left/right overlap regions are blended and the linear average of left and right overlap regions is computed, by using a weighting strategy” [The co-sited pixels meet the sampled pixel values.]).
Regarding claim 6, Gupta as modified by Matsumoto teaches/suggests: The multi-camera system of claim 1, wherein determining the pixel value is based on content of the first image and the second image in the overlap region (Gupta col. 5 ll. 41-50 “the original color values of the co-sited pixels in the left/right overlap regions are blended”).
Regarding claim 10, Gupta as modified by Matsumoto teaches/suggests: The multi-camera system of claim 1, wherein the camera configuration data includes FOV of each camera and the amount of overlapping FOV between the multiple cameras … during the factory calibration process two sets of overlap parameters and three sets of cylindrical warp parameters are determined”), the operations further comprising:
determining a first mathematical model corresponding to
determining a second mathematical model corresponding to
determining, based on
Gupta and Matsumoto are silent regarding intrinsic calibration data and extrinsic calibration data. However, official notice is taken that the concept and advantages of intrinsic and extrinsic calibration data are well known and expected in the art. It would have been obvious for the calibration of Gupta as modified by Matsumoto to include such data to determine the overlapping.
Regarding claim 14, Gupta as modified by Matsumoto teaches: A method for generating a panoramic image, comprising:
receiving a plurality of images of a scene captured by a respective plurality of camera channels (Gupta col. 2 line 63 – col. 3 line 13 “The images captured by the image sensors 102-106 are processed”);
receiving information including a spatial extent of the panoramic image to be generated (Matsumoto [0054] “when the imaging range of a wide-range image is specified in advance using the key inputting section 16, a panoramic image of a size based on the specified imaging range is generated”);
determining, based on camera configuration data associated with the plurality of camera channels and the spatial extent of the panoramic image, an overlap region between a first image of the plurality of images captured by a first camera channel and a second image of the plurality of images captured by a second camera channel, wherein the overlap region includes a representation of content in a portion of the panoramic image (Gupta col. 3 line 63 – col. 4 line 9 “The device is designed with known FOV of each camera and the amount of overlapping FOV between the multiple cameras … the horizontal overlap in pixels between two adjacent imagers” Matsumoto [0054] “when the imaging range of a wide-range image is specified in advance using the key inputting section 16, a panoramic image of a size based on the specified imaging range is generated”);
determining, based on first pixel values of the first image in the overlap region and second pixel values of the second image in the overlap region, a pixel value associated with a pixel location in the portion of the panoramic image (Gupta col. 5 ll. 41-50 “the original color values of the co-sited pixels in the left/right overlap regions are blended”); and
generating the panoramic image including the pixel value at the pixel location (Gupta col. 3 ll. 32-38 “The panoramic image streams are created using stitching techniques” col. 5 ll. 41-50 “the original color values of the co-sited pixels in the left/right overlap regions are blended”).
The same rationale to combine as set forth in the rejection of claim 1 is incorporated herein.
Regarding claim 16, Gupta as modified by Matsumoto teaches: The method of claim 14, further comprising:
determining, based on a first location of the first pixel values and a second location of the second pixel values, a first weight corresponding to the first image and a second weight corresponding to the second image (Gupta col. 5 ll. 41-50 “applies a higher weight to left/right overlap pixels when there is a large difference between the colors of co-sited pixels in the left and right overlap region”),
wherein determining the pixel value comprises one of:
determining, based on the first weight and the second weight, a weighted average of a portion of the first pixel values and the second pixel values (Gupta col. 5 ll. 41-50 “the original color values of the co-sited pixels in the left/right overlap regions are blended and the linear average of left and right overlap regions is computed, by using a weighting strategy”), or
determining, based on the first weight and the second weight, a stochastic sampling of the first pixel values and the second pixel values [This is yet to be considered because of the “one of” recitation.].
Regarding claim 18, Gupta as modified by Matsumoto teaches: The method of claim 14, wherein determining the pixel value is based on content of the first image and the second image in the overlap region (Gupta col. 5 ll. 41-50 “the original color values of the co-sited pixels in the left/right overlap regions are blended”).
Regarding claim 19, Gupta as modified by Matsumoto teaches: The method of claim 14, further comprising:
receiving first calibration data associated with the first camera channel and second calibration data associated with the second camera channel (Gupta col. 3 line 63 – col. 4 line 9 “The device is designed with known FOV of each camera and the amount of overlapping FOV between the multiple cameras”); and
adjusting, based on the first calibration data and the second calibration data, the first pixel values and the second pixel values (Gupta col. 4 ll. 48-63 “The overlap regions are dynamically adjusted by the camera system … They can be made smaller or larger in value as compared to the factory calibrated value”).
Regarding claim 20, Gupta as modified by Matsumoto does not teach/suggest: The method of claim 14, wherein determining the pixel value is based on inputting, to a machine-learned model, the first pixel values and the second pixel values. However, official notice is taken that the concept and advantages of a machine-learned model are well known and expected in the art. It would have been obvious for the co-sited pixels of Gupta as modified by Matsumoto to be determined using such a model for machine learning.
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gupta (US 11856297) in view of Matsumoto et al. (US 2012/0257007) as applied to claim 2 above, and further in view of Cutler (US 2003/0234866).
Regarding claim 3, Gupta as modified by Matsumoto does not teach/suggest: The multi-camera system of claim 2, wherein weights of the weighted average are based on a first distance between the first pixel and an edge of the overlap region and a second distance between the second pixel and the edge of the overlap region. Cutler, however, teaches/suggests weights of the weighted average are based on a first distance between the first pixel and an edge of the overlap region and a second distance between the second pixel and the edge of the overlap region (Cutler [0098] “the luminance value of the pixels in each image are weighted proportionally to their distance to the edge of the overlapping region”). Before the effective filing date of the claimed invention, the substitution of one known element (the weighting of Cutler) for another (the weighting of Gupta) would have been obvious to one of ordinary skill in the art because such substitutions would have yielded predictable results, namely for the blending.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gupta (US 11856297) in view of Matsumoto et al. (US 2012/0257007) as applied to claim 2 above, and further in view of Jiang et al. (US 2024/0177300).
Regarding claim 4, Gupta as modified by Matsumoto does not teach/suggest: The multi-camera system of claim 2, wherein weights of the weighted average are based on a first distance between the first pixel and a center pixel of the first image and a second distance between the second pixel and a center pixel of the second image. Jiang, however, teaches/suggests weights of the weighted average are based on a first distance between the first pixel and a center pixel of the first image and a second distance between the second pixel and a center pixel of the second image (Jiang [0016] “merges the same pixel from multiple patches by weighing the pixels relatively to its distance to the center”). Before the effective filing date of the claimed invention, the substitution of one known element (the weighting of Jiang) for another (the weighting of Gupta) would have been obvious to one of ordinary skill in the art because such substitutions would have yielded predictable results, namely for the blending.
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gupta (US 11856297) in view of Matsumoto et al. (US 2012/0257007) as applied to claim 6 above, and further in view of Frisken et al. (US 2004/0189644).
Regarding claim 7, Gupta as modified by Matsumoto teaches/suggests weighted average of pixel values of the first image and the second image (Gupta col. 5 ll. 41-50 “the original color values of the co-sited pixels in the left/right overlap regions are blended and the linear average of left and right overlap regions is computed, by using a weighting strategy”). Gupta as modified by Matsumoto does not teach/suggest: The multi-camera system of claim 6, the operations further comprising:
determining a frequency signature of the content;
based on the frequency signature, determining the pixel value using stochastic sampling of pixel values of the first image and the second image.
Frisken, however, teaches/suggests:
determining a frequency signature of the content (Frisken [0101] “Stochastic sampling tends to replace moir pattern aliasing with high frequency noise and has been shown to be particularly effective in reducing temporal aliasing”);
Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the co-sited pixels of Gupta as modified by Matsumoto to be sampled as taught/suggested by Frisken to reduce temporal aliasing. As such, Gupta as modified by Matsumoto and Frisken teaches/suggests:
based on the frequency signature, determining the pixel value using stochastic sampling of pixel values of the first image and the second image (Gupta col. 5 ll. 41-50 “the original color values of the co-sited pixels in the left/right overlap regions are blended” Frisken [0101] “Stochastic sampling tends to replace moir pattern aliasing with high frequency noise and has been shown to be particularly effective in reducing temporal aliasing”).
Claim(s) 8 and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gupta (US 11856297) in view of Matsumoto et al. (US 2012/0257007) as applied to claim 1 above, and further in view of Battles et al. (US 2007/0097254).
Regarding claim 8, Gupta as modified by Matsumoto does not teach/suggest: The multi-camera system of claim 6, wherein the content comprises one of: a flare or a veiling glare. Battles, however, teaches/suggests a veiling glare (Battles [0003] “bright spots in the image, such as glare, may cause the pixels imaging the bright spots to become clipped” [0016] “the number of clipped and/or dark pixels is determined based on the changed exposure settings”). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the co-sited pixels of Gupta as modified by Matsumoto to be adjusted as taught/suggested by Battles to reduce flaring.
Regarding claim 12, Gupta as modified by Matsumoto does not teach/suggest: The multi-camera system of claim 1, the operations further comprising:
determining respective exposure levels associated with the first camera channel and the second camera channel;
adjusting, based on the respective exposure levels, pixel values in the overlap region of the first image and the second image.
Battles, in view of Gupta, teaches/suggests:
determining respective exposure levels associated with the first camera channel and the second camera channel (Gupta col. 2 ll. 50-62 “The camera comprises three sensors indicated by reference numerals 102-106” Battles [0016] “the processor 106 of FIG. 1 may receive the data indicative of the changed exposure setting”);
adjusting, based on the respective exposure levels, pixel values in the overlap region of the first image and the second image (Gupta col. 5 ll. 41-50 “the original color values of the co-sited pixels in the left/right overlap regions are blended” Battles [0016] “the processor 106 may change the values of the previously generated image data so as to reflect values based on the new exposure settings”).
The same rationale to combine as set forth in the rejection of claim 8 is incorporated herein.
Claim(s) 9, 11, and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gupta (US 11856297) in view of Matsumoto et al. (US 2012/0257007) as applied to claims 1 and 14 above, and further in view of Kurtz et al. (WO 2020/263867).
Regarding claim 9, Gupta as modified by Matsumoto teaches/suggests: The multi-camera system of claim 1, wherein:
the plurality of camera channels comprise at least three camera channels (Gupta col. 2 ll. 50-62 “The camera comprises three sensors indicated by reference numerals 102-106”),
Gupta as modified by Matsumoto does not teach/suggest:
the field of view comprises a polygon of more than four sides, and
the panoramic image comprises an equirectangular panorama.
Kurtz, however, teaches/suggests:
the field of view comprises a polygon of more than four sides (Kurtz p. 10 ll. 14-35 “compared to facets with pentagonal and or hexagonal facets, as they have fewer edges to cut to provide polygonal edges on the outermost lens element, so as to define a captured polygonal FOV”), and
the panoramic image comprises an equirectangular panorama (Kurtz p. 10 ll. 14-35 “produce high quality low-parallax panoramic images from an improved multi-camera panoramic capture device … a 360-degree panorama equirectangular image”).
Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the camera of Gupta as modified by Matsumoto to be that of Kurtz for low-parallax panoramic images.
Regarding claim 11, Gupta as modified by Matsumoto teaches/suggests: The multi-camera system of claim 1, wherein determining the first camera channel comprises:
determining, based on the camera configuration data, a location on an imaging
determining that the first field of view includes the location on the imaging
Gupta as modified by Matsumoto does not teach/suggest an imaging sphere. Kurtz, however, teaches/suggests an imaging sphere (Kurtz p. 10 ll. 14-35 “a plurality of cameras arranged around a circumference of a sphere to capture a 360-degree annular FOV”). The same rationale to combine as set forth in the rejection of claim 9 is incorporated herein.
Regarding claim 15, Gupta as modified by Matsumoto teaches/suggests: The method of claim 14, wherein the plurality of camera channels comprise at least three cameras (Gupta col. 2 ll. 50-62 “The camera comprises three sensors indicated by reference numerals 102-106”). Gupta as modified by Matsumoto does not teach/suggest low-parallax cameras, wherein at least one edge of a first camera adjoins an edge of a second camera. Kurtz, however, teaches/suggests low-parallax cameras, wherein at least one edge of a first camera adjoins an edge of a second camera (Kurtz p. 1 ll. 13-17 “panoramic low-parallax multi-camera capture devices having a plurality of adjacent and abutting polygonal cameras”). The same rationale to combine as set forth in the rejection of claim 9 is incorporated herein.
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gupta (US 11856297) in view of Matsumoto et al. (US 2012/0257007) as applied to claim 1 above, and further in view of Ben-David et al. (US 2013/0009950).
Regarding claim 13, Gupta as modified by Matsumoto teaches/suggests: The multi-camera system of claim 1, wherein the panoramic image is a first panoramic image of a scene and the first image and the second image are captured from a first position of the multi-camera system (Gupta col. 2 line 63 – col. 3 line 13 “The images captured by the image sensors 102-106 are processed”). Gupta as modified by Matsumoto does not teach/suggest the operations further comprising:
receiving a set of images of the scene captured from a second position of the multi-camera system;
determining, based on the set of images, a second panoramic image; and
determining, based on the first panoramic image and the second panoramic image, a 3D model of a portion of the scene.
Ben-David, however, teaches/suggests a 3D model of a portion of the scene (Ben-David [0083] “real-time images 300A, 300B are optionally sent to an image preprocessing module 301 and then sent to a construction module 302 that generates one or more 3D models … A multi-view rendering module 336 uses the view selection criteria in combination with the 3D model and maps to select images (real-time images or stored images as appropriate) and renders a multi-view”). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the images of Gupta as modified by Matsumoto to be captured at different positions as taught/suggested by Ben-David for 3D modeling. As such, Gupta as modified by Matsumoto and Ben-David teaches/suggests:
receiving a set of images of the scene captured from a second position of the multi-camera system (Gupta col. 2 line 63 – col. 3 line 13 “The images captured by the image sensors 102-106 are processed” Ben-David [0083] “real-time images 300A, 300B are optionally sent to an image preprocessing module 301 and then sent to a construction module 302 that generates one or more 3D models”);
determining, based on the set of images, a second panoramic image (Gupta col. 3 ll. 32-38 “The panoramic image streams are created using stitching techniques”); and
determining, based on the first panoramic image and the second panoramic image, a 3D model of a portion of the scene (Gupta col. 3 ll. 32-38 “The panoramic image streams are created using stitching techniques” Ben-David [0083] “real-time images 300A, 300B are optionally sent to an image preprocessing module 301 and then sent to a construction module 302 that generates one or more 3D models”).
Claim(s) 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gupta (US 11856297) in view of Matsumoto et al. (US 2012/0257007) as applied to claim 14 above, and further in view of Mandelbaum et al. (US 2004/0100443).
Regarding claim 17, Gupta as modified by Matsumoto does not teach/suggest: The method of claim 14, further comprising:
receiving an object track associated with two or more camera channels of the plurality of camera channels, wherein determining the first image and the second image is based on the object track.
Mandelbaum, however, teaches/suggests:
receiving an object track associated with two or more camera channels of the plurality of camera channels, wherein determining the first image and the second image is based on the object track (Mandelbaum [0024] “track moving objects within the panoramic viewing area” [0026] “the fields of view 14 overlap 16 so as to enable smooth blending of neighboring fields of view 14” [Tracking the object in the overlapping FOV meets the determining.]).
Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the overlapping FOV of Gupta as modified by Matsumoto to include the objects of Mandelbaum for tracking.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
US 2016/0246061 – panoramic image extent
US 2017/0264832 – panoramic image size
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANH-TUAN V NGUYEN whose telephone number is 571-270-7513. The examiner can normally be reached on M-F 9AM-5PM ET. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JASON CHAN can be reached on 571-272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANH-TUAN V NGUYEN/
Primary Examiner, Art Unit 2619
Prosecution Insights