Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al. (US Pub 2019/0132576 A1) in view of Su et al. (US Pub 2024/0046596 A1).
As to claim 1, Zhang discloses a computer-implemented method for modelling a projection of a scene in three-dimensional space into a composite image by a camera system with a plurality of cameras (Zhang, abstract, Fig. 8.), comprising:
subsequently projecting the scene onto a plurality of camera unit spheres and a compositing (Zhang, Fig. 5, ¶0025, “Rectification may be used to transform images by projecting two-or-more images onto a common image plane or spherical surface.” ¶0042, “a corrected and re-projected ultra-wide FOV fisheye image 500. Given a 3D world point P (X, Y, Z), its corresponding 3D vector (x, y, z) on a unit sphere can be computed using its angular spherical coordinates (θ, φ)” ¶0043, “For any point in the fisheye image plane, equations (1), (2) and (3) are used to derive the corresponding angular spherical coordinates in the unit sphere. Those spherical coordinates are then used to re-project the point on the unit sphere to an equirectangular image plane” ¶0044, “the circular camera array may be divided into two half circles for processing. The camera array is divided to calculate the extrinsic parameters due to the wide FOV captured by each camera.” ¶0052, “Correcting the image may also include undistorting the images. The undistorted image may be projected onto a unit sphere. The projection onto the sphere may then be used to re-project the image from the unit sphere to an equirectangular image plane.”);
wherein each camera unit sphere of the plurality of camera unit spheres represents one camera of the plurality of cameras, respectively (Zhang, Fig. 5, Fig.9, ¶0042, “For any point in the fisheye image plane, equations (1), (2) and (3) are used to derive the corresponding angular spherical coordinates in the unit sphere.”);
wherein a radius of the camera unit spheres and the compositing (Zhang, ¶0025, “An image corrector and projector 112 can obtain calibrated camera intrinsic parameters from the intrinsic calibrator 110 and apply fisheye correction and spherical projection to all fisheye images. The correction will undistort the fisheye image, and the re-projection will project the image content from a fisheye image plane to an equirectangular image plane. An extrinsic calibrator 114 may be used to estimate camera extrinsic parameters. Extrinsic parameters are parameters used to describe the transformation between the camera and its external world. To determine the extrinsic parameters, a number of reference cameras may be selected, and a stereo calibration may be performed between each camera pair. A stereo calibration may then be performed between the reference cameras to complete a full camera circle extrinsic calibration. The images may also be rectified and aligned so that pixels between each image pair are aligned along the same horizontal line. Rectification may be used to transform images by projecting two-or-more images onto a common image plane or spherical surface. For example, instead of maintaining perfect camera alignment, rectification of images may be used to align the images taken by cameras that may be misaligned. The resulting images have no vertical disparities nor any fisheye distortion. In this manner, a smooth view transition between each neighboring view is enabled.” ¶0045, “FIG. 6 and FIG. 7 show the top view and side view of the cameras positioned according to the estimated rotations and translations (extrinsic calibration results) for half of the circle. Stereo calibration is then performed between the two reference cameras to finish the full camera circle extrinsic calibration.” ¶00406-0047.).
Zhang does not disclose wherein the compositing image unit sphere unifies the plurality of camera unit spheres, wherein a compositing unit sphere centre of the compositing unit sphere is equally distanced by a unified offset (β) to each camera unit sphere centre of the plurality of camera unit spheres;
Su teaches the compositing image unit sphere unifies the plurality of camera unit spheres, wherein a compositing unit sphere centre of the compositing unit sphere is equally distanced by a unified offset (β) to each camera unit sphere centre of the plurality of camera unit spheres (SU, ¶0010, “[0010] According to a second aspect, the present application provides an image splicing apparatus, and the apparatus includes: [0011] an acquisition module, being configured to respectively acquire internal parameters and external parameters of lens of multiple cameras, the distance from the centers of circles formed by the optical centers of the lens of the multiple cameras to a preset plane above or below the circles, and multiple images collected by the multiple cameras respectively, wherein the multiple images are images collected by the lens of the multiple cameras at the same time respectively; [0012] a first mapping module, being configured to calculate a first mapping table corresponding to the lens of each camera according to the internal parameters of the lens of each camera, and project images collected by the corresponding lens of each camera at the same time onto a unit sphere according to the first mapping table corresponding to the lens of each camera so as to obtain multiple first spherical images respectively corresponding to the lens of each camera; [0013] a second mapping module, being configured to calculate a second mapping table corresponding to the lens of each camera according to the external parameters of the lens of each camera and the distance from the centers of circles formed by the optical centers of the lens of the multiple cameras to a preset plane above or below the circles, and map the multiple first spherical images respectively corresponding to the lens of each camera to a unified unit sphere according to the second mapping table corresponding to the lens of each camera, and splice the first spherical images to obtain a panoramic spherical image.” ¶0051, “the maximum distance of the preset distance range is infinity, and the minimum distance is determined by the vertical viewing angle of the lens. That is, the object smaller than the distance is outside the field of view of the lens. The minimum distance is r.Math.arc tan(2/(α−π)), and α is greater than 180°. [0052] S1012: mapping multiple first spherical images respectively corresponding to the lens of each camera to the unified unit sphere according to the second mapping table corresponding to the lens of each camera to obtain multiple unified unit spherical images respectively corresponding to the lens of each camera.” ¶0053, “[0053] For example, if 20 different distances are trialed, then 20 unified unit spherical images respectively corresponding to the lens of each camera are obtained. [0054] S1013. for each of the unified unit spherical images corresponding to the lens of each camera, projecting a part of the upper hemisphere of the unified unit spherical image to a plane which is at the selected distance from the centers of circles formed by the optical centers of the lens of the multiple cameras and above the circles according to the plane projection mode so as to obtain top projection pictures, and projecting a part of the lower hemisphere of the unified unit spherical image to a plane which is at the selected distance from the centers of circles formed by the optical centers of the lens of the multiple cameras and below the circles according to the plane projection mode so as to obtain bottom projection pictures. [0055] S1014: calculating an overall alignment error of the top projection pictures or the bottom projection pictures; and taking the distance corresponding to the minimum overall alignment error of the top projection pictures or the bottom projection pictures as the distance from the centers of circles formed by the optical centers of the lens of the multiple cameras to a preset plane above or below the circles.” ¶0064, “the unified unit sphere is specifically a unit sphere with the centers of circles formed by the optical centers of the lens of the multiple cameras serving as the center of sphere.” ¶0068, “coordinates p.sub.i.sup.s of the pixel on the unified unit sphere with the centers of circles formed by the optical centers of the lens of the multiple cameras serving as the center of sphere are obtained by normalization.”).
Zhang and Su are considered to be analogous art because all pertain to image processing. It would have been obvious before the effective filing date of the claimed invention to have modified Zhang with the features of “wherein the compositing image unit sphere unifies the plurality of camera unit spheres, wherein a compositing unit sphere centre of the compositing unit sphere is equally distanced by a unified offset (β) to each camera unit sphere centre of the plurality of camera unit spheres;” as taught by Su. The suggestion/motivation would have been in order to obtain more natural and realistic picture spliced at the top and bottom (Su, Abstract).
As to claim 2, claim 1 is incorporated and the combination of Zhang and Su discloses wherein subsequently projecting the scene onto a plurality of camera unit spheres and a compositing unit sphere comprises:
transforming the points of the scene from image coordinates into camera coordinates (Zhang, ¶0040, EQ2. ¶0044.); and
transforming the points of the scene from the camera coordinates into extrinsic coordinates, wherein the compositing unit sphere centre defines a coordinate system centre of the camera coordinate system (Zhang, ¶0066, “The extrinsic calibrator may be used to estimate camera extrinsic parameters. Extrinsic parameters are parameters used to describe the transformation between the camera and its external world.” Su, ¶0064, “the unified unit sphere is specifically a unit sphere with the centers of circles formed by the optical centers of the lens of the multiple cameras serving as the center of sphere.”).
As to claim 3, claim 1 is incorporated and the combination of Zhang and Su discloses before subsequently projecting the scene onto a plurality of camera unit spheres and a compositing unit sphere, the method comprises:
acquiring the alignment distance; determining a common origin for the compositing unit sphere where extrinsic distances amongst cameras in the plurality of cameras are used; determining the unified offset (β) using the alignment distance and common origin for the compositing unit sphere (SU, [0055] S1014: calculating an overall alignment error of the top projection pictures or the bottom projection pictures; and taking the distance corresponding to the minimum overall alignment error of the top projection pictures or the bottom projection pictures as the distance from the centers of circles formed by the optical centers of the lens of the multiple cameras to a preset plane above or below the circles.” ¶0069, “only the alignment condition between the lens of the camera and the lens of two cameras adjacent thereto are considered during the optimization.”).
As to claim 4, claim 3 is incorporated and the combination of Zhang and Su discloses
wherein the alignment distance is input by a user (Zhang, ¶0046, “To further correct the imperfect alignment of the cameras, a general camera pose correction algorithm may be applied to images from each camera to rotate each camera so that the Y axis of each camera is perpendicular to the camera optical center plane, and the Z axis of each camera is perpendicular to the camera circle.” Su, ¶0049, “m is an integer greater than or equal to 2, m may be preset or may be determined by the user according to actual situations”);
wherein the extrinsic distance is known from the properties of the camera system (Zhang, ¶0025, “Extrinsic parameters are parameters used to describe the transformation between the camera and its external world. To determine the extrinsic parameters, a number of reference cameras may be selected, and a stereo calibration may be performed between each camera pair. A stereo calibration may then be performed between the reference cameras to complete a full camera circle extrinsic calibration.”).
As to claim 5, claim 1 is incorporated and the combination of Zhang and Su discloses
wherein each camera unit sphere of the plurality of camera unit spheres are each represented by a camera model (Zhang, ¶0039, “a generic projection model may be used to approximate the fisheye projection. In examples, the projection model may be a Taylor projection model to approximate the fisheye projection”).
As to claim 6, claim 1 is incorporated and the combination of Zhang and Su discloses
wherein the camera model comprises a pinhole camera model, a unified camera model, an extended unified camera model, a Kannala-Brandt camera model, a field-of-view camera model or a double sphere camera model (Zhang, ¶0037, “A spherical camera with a 220-degree FOV lens can capture more light rays than a pinhole camera model with regular lenses.” Su, ¶0013, “corresponding to the lens of each camera to a unified unit sphere”).
As to claim 7, claim 2 is incorporated and the combination of Zhang and Su discloses
wherein the alignment distance relates to an extrinsic distance between the coordinate system centre of the camera coordinate system and a point of interest, where parallax is minimized (Su, ¶0023, “not only the alignment error caused by parallax is solved, but a simple calculation process is still kept”).
As to claim 8, claim 1 is incorporated and the combination of Zhang and Su discloses
wherein the composite image is a panorama image (Su, ¶0004, “panorama camera”).
As to claim 9, claim 1 is incorporated and the combination of Zhang and Su discloses
wherein a field-of-view used from each of the plurality of cameras for the composite image is dependent on the alignment distance (Su, 0051, “the object smaller than the distance is outside the field of view of the lens. The minimum distance is r·arc tan(2/(α−π)), and α is greater than 180°.”).
As to claim 10, claim 9 is incorporated and the combination of Zhang and Su discloses
wherein a lower alignment distance leads to a bigger field-of-view used from each of the plurality of cameras for the composite image (Su, ¶0023, “a second mapping table corresponding to the lens of each camera is calculated according to the external parameters of the lens of each camera and the distance from the centers of circles formed by the optical centers of the lens of the multiple cameras to a preset plane above or below the circles, and multiple first spherical images respectively corresponding to the lens of each camera are mapped to a unified unit sphere according to the second mapping table corresponding to the lens of each camera”).
As to claim 11, claim 1 is incorporated and the combination of Zhang and Su discloses
wherein the plurality of cameras are large field-of-view cameras (Zhang, ¶0019, “use cameras with a wide field of view, such as a 220-degree FOV fisheye lens, to capture content of the scene.”).
As to claim 12, claim 11 is incorporated and the combination of Zhang and Su discloses
wherein the large field-of-view cameras comprise a field-of-view that is larger than 180 degrees (Zhang, ¶0019, “use cameras with a wide field of view, such as a 220-degree FOV fisheye lens, to capture content of the scene.”).
As to claim 13, claim 11 is incorporated and the combination of Zhang and Su discloses
wherein the large field-of-view cameras comprise a camera with a fish-eye lens (Zhang, ¶0005-0006, “FIG. 3 is an illustration of fisheye camera array design”).
As to claim 14, the combination of Zhang and Su discloses
a computer-implemented method of providing a composite image of a scene using a camera system comprising a plurality of cameras, comprising:
pre-computing a representation of the camera system, comprising a plurality of camera unit spheres and a compositing unit sphere;
subsequently projecting the scene onto a plurality of camera unit spheres and a compositing unit sphere, and;
wherein each camera unit sphere of the plurality of camera unit spheres represents one camera of the plurality of cameras, respectively;
wherein the compositing unit sphere unifies the plurality of camera unit spheres, wherein a compositing unit sphere centre of the compositing unit sphere is equally distanced by a unified offset (β) to each camera unit sphere centre of the plurality of camera unit spheres;
wherein a radius of the camera unit spheres and the compositing unit sphere corresponds to an alignment distance, wherein the alignment distance relates to an extrinsic distance between the camera system and a point of interest;
creating a composite image from individual camera images of the plurality of cameras;
applying final distortion correction or optimizing parallax correction to the composite image (See claim 1 for detailed analysis. Distortion correction and parallax correction can be seen in both Zhang and Su.).
As to claim 15, claim 14 is incorporated and the combination of Zhang and Su discloses
pre-computing the representation of the camera system comprises:
calibrating the representation of the camera system (Zhang, ¶0024, “The intrinsic calibrator 110 may calculate parameters to be applied to the fisheye camera. Intrinsic parameters may include the parameters intrinsic to the camera itself, such as the focal length and lens distortion. In some examples, a checkerboard may be used to perform intrinsic calibration.”);
performing direct alignment or feature based alignment (Zhang, ¶0046, “To further correct the imperfect alignment of the cameras, a general camera pose correction algorithm may be applied to images from each camera to rotate each camera so that the Y axis of each camera is perpendicular to the camera optical center plane, and the Z axis of each camera is perpendicular to the camera circle. In embodiments, the camera poses may be estimated using a general structure-from-motion with global bundle adjustment.” Su, ¶0053, “calculating an overall alignment error of the top projection pictures or the bottom projection pictures; and taking the distance corresponding to the minimum overall alignment error of the top projection pictures or the bottom projection pictures as the distance from the centers of circles formed by the optical centers of the lens of the multiple cameras to a preset plane above or below the circles.”).
As to claim 16, claim 14 is incorporated and the combination of Zhang and Su discloses
wherein creating the composite image from individual camera images of the plurality of cameras comprises:
finding seam lines amongst camera images that are viewing similar parts of the scene (Su, ¶0003, “analyze the overlapping areas at the seams to find out the dense matching in the overlapping areas by using an optical flow method or a feature point method, and then modify the mapping table to achieve overlapping at the seams as much as possible”); and
blending a content, i.e. a number of pixels, of each of the camera images that share a seam line (Su, ¶0003, “the pixels of the same object in different lens may be mapped to different positions in the final picture, which results in picture dislocation at seams between areas managed by different lens on the spherical surface.”).
As to claim 17, the combination of Zhang and Su discloses
an apparatus, comprising:
one or more processors executing locally stored instructions to cause the processors to perform operations, including:
subsequently projecting the scene onto a plurality of camera unit spheres and a compositing unit sphere, and;
wherein each camera unit sphere of the plurality of camera unit spheres represents one camera of the plurality of cameras, respectively;
wherein the compositing unit sphere unifies the plurality of camera unit spheres, wherein a compositing unit sphere centre of the compositing unit sphere is equally distanced by a unified offset (β) to each camera unit sphere centre of the plurality of camera unit spheres;
wherein a radius of the camera unit spheres and the compositing unit sphere corresponds to an alignment distance, wherein the alignment distance relates to an extrinsic distance between the camera system and a point of interest (See claim 1 for detailed analysis.).
As to claim 18, the combination of Zhang and Su discloses
an apparatus, comprising:
one or more processors executing locally stored instructions to cause the processors to perform operations, including:
pre-computing a representation of the camera system, comprising a plurality of camera unit spheres and a compositing unit sphere;
subsequently projecting the scene onto a plurality of camera unit spheres and a compositing unit sphere, and:
wherein each camera unit sphere of the plurality of camera unit spheres represents one camera of the plurality of cameras, respectively:
wherein the compositing unit sphere unifies the plurality of camera unit spheres, wherein a compositing unit sphere centre of the compositing unit sphere is equally distanced by a unified offset (β) to each camera unit sphere centre of the plurality of camera unit spheres;
wherein a radius of the camera unit spheres and the compositing unit sphere corresponds to an alignment distance, wherein the alignment distance relates to an extrinsic distance between the camera system and a point of interest:
creating a composite image from individual camera images of the plurality of cameras;
applying final distortion correction or optimizing parallax correction to the composite image (See claim 1 for detailed analysis.).
As to claim 19, the combination of Zhang and Su discloses
a non-volatile computer readable media comprising instructions which, when executed by at least one processor, causes the at least one processor to;
subsequently projecting the scene onto a plurality of camera unit spheres and a compositing unit sphere, and;
wherein each camera unit sphere of the plurality of camera unit spheres represents one camera of the plurality of cameras, respectively:
wherein the compositing unit sphere unifies the plurality of camera unit spheres, wherein a compositing unit sphere centre of the compositing unit sphere is equally distanced by a unified offset (β) to each camera unit sphere centre of the plurality of camera unit spheres:
wherein a radius of the camera unit spheres and the compositing unit sphere corresponds to an alignment distance, wherein the alignment distance relates to an extrinsic distance between the camera system and a point of interest (See claim 1 for detailed analysis.).
As to claim 20, the combination of Zhang and Su discloses
a non-volatile computer readable media comprising instruction which, when executed by at least one processor causes the at least one processor to:
pre-compute a representation of the camera system, comprising a plurality of camera unit spheres and a compositing unit sphere;
subsequently project the scene onto a plurality of camera unit spheres and a compositing unit sphere, and;
wherein each camera unit sphere of the plurality of camera unit spheres represents one camera of the plurality of cameras, respectively;
wherein the compositing unit sphere unifies the plurality of camera unit spheres, wherein a compositing unit sphere centre of the compositing unit sphere is equally distanced by a unified offset (β) to each camera unit sphere centre of the plurality of camera unit spheres;
wherein a radius of the camera unit spheres and the compositing unit sphere corresponds to an alignment distance, wherein the alignment distance relates to an extrinsic distance between the camera system and a point of interest;
create a composite image from individual camera images of the plurality of cameras:
apply final distortion correction or optimizing parallax correction to the composite image (See claim 1 for detailed analysis.).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Rose et al. (US Pub 2019/0394484 A10 discloses predictive coding of spherical or 360-degree video with dynamics dominated by camera motion.
Long, et al. (US Patent 9,998,664 B1) discloses generating multi-resolution spherical videos, comprising receiving a plurality of video recordings recorded by a viewpoint, wherein the viewpoint is a camera array comprising a plurality of cameras arranged around a viewpoint center; and generating a spherical video from the plurality of video recordings by non-concentric spherical projection, wherein each of the plurality of video recordings is mapped to respective portions of a projection sphere, wherein a center of the projection sphere is positioned at a non-zero distance away from the viewpoint center, and wherein a visual resolution of the spherical video in a first portion of the projection sphere is higher than a visual resolution of the spherical video in a second portion of the projection sphere.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YU CHEN whose telephone number is (571)270-7951. The examiner can normally be reached on M-F 8-5 PST Mid-day flex.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached on 571-272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YU CHEN/Primary Examiner, Art Unit 2613