DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Priority is acknowledged from Provisional application PRO 63/603,445 with a filing date of 11/28/2023.
Drawings
The 5-page drawings have been considered and placed on record in the file.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more, and the claimed invention is directed to non-statutory subject matter as follows. The claims recite identifying feature matches across images, determining a set of homographies based on the matches, performing uncalibrated rectification on the images based on the homographies, and performing depth estimation based on the rectified image.
Step 1:
With regard to Step 1, the instant claims are directed to a method, which is among the statutory categories of invention.
Step 2A – Prong 1:
With regard to Step 2A – Prong 1, for example in Claim 1, the limitations of "A method for depth estimation, the method comprising: identifying feature matches across a plurality of images; determining a set of homographies based on the identified feature matches…; and performing depth estimation based on at least the rectified image", as drafted only involves mental processes or mathematical calculations, such as the identification of matching features and estimation of depth of an image. That is, nothing in the above-described claim elements preclude the steps from practically being performed in the mind or on a piece of paper. If a claim limitation, under its broadest reasonably interpretation covers performance of the limitation in the mind or through mathematical calculations, but for the recitation of a generic apparatus components, such as a processor, computer program, or machine-readable media, then it falls within the "mental processes", which include concepts performed in the human mind, including an observation, evaluation, judgement, opinion, or mathematical calculations groupings of the abstract idea. Accordingly, the claim recites an abstract idea.
Step 2A – Prong 2:
The 2019 PEG defines the phrase “integration into a practical application” to require an additional element or a combination of additional elements in the claim to apply, rely on, or use the judicial exception. In the instant case, the additional elements in the claims do not apply, rely on, or use the judicial exception.
This judicial exception is not integrated into a practical application because the claim only recites the following additional steps "performing an uncalibrated rectification on a first image of the plurality of images based on the set of determined homographies", i.e., applying the abstract idea on a generic computer and lacking a technological improvement. The other additional recited element in certain other claims is just a processor and a computer-readable storage medium, which are generic computer components. Accordingly, these additional elements do not integrate the abstract idea into a practical application because it is a field-of-use limitation that does not impose any meaningful limits on practicing the abstract idea. Therefore, the claim as a whole, recites an abstract idea.
Step 2B:
Because the claim fails under Step 2A, the claims are further evaluated under Step 2B. The claim herein does not include additional steps that are sufficient to amount to significantly more than the judicial exception because as discussed above with respect to integration of the abstract idea into practical application, the additional elements/steps amount to no more than insignificant extra-solution activities. Mere instructions to apply an exception using generic apparatus component, such as a processor, cannot provide an inventive concept. The claim is not patent eligible. It should be noted that a similar analysis may be performed with respect to independent Claims 14 and 20.
Further, with regard to dependent Claims 2-13 and 15-19 viewed individually, these additional steps are under their broadest reasonable interpretation, cover performance of the limitation in the mind and do not provide meaningful limitations to transform the abstract idea into a patent eligible application of the abstract idea such that the claims limitations amount to significantly more than the abstract idea itself. For example, identifying feature matches comprises filtering candidate matches as recited in Claim 4 or depth estimation generates a disparity map as recited in Claim 9 are only examples of routine and conventional image processing steps or steps that could be completed within the human mind and do not amount to significantly more to consider as inventive steps. Accordingly, Claims 1-20 are rejected under 35 U.S.C. 101.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 4-5, 9-10, 14 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Hsiao et al. (US 20230274399 A1) in view of Kim et al. (US 20140118501 A1) and Sadeghi et al. (US 20220101549 A1).
Regarding Claim 1, Hsiao teaches "A method for depth estimation, the method comprising: identifying feature matches across a plurality of images"; (Hsiao, FIG. 7 and Paras. 64 and 73, teaches detecting feature points in the target frame and the reference frame and identify matching pairs, i.e., identify feature matches across a plurality of images);
"determining a set of homographies based on the identified feature matches"; (Hsiao, Paras. 68-70, teaches a set of candidate homographies can be computed between the target frame and the reference frame and the homography that minimizes or reduces the resulting alignment error between the matching pairs of feature points can be utilized as the homography, i.e., determine a set of homographies based on the feature matches).
However, Hsiao does not explicitly teach "performing an uncalibrated rectification on a first image of the plurality of images based on the set of determined homographies; and performing depth estimation based on at least the rectified image”.
In an analogous field of endeavor, Kim teaches "performing an uncalibrated rectification on a first image of the plurality of images based on the set of determined homographies"; (Kim, Paras. 30-31, teaches a first rectification image generator which generates a rectification stereo image without using information about the camera parameter wherein the first rectification image generator extracts feature points of left and right images from the color-calibrated stereo image, generates a fundamental matrix and a rectification homography, and acquires the rectification stereo image through an image conversion operation in which the scheme is called uncalibrated rectification, i.e., uncalibrated rectification is performed on the image based on the homography).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Hsiao wherein the homography includes a set of homographies by including the uncalibrated rectification on an image based on the homography taught by Kim. One of ordinary skill in the art would be motivated to combine the references since it enhances a distortion calibration result (Kim, Para. 3, teaches the motivation of combination to be to enhance a distortion calibration result of the stereo image).
However, the combination of references of Hsiao in view of Kim does not explicitly teach "and performing depth estimation based on at least the rectified image".
In an analogous field of endeavor, Sadeghi teaches "and performing depth estimation based on at least the rectified image"; (Sadeghi, Paras. 69-70 and 165-166, teaches a depth estimator which receives rectified stereo image pairs to compute a depth estimate in the form of a depth image, i.e., perform depth estimation based on at least the rectified image).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Hsiao and Kim by including the depth estimation based on the rectified image taught by Sadeghi. One of ordinary skill in the art would be motivated to combine the references since it accounts for misalignment (Sadeghi, Para. 166, teaches the motivation of combination to be to account for any misalignment of the image capture units).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date.
Regarding Claim 4, the combination of references of Hsiao in view of Kim and Sadeghi teaches "The method of claim 1, wherein identifying feature matches comprises filtering a set of candidate feature matches to identify the feature matches"; (Hsiao, FIG. 7 and Paras. 64-66, teaches potential feature correspondences between the target frame and the reference frame are identified using an appropriate method in which outliers are discarded by estimating fundamental matrices using random sample consensus, i.e., identifying feature matches comprises filtering candidate matches being the discarding of outliers).
Regarding Claim 5, the combination of references of Hsiao in view of Kim and Sadeghi teaches "The method of claim 4, wherein filtering the set of candidate feature matches is based on a confidence level associated with each candidate feature match"; (Hsiao, FIG. 7 and Paras. 40, 64-66, and 74, teaches potential feature correspondences between the target frame and the reference frame are identified using an appropriate method in which outliers are discarded by estimating fundamental matrices using random sample consensus in which a matching score is calculated and compared to a threshold and wherein identified corresponding pixels have a correlation above a threshold, i.e., filtering the matches is based on confidence level associated with the match being the matching score).
Regarding Claim 9, the combination of references of Hsiao in view of Kim and Sadeghi teaches "The method of claim 1, wherein performing depth estimation comprises generating a disparity map"; (Sadeghi, Paras. 69-70 and 165-166, teaches a depth estimator which receives rectified stereo image pairs to compute a depth estimate in the form of a depth image wherein the depth estimate extracted is in the form of a disparity map, i.e., depth estimation comprises generating a disparity map).
The proposed combination as well as the motivation for combining the Hsiao in view of Kim and Sadeghi references presented in the rejection of Claim 1, applies to claim 9. Thus, the method recited in claim 9 is met by Hsiao in view of Kim and Sadeghi.
Regarding Claim 10, the combination of references of Hsiao in view of Kim and Sadeghi teaches "The method of claim 1, wherein performing depth estimation comprises generating a depth map"; (Sadeghi, Para. 69, teaches each depth estimate may be in the form of a depth map).
The proposed combination as well as the motivation for combining the Hsiao in view of Kim and Sadeghi references presented in the rejection of Claim 1, applies to claim 10. Thus, the method recited in claim 10 is met by Hsiao in view of Kim and Sadeghi.
Claim 14 recites a system with elements corresponding to the steps recited in Claim 1. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Hsiao in view of Kim and Sadeghi references, presented in rejection of Claim 1, apply to this claim. Finally, the combination of the Hsiao in view of Kim and Sadeghi references discloses a processor and memory (for example, see Hsiao, Paragraph 13).
Claim 16 recites a system with elements corresponding to the steps recited in Claims 4-5. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Hsiao in view of Kim and Sadeghi references, presented in rejection of Claims 4-5, apply to this claim. Finally, the combination of the Hsiao in view of Kim and Sadeghi references discloses a processor and memory (for example, see Hsiao, Paragraph 13).
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Hsiao in view of Kim, Sadeghi, and Sangappa et al. (US 20250259417 A1).
Regarding Claim 2, the combination of references of Hsiao in view of Kim and Sadeghi does not explicitly teach "The method of claim 1, wherein identifying feature matches comprises utilizing Local Feature Matching with Transformers".
In an analogous field of endeavor, Sangappa teaches "The method of claim 1, wherein identifying feature matches comprises utilizing Local Feature Matching with Transformers"; (Sangappa, Claims 1 and 6-7, teaches detecting a match between the reference image and the query image wherein comparing the images comprises aligning the cropped images according to a Local Feature Matching with Transformers ML model).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Hsiao in view of Kim and Sadeghi by including the identification of features using Local feature matching with transformers taught by Sangappa. One of ordinary skill in the art would be motivated to combine the references since it determines authenticity of objects (Sangappa, Abstract, teaches the motivation of combination to be to determine authenticity of the objects in the images based on the detected match).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date.
Claims 3 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Hsiao in view of Kim, Sadeghi, and Karvounis (US 20150304634 A1).
Regarding Claim 3, the combination of references of Hsiao in view of Kim and Sadeghi does not explicitly teach "The method of claim 1, wherein identifying feature matches comprises identifying sub-pixel accurate feature matches".
In an analogous field of endeavor, Karvounis teaches "The method of claim 1, wherein identifying feature matches comprises identifying sub-pixel accurate feature matches"; (Karvounis, Claim 7, teaches identifying the candidate feature comprises refining a location of the left feature to a sub-pixel accuracy, refining a location of the right feature to a sub-pixel accuracy, and matching the left and right features to create a stereo feature, i.e., identifying feature matches comprises identifying sub-pixel accurate feature matches).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Hsiao in view of Kim and Sadeghi by including the identification of sub-pixel accurate feature matches taught by Karvounis. One of ordinary skill in the art would be motivated to combine the references since it improves maps (Karvounis, Para. 68, teaches the motivation of combination to be to improve and produce maps).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date.
Claim 15 recites a system with elements corresponding to the steps recited in Claim 3. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Hsiao in view of Kim, Sadeghi, and Karvounis references, presented in rejection of Claim 3 apply to this claim. Finally, the combination of the Hsiao in view of Kim, Sadeghi, and Karvounis references discloses a processor and memory (for example, see Hsiao, Paragraph 13).
Claims 6-7 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Hsiao in view of Kim, Sadeghi, and Zhang et al. (US 6606404 B1).
Regarding Claim 6, the combination of references of Hsiao in view of Kim and Sadeghi does not explicitly teach "The method of claim 1, wherein the set of determined homographies comprises at least one selected from the group consisting of a vertical alignment homography, a global distortion minimization homography, and a horizontal alignment homography".
In an analogous field of endeavor, Zhang teaches "The method of claim 1, wherein the set of determined homographies comprises at least one selected from the group consisting of a vertical alignment homography, a global distortion minimization homography, and a horizontal alignment homography"; (Zhang, Cols. 10 and 13-14 and Claims 14, 20, and 36, teaches a shearing transform Hs is obtained by scaling and translating the images in a horizontal direction while preserving resolution and reducing distortion, similarity transform Hr is computed by rotating and translating images such that the epipolar lines are horizontally aligned, and the special projective transform Hp which map the epipoles to points at infinity wherein if the weights assigned to the points are identical then there is no projective distortion and the homography is necessarily an affine transform which is the basis of the distortion minimization and wherein the assigning of weights to pixel elements defining the images and minimizes variations of the weights over all pixels such that the transformation is near affine, i.e., set of homographies include vertical alignment by the necessarily horizontally aligned lines, global distortion minimization being the special projective transform minimizing distortion across all pixels, and horizontal alignment being the shearing transform scaling and translating images in a horizontal direction).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Hsiao in view of Kim and Sadeghi wherein rectification is uncalibrated by including the homographies include alignment and distortion minimization taught by Zhang. One of ordinary skill in the art would be motivated to combine the references since it optimizes the reduction of distortion (Zhang, Abstract, teaches the motivation of combination to be to optimize the reduction of distortion).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date.
Regarding Claim 7, the combination of references of Hsiao in view of Kim, Sadeghi, and Zhang teaches "The method of claim 6, wherein performing the uncalibrated rectification comprises applying a combination of the vertical alignment homography, the global distortion minimization homography, and the horizontal alignment homography"; (Zhang, Col. 14, teaches the combined transform of Hs, Hr, and Hp rectify the images with minimal distortion, i.e., rectification comprises applying a combination of the three homographies).
The proposed combination as well as the motivation for combining the Hsiao in view of Kim, Sadeghi, and Zhang references presented in the rejection of Claim 6, applies to claim 7. Thus, the method recited in claim 7 is met by Hsiao in view of Kim, Sadeghi, and Zhang.
Claim 17 recites a system with elements corresponding to the steps recited in Claims 6-7. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Hsiao in view of Kim, Sadeghi, and Zhang references, presented in rejection of Claims 6-7, apply to this claim. Finally, the combination of the Hsiao in view of Kim, Sadeghi, and Zhang references discloses a processor and memory (for example, see Hsiao, Paragraph 13).
Claims 8 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Hsiao in view of Kim, Sadeghi, Zhang, and Ando (US 20040070667 A1).
Regarding Claim 8, the combination of references of Hsiao in view of Kim, Sadeghi, and Zhang does not explicitly teach "The method of claim 6, wherein determining the set of homographies comprises determining whether to utilize a horizontal alignment homography based on an angle change between the first image and a second image of the plurality of images".
In an analogous field of endeavor, Ando teaches "The method of claim 6, wherein determining the set of homographies comprises determining whether to utilize a horizontal alignment homography based on an angle change between the first image and a second image of the plurality of images"; (Ando, Abstract and Para. 34 and Table 1, teaches a horizontal shift distance of the center of an image is made with respect to a change in the convergence angle between the left and right images wherein at a certain convergence angle a horizontal shift length may be zero and at a higher angle the horizontal shift length increases, i.e., determine whether to utilize a horizontal alignment homography being the horizontal translate based on the angle between the images).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Hsiao in view of Kim, Sadeghi, and Zhang by including the determination whether to use an alignment homography based on an angle between the images taught by Ando. One of ordinary skill in the art would be motivated to combine the references since it improves the stereoscopic effects of the image (Ando, Para. 46, teaches the motivation of combination to be to improve the stereoscopic effects of the image).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date.
Claim 18 recites a system with elements corresponding to the steps recited in Claim 8. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Hsiao in view of Kim, Sadeghi, Zhang, and Ando references, presented in rejection of Claim 8 apply to this claim. Finally, the combination of the Hsiao in view of Kim, Sadeghi, Zhang, and Ando references discloses a processor and memory (for example, see Hsiao, Paragraph 13).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Hsiao in view of Kim, Sadeghi, and Wu et al. (US 20160188994 A1).
Regarding Claim 11, the combination of references of Hsiao in view of Kim and Sadeghi does not explicitly teach "The method of claim 1 further comprising determining extrinsics of a set of cameras associated with the plurality of images based on the identified feature matches".
In an analogous field of endeavor, Wu teaches "The method of claim 1 further comprising determining extrinsics of a set of cameras associated with the plurality of images based on the identified feature matches"; (Wu, Para. 3, teaches feature matching is used in camera calibration to set the proper position or orientation for each camera in the array, i.e., determining extrinsics of the cameras associated with the images based on the feature matches).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Hsiao in view of Kim and Sadeghi by including the determination of camera extrinsics based on feature matches taught by Wu. One of ordinary skill in the art would be motivated to combine the references since it sets accurate final point locations (Wu, Para. 3, teaches the motivation of combination to be to set an accurate final point location).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date.
Claims 12-13 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Hsiao in view of Kim, Sadeghi, Wu, and Heinzle et al. (US 20120182397 A1).
Regarding Claim 12, the combination of references of Hsiao in view of Kim, Sadeghi, and Wu does not explicitly teach "The method of claim 11, wherein determining extrinsics comprises computing epipolar constraints based on the identified feature matches".
In an analogous field of endeavor, Heinzle teaches "The method of claim 11, wherein determining extrinsics comprises computing epipolar constraints based on the identified feature matches"; (Heinzle, Paras. 83-84, teaches rectifying homographies are estimated from the epipolar geometry in real-time using feature matching wherein image correspondences are used to estimate the fundamental matrix from which corresponding homographies are computed, i.e., determining the extrinsics comprises computing epipolar constraints being the epipolar geometry with fundamental matrix based on the feature matching).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Hsiao in view of Kim, Sadeghi, and Wu wherein rectification is uncalibrated by including the determination of extrinsics comprises computing epipolar constraints based on feature matches taught by Heinzle. One of ordinary skill in the art would be motivated to combine the references since it improves stability and accuracy (Heinzle, Para. 90, teaches the motivation of combination to be to improve stability and accuracy of the estimate).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date.
Regarding Claim 13, the combination of references of Hsiao in view of Kim, Sadeghi, Wu, and Heinzle teaches "The method of claim 11, wherein determining extrinsics of the set of cameras is performed in parallel with determining the set of homographies and performing an uncalibrated rectification"; (Heinzle, Paras. 27, 66, and 83-84, teaches the stream analyze configuration containing disparity calculations, image rectification, feature matching, alignment estimation, and homography estimation wherein the control module allows for the processing plug-ins to operate in parallel, i.e., determining extrinsics is performed in parallel with determining homographies and rectification).
The proposed combination as well as the motivation for combining the Hsiao in view of Kim, Sadeghi, Wu, and Heinzle references presented in the rejection of Claim 12, applies to claim 13. Thus, the method recited in claim 13 is met by Hsiao in view of Kim, Sadeghi, Wu, and Heinzle.
Claim 19 recites a system with elements corresponding to the steps recited in Claims 11-13. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Hsiao in view of Kim, Sadeghi, Wu, and Heinzle references, presented in rejection of Claims 11-13, apply to this claim. Finally, the combination of the Hsiao in view of Kim, Sadeghi, Wu, and Heinzle references discloses a processor and memory (for example, see Hsiao, Paragraph 13).
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Hsiao in view of Choi et al. (US 20160234473 A1), Zhang, Chan et al. (US 20180150939 A1), Liu et al. (US 20130300820 A1), and Sadeghi.
Regarding Claim 20, Hsiao teaches "A method for depth estimation, the method comprising: identifying feature matches across a plurality of images"; (Hsiao, FIG. 7 and Paras. 64 and 73, teaches detecting feature points in the target frame and the reference frame and identify matching pairs, i.e., identify feature matches across a plurality of images).
However, Hsiao does not explicitly teach “determining a vertical alignment homography based on the identified feature matches; performing a first transformation on at least one of the plurality of images based on the determined vertical alignment homography to generate a first transformed image; determining a global distortion minimization homography based on feature matches identified based on the first transformed image; performing a second transformation on the first transformed image based on the determined global distortion minimization homography to generate a second transformed image; determining a horizontal alignment homography based on feature matches identified based on the second transformed image; performing a third transformation on the second transformed based on the determined horizontal alignment homography to generate a rectified image; and performing depth estimation based on at least the rectified image”.
In an analogous field of endeavor, Choi teaches "determining a vertical alignment homography based on the identified feature matches"; (Choi, Claims 8-9 and Pars. 47 and 51-52, teaches the image rectification method includes calculating a homography matrix from among the plurality of images based on the corresponding pairs of points wherein the corresponding pairs of points are extracted from the images by matching the feature points and wherein the image rectification rectifies the image so that the vertical position of the points on the epipolar line are identical to the vertical position of the point x, i.e., homography determined for vertical alignment based on identified feature matches).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Hsiao by including the vertical alignment homography based on feature matches taught by Choi. One of ordinary skill in the art would be motivated to combine the references since it improves visual perception (Choi, Para. 23, teaches the motivation of combination to be to improve visual perception).
However, the combination of references of Hsiao in view of Choi does not explicitly teach “performing a first transformation on at least one of the plurality of images based on the determined vertical alignment homography to generate a first transformed image; determining a global distortion minimization homography based on feature matches identified based on the first transformed image; performing a second transformation on the first transformed image based on the determined global distortion minimization homography to generate a second transformed image; determining a horizontal alignment homography based on feature matches identified based on the second transformed image; performing a third transformation on the second transformed based on the determined horizontal alignment homography to generate a rectified image; and performing depth estimation based on at least the rectified image”.
In an analogous field of endeavor, Zhang teaches "performing a first transformation on at least one of the plurality of images based on the determined vertical alignment homography to generate a first transformed image";(Zhang, FIG. 10 and Cols. 10 and 13-14 and Claims 14, 20, and 36, teaches a similarity transform Hr is computed by rotating and translating images such that the epipolar lines are horizontally aligned wherein Figure 10 shows the image pair transformed by the similarity transform and a special projective transform Hp which map the epipoles to points at infinity so that the epipolar lines become parallel wherein if the weights assigned to the points are identical then there is no projective distortion and the homography is necessarily an affine transform which is the basis of the distortion minimization and wherein the assigning of weights to pixel elements defining the images and minimizes variations of the weights over all pixels such that the transformation is near affine in which Figure 9 shows the image pair transformed by the specialized projective mapping, i.e., performing a first transformation on the images based on vertical alignment by the combination of the special projective transform making epipolar lines parallel and the similarity transform computing rotation and translation);
"
"performing a second transformation on the first transformed image based on the determined global distortion minimization homography to generate a second transformed image";(Zhang, FIG. 11 and Cols. 10 and 13-14 and Claims 14, 20, and 36, teaches a shearing transform Hs is obtained by scaling and translating the images in a horizontal direction while preserving resolution and reducing distortion in which Figure 11 shows the image after shearing transform, i.e., second transformation being the shearing transform on the previous transformed image to determine global distortion minimization homography for the second generated image);
"
"performing a third transformation on the second transformed based on the determined horizontal alignment homography to generate a rectified image";(Zhang, Cols. 10 and 13-14, teaches additional uniform scaling and translation may be applied after the combined transform of Hs, Hr, and Hp are applied with minimal distortion in which translations are computed in u so that the minimal pixel coordinate has a u-coordinate of 0 and wherein a similar translation is found for the v direction, but the minimum is taken over both images to preserve rectification, i.e., third transformation being the additional uniform scaling and translation in the u direction is the horizontal alignment homography to generate a rectified image).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Hsiao in view of Choi by including a first, second, and third transformation based on determined homographies taught by Zhang. One of ordinary skill in the art would be motivated to combine the references since it optimizes the reduction of distortion (Zhang, Abstract, teaches the motivation of combination to be to optimize the reduction of distortion).
However, the combination of references of Hsiao in view of Choi and Zhang does not explicitly teach “determining a global distortion minimization homography based on feature matches identified based on the first transformed image; determining a horizontal alignment homography based on feature matches identified based on the second transformed image; and performing depth estimation based on at least the rectified image”.
In an analogous field of endeavor, Chan teaches "determining a global distortion minimization homography based on feature matches identified based on the first transformed image"; (Chan, Abstract, teaches matching feature points for two images, selecting at least five most suitable feature point sets, and calculating a most suitable radial distortion homography between the images according to the suitable feature point sets, i.e., determine global distortion minimization based on identified feature matches).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Hsiao, Choi, and Zhang wherein the image is a first transformed image by including the determination of a global distortion minimization homography based on matched features of the image taught by Chan. One of ordinary skill in the art would be motivated to combine the references since it improves the quality of the images (Chan, Para. 37, teaches the motivation of combination to be to improve quality of images after image fusion).
However, the combination of references of Hsiao in view of Choi, Zhang, and Chan does not explicitly teach “determining a horizontal alignment homography based on feature matches identified based on the second transformed image; and performing depth estimation based on at least the rectified image”.
In an analogous field of endeavor, Liu teaches "determining a horizontal alignment homography based on feature matches identified based on the second transformed image"; (Liu, Para. 122, teaches a match relationship is found between feature points and the homography matrix between the images is determined in which the two images may be combined together through coordinate conversion, and pixels of the overlapping zone are aligned wherein the model has rotation in the horizontal direction, i.e., horizontal alignment homography based on feature matches).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Hsiao in view of Choi, Zhang, and Chan wherein the image is a second transformed image by including the horizontal alignment homography based on matched features of the image taught by Liu. One of ordinary skill in the art would be motivated to combine the references since it aligns images for seamless panoramic images (Liu, Para. 122-123, teaches the motivation of combination to be to align images for a seamless panoramic image).
However, the combination of references of Hsiao in view of Choi, Zhang, Chan, and Liu does not explicitly teach "and performing depth estimation based on at least the rectified image".
In an analogous field of endeavor, Sadeghi teaches "and performing depth estimation based on at least the rectified image"; (Sadeghi, Paras. 69-70 and 165-166, teaches a depth estimator which receives rectified stereo image pairs to compute a depth estimate in the form of a depth image, i.e., perform depth estimation based on at least the rectified image).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Hsiao, Choi, Zhang, Chan, and Liu by including the depth estimation based on the rectified image taught by Sadeghi. One of ordinary skill in the art would be motivated to combine the references since it accounts for misalignment (Sadeghi, Para. 166, teaches the motivation of combination to be to account for any misalignment of the image capture units).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW STEVEN BUDISALICH whose telephone number is (703)756-5568. The examiner can normally be reached Monday - Friday 8:30am-5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571) 272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW S BUDISALICH/Examiner, Art Unit 2662
/AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662
Prosecution Insights