DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description: reference character 114 present in FIG. 1 is not present in the specification, paragraph [0031] describes keypoint depth but is absent reference character 114, reference character 114 may be added to this paragraph to overcome this issue. Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
CLAIM INTERPRETATION
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
Claims 1-20 have been interpreted under 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph) to not invoke 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph) claim interpretation.
Claim Objections
Claims 14 and 15 are objected to because of the following informalities:
Dependent claim 14 claims “based at least in part on the set of 2-D keypoints” while parent claim 13 claims “a set of keypoints for the plurality of image frames” and follows with claiming “the set of keypoints”, thus, in dependent claim 14 claim “the set of 2-D keypoints” lacks antecedent basis in the claims. In claim 14 “the set of 2-D keypoints” should be “the set of keypoints”. Dependent claim 15 inherits and does not correct this issue. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 at line 10 and claim 13 at line 8 each claim “determine a corresponding set of depth priors;”. This claim limitation does not claim a subject corresponding to the “set of depth priors”. Therefore, the claimed “determine a corresponding set of depth priors;” is unclear as to the metes and bounds of the claimed “set of depth priors”.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 6-20 of U.S. Patent No. 12,046,002. Although the claims at issue are not identical, they are not patentably distinct from each other because the pending claims are broader versions of the patented claims. Refer to the following table which correlates the pending claims with the patented claims and to the following table which provides a side by side comparison of pending system claims 1-12, pending non-transitory computer-readable storage medium claims 13-20, patented system claims 6-13, and patented non-transitory computer-readable storage medium claims 14-20.
Pending system claims
1
2
3
4
5
6
Patented system claims
6
6
6
6
6
7
Pending system claims
7
8
9
10
11
12
Patented system claims
8
9
10
11
12
13
Pending non-transitory computer-readable storage medium claims
13
14
15
16
17
18
Patented non-transitory computer-readable storage medium claims
14
14
14
14
15
16
Pending non-transitory computer-readable storage medium claims
19
20
Patented non-transitory computer-readable storage medium claims
18
19
Claims filed on 06/20/2024,
System claims 1-12
1. A system, comprising:
one or more processors; and
memory storing executable instructions that, as a result of execution by the one or more processors, cause the system to:
obtain a plurality of image frames from a digital content item that corresponds to a scene;
determine, based at least in part on a correspondence search, a set of keypoints for the plurality of image frames;
determine a dense depth map for one or more of the plurality of image frames;
determine a corresponding set of depth priors;
perform an initialization based at least in part on the set of keypoints and the corresponding set of depth priors to determine an initial image frame pair
determine a camera pose based at least in part on the initial image pair; and
perform a depth-constrained optimization to incrementally update the camera pose.
2. The system of claim 1, wherein the executable instructions include further instructions that, as a result of execution by the one or more processors, further cause the system to:
obtain a depth estimator, wherein the dense depth map is determined based on the depth estimator.
3. The system of claim 1,
wherein the set of depth priors is based on the set of keypoints and the dense map.
4. The system of claim 1,
wherein the depth-constrained optimization is based at least in part on the set of keypoints and the set of depth priors.
5. The system of claim 1,
wherein the set of keypoints include a set of 2-D keypoints.
6. The system of claim 1,
wherein the instructions to perform the initialization based at least in part on the set of keypoints and the corresponding set of depth priors to determine an initial image pair include instructions that, as a result of execution by the one or more processors, cause the system to:
determine, for the plurality of image frames, a plurality of 3-D points comprising at least a first 3-D point, wherein the first 3-D point is determined based at least in part on:
a first 2-D keypoint of a first image frame;
a first depth prior of the first 2-D keypoint; and
a first intrinsic matrix associated with the first image frame.
7. The system of claim 1,
wherein the initial image frame pair comprises a first image frame and second image frame of the plurality of image frames, and
wherein the initial image frame pair is selected based at least in part on how many 2-D keypoint correspondences are in the second image frame and the first image frame.
8. The system of claim 1, wherein the set of depth prior are extracted from a plurality of dense depth maps using bilinear interpolation.
9. The system of claim 1, wherein the executable instructions include further instructions that, as a result of execution by the one or more processors, further cause the system to:
determine a camera pose based at least in part on a first objective function, wherein the first objective function that is used to minimize a first loss determined based at least in part on:
a reprojection error for a set of inlier keypoints; and
a depth consistency error for the set of inlier keypoints.
10. The system of claim 1, wherein the executable instructions include further instructions that, as a result of execution by the one or more processors, further cause the system to:
add one or more 3-D points to a point cloud via triangulation that is refined based at least in part on a second objective function that is used to minimize a second loss determined based at least in part on:
a reprojection error for the one or more 3-D points; and
a depth consistency error for the one or more 3-D points.
11. The system of claim 2,
wherein the depth estimator is a pretrained depth estimator.
12. The system of claim 1, wherein a majority of the plurality of image frames have parallax of 1.0 or less.
Claims filed on 06/20/2024,
Non-transitory computer-readable storage medium claims 13-20
13. A non-transitory computer-readable storage medium storing executable instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to at least:
obtain a plurality of image frames from a digital content item that corresponds to a scene;
determine, based at least in part on a correspondence search, a set of keypoints for the plurality of image frames;
determine, a dense depth map for one of the plurality of image frames;
determine a corresponding set of depth priors;
perform an initialization based at least in part on the set of keypoints and the corresponding set of depth priors to determine an initial image frame pair; and
determine a camera pose based at least in part on the initial image pair.
14. The non-transitory computer-readable storage medium of claim 13, wherein the executable instructions include further instructions that, as a result of execution by the one or more processors, further cause the computer system to:
perform a depth-constrained optimization to incrementally update the camera pose based at least in part on the set of 2-D keypoints, and the corresponding set of depth priors.
16. The non-transitory computer-readable storage medium of claim 13,
wherein the set of depth priors is based on the set of keypoints and the dense map.
15. The non-transitory computer-readable storage medium of claim 14,
wherein the depth-constrained optimization is based at least in part on the set of keypoints and the set of depth priors.
17. The non-transitory computer-readable storage medium of claim 13,
wherein the instructions to perform the initialization based at least in part on the set of keypoints and the corresponding set of depth priors to determine an initial image pair include instructions that, as a result of execution by the one or more processors, cause the computer system to:
determine, for the plurality of image frames, a plurality of 3-D points comprising at least a first 3-D point, wherein the first 3-D point is determined based at least in part on:
a first 2-D keypoint of a first image frame;
a first depth prior of the first 2-D keypoint; and
a first intrinsic matrix associated with the first image frame.
18. The non-transitory computer-readable storage medium of claim 13,
wherein the initial image frame pair comprises a first image frame and second image frame of the plurality of image frames, and
wherein the initial image frame pair is selected based at least in part on how many keypoints correspondences are in the second image frame and the first image frame.
19. The non-transitory computer-readable storage medium of claim 13, wherein the executable instructions include further instructions that, as a result of execution by the one or more processors, further cause the computer system to:
determine a camera pose based at least in part on a first objective function, wherein the first objective function that is used to minimize a first loss determined based at least in part on:
a reprojection error for a set of inlier keypoints; and
a depth consistency error for the set of inlier keypoints.
20. The non-transitory computer-readable storage medium of claim 13, wherein the executable instructions include further instructions that, as a result of execution by the one or more processors, further cause the computer system to:
add one or more 3-D points to a point cloud via triangulation that is refined based at least in part on a second objective function that is used to minimize a second loss determined based at least in part on:
a reprojection error for the one or more 3-D points; and
a depth consistency error for the one or more 3-D points.
US 12,046,002 B1
System claims 6-13
6. A system, comprising:
one or more processors; and
memory storing executable instructions that, as a result of execution by the one or more processors, cause the system to:
obtain a plurality of image frames from a digital content item that corresponds to a scene;
determine, based at least in part on a correspondence search, a set of 2-D keypoints for the plurality of image frames;
obtain a depth estimator;
determine, based at least in part on the depth estimator, a plurality of dense depth maps for the plurality of image frames;
determine, based at least in part on the set of 2-D keypoints and the plurality of dense depth maps, a corresponding set of depth priors;
perform an initialization based at least in part on the set of 2-D keypoints and the corresponding set of depth priors to determine an initial image frame pair, the initial image frame pair comprises a first image frame and second image frame of the plurality of image frames;
determine a camera pose based at least in part on the first image frame and the second image frame; and
perform a depth-constrained optimization to incrementally update the camera pose based at least in part on the set of 2-D keypoints, and the corresponding set of depth priors.
This limitation is present in patent claim 6:
“obtain a depth estimator;
determine, based at least in part on the depth estimator, a plurality of dense depth maps for the plurality of image frames;”.
This limitation is present in patent claim 6:
“determine, based at least in part on the set of 2-D keypoints and the plurality of dense depth maps, a corresponding set of depth priors;”.
This limitation is present in patent claim 6:
“perform a depth-constrained optimization to incrementally update the camera pose based at least in part on the set of 2-D keypoints, and the corresponding set of depth priors.”.
This limitation is present in patent claim 6:
“determine, based at least in part on a correspondence search, a set of 2-D keypoints for the plurality of image frames;”.
7. The system of claim 6,
wherein the instructions to perform the initialization based at least in part on the set of 2-D keypoints and the corresponding set of depth priors to determine an initial image pair include instructions that, as a result of execution by the one or more processors, cause the system to:
determine, for the plurality of image frames, a plurality of 3-D points comprising at least a first 3-D point, wherein the first 3-D point is determined based at least in part on:
a first 2-D keypoint of a first image frame;
a first depth prior of the first 2-D keypoint; and
a first intrinsic matrix associated with the first image frame.
8. The system of claim 6,
wherein the initial image frame pair is selected based at least in part on how many 2-D keypoint correspondences are in the second image frame and the first image frame.
9. The system of claim 6, wherein the set of depth prior are extracted from the plurality of dense depth maps using bilinear interpolation.
10. The system of claim 6, wherein the executable instructions include further instructions that, as a result of execution by the one or more processors, further cause the system to:
determine a camera pose based at least in part on a first objective function, wherein the first objective function that is used to minimize a first loss determined based at least in part on:
a reprojection error for a set of inlier keypoints; and
a depth consistency error for the set of inlier keypoints.
11. The system of claim 6, wherein the executable instructions include further instructions that, as a result of execution by the one or more processors, further cause the system to:
add one or more 3-D points to a point cloud via triangulation that is refined based at least in part on a second objective function that is used to minimize a second loss determined based at least in part on:
a reprojection error for the one or more 3-D points; and
a depth consistency error for the one or more 3-D points.
12. The system of claim 6,
wherein the depth estimator is a pretrained depth estimator.
13. The system of claim 6, wherein a majority of the plurality of image frames have parallax of 1.0 or less.
US 12,046,002 B1
Non-transitory computer-readable storage medium claims 14-20
14. A non-transitory computer-readable storage medium storing executable instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to at least:
obtain a plurality of image frames from a digital content item that corresponds to a scene;
determine, based at least in part on a correspondence search, a set of 2-D keypoints for the plurality of image frames;
obtain a depth estimator;
determine, based at least in part on the depth estimator, a plurality of dense depth maps for the plurality of image frames;
determine, based at least in part on the set of 2-D keypoints and the plurality of dense depth maps, a corresponding set of depth priors;
perform an initialization based at least in part on the set of 2-D keypoints and the corresponding set of depth priors to determine an initial image frame pair, the initial image frame pair comprises a first image frame and second image frame of the plurality of image frames;
determine a camera pose based at least in part on the first image frame and the second image frame; and
perform a depth-constrained optimization to incrementally update the camera pose based at least in part on the set of 2-D keypoints, and the corresponding set of depth priors.
This limitation is present in patent claim 14:
“obtain a depth estimator;
determine, based at least in part on the depth estimator, a plurality of dense depth maps for the plurality of image frames;”.
This limitation is present in patent claim 14:
“determine, based at least in part on the depth estimator, a plurality of dense depth maps for the plurality of image frames;”.
This limitation is present in patent claim 14:
“perform a depth-constrained optimization to incrementally update the camera pose based at least in part on the set of 2-D keypoints, and the corresponding set of depth priors.”
This limitation is present in patent claim 14:
“determine, based at least in part on a correspondence search, a set of 2-D keypoints for the plurality of image frames;”.
15. The non-transitory computer-readable storage medium of claim 14,
wherein the instructions to perform the initialization based at least in part on the set of 2-D keypoints and the corresponding set of depth priors to determine an initial image pair include instructions that, as a result of execution by the one or more processors, cause the computer system to:
determine, for the plurality of image frames, a plurality of 3-D points comprising at least a first 3-D point, wherein the first 3-D point is determined based at least in part on:
a first 2-D keypoint of a first image frame;
a first depth prior of the first 2-D keypoint; and
a first intrinsic matrix associated with the first image frame.
16. The non-transitory computer-readable storage medium of claim 14,
wherein the initial image frame pair is selected based at least in part on how many 2-D keypoints correspondences are in the second image frame and the first image frame.
17. The non-transitory computer-readable storage medium of claim 14, wherein the set of depth prior are extracted from the plurality of dense depth maps using bilinear interpolation.
18. The non-transitory computer-readable storage medium of claim 14, wherein the executable instructions include further instructions that, as a result of execution by the one or more processors, further cause the computer system to:
determine a camera pose based at least in part on a first objective function, wherein the first objective function that is used to minimize a first loss determined based at least in part on:
a reprojection error for a set of inlier keypoints; and
a depth consistency error for the set of inlier keypoints.
19. The non-transitory computer-readable storage medium of claim 14, wherein the executable instructions include further instructions that, as a result of execution by the one or more processors, further cause the computer system to:
add one or more 3-D points to a point cloud via triangulation that is refined based at least in part on a second objective function that is used to minimize a second loss determined based at least in part on:
a reprojection error for the one or more 3-D points; and
a depth consistency error for the one or more 3-D points.
20. The non-transitory computer-readable storage medium of claim 14,
wherein the depth estimator is a pretrained depth estimator.
Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Moeglein et al., US Patent Application Publication No. 2015/0094952, describes computer visions which estimate camera pose relative to a current image frame by utilizing keypoints, refer to paragraph [0095].
[0095] As outlined above, CVM 155 may employ any of a variety of algorithms to determine and detect keypoints in image frames to estimate a 6 Degrees Of Freedom (6DOF) camera/UE pose relative to a current image frame. Some subset of the captured image frames may be stored as keyframes. In some embodiments, CVM 155 may use the pose determined in a prior image frame as an initial pose, which may then be refined based on the locations of keypoints in the image frame. When the pose of UE 100 has changed by more than some threshold, the current image frame may be stored by CM 155 as a new keyframe. The locations of keypoints and other image correspondences relative to a prior image frame or a stored keyframe may also be used to estimate scene geometry. For example, CVM 155 may assign depth information to features of structure 427 and 3D model of structure 427 may be iteratively created. In some embodiments, MW assumptions may be used to simplify 3D external envelope determination for a structure.
The article titled Depth-Guided Sparse Structure-from-Motion for Movies and TV Shows has common authors to the inventors of this application and common subject matter to this application.
Allowable Subject Matter
Claims 1-20 would be allowable if rewritten or amended to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, set forth in this Office action and is a proper terminal disclaimer is filed.
The following is a statement of reasons for the indication of allowable subject matter:
Claims 1-12:
The prior art of record fails to teach or suggest in the context of independent claim 1:
“perform an initialization based at least in part on the set of keypoints and the corresponding set of depth priors to determine an initial image frame pair
determine a camera pose based at least in part on the initial image pair; and
perform a depth-constrained optimization to incrementally update the camera pose.”.
Claims 13-20:
The prior art of record fails to teach or suggest in the context of independent claim 13:
“perform an initialization based at least in part on the set of keypoints and the corresponding set of depth priors to determine an initial image frame pair; and
determine a camera pose based at least in part on the initial image pair.”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JEFFERY A BRIER whose telephone number is (571)272-7656. The examiner can normally be reached on Mon-Fri from 8:30am-3:00pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao M Wu, can be reached at telephone number 571-272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center. Status information for unpublished applications is available through Patent Center for authorized users only. Should you have questions about access to Patent Center, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
JEFFERY A. BRIER
Primary Examiner
Art Unit 2613
/JEFFERY A BRIER/Primary Examiner, Art Unit 2613