DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 11/13/2024, 1/8/2025, and 4/10/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered and attached by the examiner.
Claim Objections
Claim 5 is objected to because of the following informalities: the phrase “a twist of the first rolling shutter camera” is unclear. Please specify what is incorporated in the twist of the camera or how the twist is determined. Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-4, 10-14, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Wagner (U.S. Patent Pub. No. 2017/0374256) in view of Vajna (U.S. Patent Pub. No. 10867190).
Regarding Claim 1, Wagner teaches a depth sensing system configured to determine a 3D position of an object represented in each of a first image and a second image comprising:
• a first rolling shutter camera configured to capture the first image; and (Fig. 1, 114; ¶27 device 100 is a mobile/portable platform (e.g., client). Device 100 can include a means for capturing an image, such as RS camera 114 and may optionally include motion sensors 111, such as accelerometers, gyroscopes, electronic compass, or other similar motion sensing elements. Device 100 may also capture images on a front or rear-facing camera (e.g., RS camera 114).)
• a processing system (Fig. 1, 161 Processor) configured to:
• determine a correspondence between a first feature from the first image and a second
feature from the second image (¶29 a RSC capable device (e.g., device 100) can also perform 6DOF SLAM (e.g., SLAM/SfM module 173) or structure from motion (SfM) tracking of an object or environment. 6DOF SLAM or SfM tracking can associate features observed from input images from RS camera 114 to a 3D object or environment map,) wherein the first feature and the second feature are associated with the object, wherein the first feature is located at a pixel coordinate in the first image (¶31 device 100 extracts features from a captured image. A feature (e.g., feature point or interest point) as used herein is as an interesting or notable part of an image. The features extracted from the captured image may represent distinct points along three-dimensional space (e.g., coordinates on axes X, Y, and Z) and every feature point may have an associated feature location. The features in keyframes either match or fail to match (i.e., are the same or correspond to) the features of previously captured images;)
• determine a first feature capture time for the first feature (¶33 FIG. 2 illustrates the timing of scanline captures from a Rolling Shutter, in one embodiment. As introduced above, a RS camera creates an image via scanlines that occur over time. For example, as illustrated in FIG. 2, a first scanline s1 may be captured at time t0, followed by scanline s2 captured at time t1, until a final scanline for the image is captured (e.g., the last illustrated scanline starting at t4 and completed at time tn) based on a delay map and the pixel coordinate , wherein the delay map:
• comprises pixel capture delays for pixels in images captured by the first rolling shutter camera; and
• is corrected for a lens distortion of the first rolling shutter camera (¶33 In the illustrated example of FIG. 2, the read-out time (i.e., RS delay/duration) may be the difference between the first (i.e., s1), and the last scanline (i.e., s4) which would be calculated as t4-t0. In some embodiments, each scanline has an associated pose and a reference pose is selected to correct other scanline poses within the image. For example, in the illustrated example of FIG. 2, given five scanlines, a middle scanline (e.g., scanline s3) may be selected as the reference scanline such that scanlines s1, s2, s4, and s5 may be corrected to the reference pose provided by scanline s3. In other embodiments, other scanlines instead of a middle scanline may be selected as the reference scanline;)
• determine a first feature capture pose for the first rolling shutter camera based on the first feature capture time and a set of motion information; and (¶34 At block 305, the embodiment (e.g., RSC) receives, from a rolling shutter camera, an image having a plurality of scanlines captured at different times, where each scanline includes a plurality of 2D pixels, and where each scanline has an associated camera pose.)
• calculate the 3D position of the object by performing triangulation on the object based on the first feature and the second feature using the first feature capture pose (¶30 Feature point associations may be used to determine the camera position and orientation (i.e., pose) related to a respective camera image. An environment map or object may include 3D feature points triangulated from two or more image frames or keyframes; ¶44 With movement of the device away from an initial reference image position, the device can capture additional images from alternate views. After extracting features and triangulating from additional keyframes, increased accuracy of the augmentation can be achieved (e.g., borders around an object may fit more precisely, the representation of the object in the scene will appear more realistic, and target placement can be more accurate relative to the camera pose))
Wagner discloses associating features between images using SLAM, but does not explicitly disclose • determine a correspondence between a first feature from the first image and a second feature from the second image wherein the first feature and the second feature are associated with the object.
• calculate the 3D position of the object by performing triangulation on the object based on the first feature and the second feature.
Vajna is in the same field of art of image analysis. Further, Vajna teaches • determine a correspondence between a first feature from the first image and a second feature from the second image wherein the first feature and the second feature are associated with the object (Fig. 1; Col 19 Lines 4-6: The disparity is calculated as the difference between the image coordinates of the first and second image detections along the epipolar direction.)
• calculate the 3D position of the object by performing triangulation on the object based on the first feature and the second feature (Col 7-8 Lines 54-5: a 3D projection module (a 3D projection module 24 in the embodiment of FIG. 1, labelled as ‘3D projection’; this module is adapted for 3D projection) adapted for generating a 3D lane detection data block by means of triangulation, using calibration data corresponding to the first image and the second image, based on a first data block part of the data block of image space detection pairs corresponding to a first member of the image space detection pair)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wagner by determining a correspondence between image features and calculating the 3D position of an object using triangulation that is taught by Vajna; thus, one of ordinary skilled in the art would be motivated to combine the references for more efficient lane detection (Vajna Col 2 Line 10).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 2, Wagner in view of Vajna discloses the depth sensing system of claim 1, wherein determining the first feature capture time comprises selecting a predetermined pixel capture delay at the pixel coordinate from the delay map (Wagner, ¶33 the read-out time (i.e., RS delay/duration) may be the difference between the first (i.e., s1), and the last scanline (i.e., s4) which would be calculated as t4-t0. In some embodiments, each scanline has an associated pose and a reference pose is selected to correct other scanline poses within the image. For example, in the illustrated example of FIG. 2, given five scanlines, a middle scanline (e.g., scanline s3) may be selected as the reference scanline such that scanlines s1, s2, s4, and s5 may be corrected to the reference pose provided by scanline s3. In other embodiments, other scanlines instead of a middle scanline may be selected as the reference scanline.)
Regarding Claim 3, Wagner in view of Vajna discloses the depth sensing system of claim 1, wherein:
• the processing system is further configured to correct the first image for the lens distortion of the first rolling shutter camera; and (Wagner, ¶23 Camera lenses typically introduce some form of lens distortion that may be measured and adjusted (e.g., fixed or improved). Distortion adjustment/compensation may move features to different coordinates. After correcting for lens distortion, other effects (e.g., intrinsics such as focal length, non-square pixels, and the projection center) may also be adjusted or corrected)
• the pixel coordinate comprises a coordinate in the corrected first image (Wagner, ¶42 At block 325, the embodiment provides, in response to the reprojecting, reference timeframe corrected 2D coordinates for the one or more 2D pixels in the first scanline. RS correction may be provided in response to processing the first image and before a next image is received.)
Regarding Claim 4, Wagner in view of Vajna discloses the depth sensing system of claim 1, wherein the 3D position of the object is calculated before a subsequent image is captured by the first rolling shutter camera (Wagner, Fig. 3 shows no image is taken during the process of 3D position determination.)
Regarding Claim 10, Wagner in view of Vajna discloses the depth sensing system of claim 1, wherein the first rolling shutter camera is mounted on an autonomous vehicle, wherein the 3D position of the object is used to generate operating instructions for the autonomous vehicle (Vajna, Col 1 Lines 13-19: The perception of the three-dimensional environment of road vehicles is crucial in autonomous driving applications. Road surface estimation and the detection of lane markers, or boundaries are necessary for lane keeping and lane change maneuvers, as well as to position obstacles at lane level. Lane geometry and obstacle information are key inputs of devices responsible to control the vehicle.)
Regarding claim 11, claim 11 has been analyzed with regard to claim 1 and is rejected for the same reasons of obviousness as used above as well as in accordance with Wagner further teaching on: A method (Claim 1: A method to correct rolling shutter artifacts.)
Claim 12 recites limitations similar to claim 2 and is rejected under the same rationale and reasoning.
Claim 13 recites limitations similar to claim 3 and is rejected under the same rationale and reasoning.
Claim 14 recites limitations similar to claim 4 and is rejected under the same rationale and reasoning.
Claim 20 recites limitations similar to claim 10 and is rejected under the same rationale and reasoning.
Claims 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Wagner (U.S. Patent Pub. No. 2017/0374256) in view of Vajna (U.S. Patent No. 10867190) in view of Yuan (U.S. Patent Pub. No. 2008/0273751).
Regarding Claim 9, Wagner in view of Vajna teaches the depth sensing system of claim 1, and a rolling shutter camera.
Wagner in view of Vajna does not explicitly disclose wherein determination of the correspondence between the first feature and the second feature comprises evaluating corresponding pixels deviating from an epipolar constraint.
Yuan is in the same field of art of image analysis. Further, Yuan teaches wherein determination of the correspondence between the first feature and the second feature comprises evaluating corresponding pixels deviating from an epipolar constraint (¶88 The epipolar constraint is a geometric constraint for motion detection in two views, which encapsulates the relative orientation between two cameras; ¶89 A pixel-to-line distance d.sub.epi.psi. is defined to measure how much the pixel pair deviates from the epipolar lines.)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wagner in view of Vajna by determining a correspondence between images with an epipolar constraint that is taught by Yuan; thus, one of ordinary skilled in the art would be motivated to combine the references to provide an additional cue for pixel classification (Yuan¶9).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 9 recites limitations similar to claim 19 and is rejected under the same rationale and reasoning.
Allowable Subject Matter
Claims 5-8 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The objection to claim 5 also needs to be addressed in order for claim 5 to be allowable.
Regarding claim 5, no prior art teaches wherein the set of motion information comprises a twist (see objection) of the first rolling shutter camera during a time window between:
• an effective global shutter time for the first image; and
• a capture time of a previous image captured by the first rolling shutter camera; And
wherein the first feature capture pose is determined from a pose interpolation over the time window using the twist of the first rolling shutter camera.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DUSTIN BILODEAU whose telephone number is (571)272-1032. The examiner can normally be reached 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUSTIN BILODEAU/Examiner, Art Unit 2664
/JENNIFER MEHMOOD/Supervisory Patent Examiner, Art Unit 2664
Prosecution Insights