DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 5 and 14 are objected to because of the following informalities:
Regarding claim 5, “less or more than” should perhaps read --less than or more than--.
Regarding claim 14, “less or more than” should perhaps read --less than or more than--.
Appropriate correction is requested.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 7-8 and 16-17 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 7 and 16 recite a singular “reference time” as comprising six distinct times joined by “and.” It is unclear whether all six times are required collectively or whether a single one of the listed times serves as the reference time. Applicant may overcome the indefiniteness by amending the claims to recite --wherein the reference time is one of a beginning of the scan cycle, a beginning of a previous scan cycle associated with the LIDAR sensor, a time within the previous scan cycle, a time from a reference clock, a time when the TOF sensor was triggered to capture data during one or more previous scan cycles associated with the LIDAR sensor, or a time associated with a data capture operation performed by the TOF sensor during the previous scan cycle--, in accordance with Spec. ¶¶ 105 and 116.
Claims 8 and 17 recite “an exposure time of each of the one or more exposures” and subsequently recite that the time offset comprises “the exposure time.” As multiple preceding exposures are present, it is unclear whether “the exposure time” refers to one particular exposure time, each exposure time, or an aggregate of the exposure times. Applicant may overcome the indefiniteness by amending the claims to recite --wherein the time offset comprises the exposure time of each of the one or more exposures and the time delay--, in accordance with Spec. ¶ 123.
Allowable Subject Matter
Claims 1-6, 9-15 and 18-20 are allowed. Claims 7-8 and 16-17 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) set forth in this Office action. The following is a statement of reasons for the indication of allowable subject matter:
Gassend (US 20190098233 A1), in regard to claim 1, discloses a system (Fig. 2A, device 200, including controller 104 of Fig. 1; ¶¶ 84, 101) comprising:
a memory (Fig. 1, data storage of controller 104; ¶ 35); and
one or more processors coupled to the memory (Fig. 1, processors of controller 104 executing instructions stored in the data storage; ¶ 35),
the one or more processors being configured to: determine a frequency of each scan cycle of a light detection and ranging (LIDAR) sensor configured to collect sensor data for different regions of space as the LIDAR sensor scans in different directions during each scan cycle (Figs. 2A, 2D, and 2E, LIDAR 206; ¶¶ 85, 91, 107- 113, LIDAR 206 collects reflected light data while rotating through different pointing directions at 10 Hz, corresponding to one complete scan every 100 ms);
select an exposure [1: ...] to align with LIDAR data from a LIDAR scan associated with the sensor data collected by the LIDAR sensor during a scan cycle (Figs. 2B and 2C, center vertical row of image sensor 226 containing sensing element 228c; ¶¶ 101-104, 112-115, successive row exposures produce image data and the center row exposure is timed to align with detected reflections from LIDAR 206);
based on the frequency of the scan cycle of the LIDAR sensor, a field-of-view (FOV) of the LIDAR sensor, [2: ...], and a location of the exposure within the exposure sequence, determine an amount of time estimated to lapse between a reference time and an alignment time during the scan cycle when a first point within the FOV of the LIDAR sensor is aligned in space [3: ...] (Figs. 2B, 2D, and 2E, LIDAR 206 and the center row containing sensing element 228c; ¶¶ 94, 111-115, the 10 Hz rotation establishes a 100 ms scan cycle, the reference orientation occurs at zero ms, the LIDAR FOV overlaps the center of the 90 degree camera FOV at 50 ms, and the selected center row occurs 30 ms into the 60 ms rolling exposure period);
based on the amount of time estimated to lapse between the reference time and the alignment time during the scan cycle, determine a time offset [4: ...] (Fig. 1, controller 104; Fig. 2B, camera 208a; ¶ 114, subtracting the 30 ms center row location from the 50 ms alignment time produces a 20 ms trigger offset); and
[5: ...] to capture the data associated with the exposure sequence at a time associated with the time offset (Fig. 1, controller 104, as used in device 200; Fig. 2B, camera 208a; ¶¶ 34, 101, 114, controller 104 sends control signals governing camera capture times and camera 208a is triggered at 20 ms).
However, Gassend does not disclose:
(1) [select an exposure] “from an exposure sequence generated based on data captured by a time-of-flight (TOF) sensor”;
(2) “a FOV of the TOF sensor”;
(3) [a first point within the FOV of the LIDAR sensor is aligned in space] “with a second point within the FOV of the TOF sensor”;
(4) [a time offset] “for triggering the TOF sensor to capture data associated with the exposure sequence”; and,
(5) “send, to the TOF sensor, a signal configured to trigger the TOF sensor” [to capture the data associated with the exposure sequence at a time associated with the time offset].
Gassend instead uses camera 208a and image sensor 226 (¶¶ 86, 95, 101) and is silent towards employment of a TOF sensor. Furthermore, the rolling exposures of Gassend generates image pixel data (¶¶ 101, 102), remaining silent towards generation of an exposure sequence from previously captured TOF data.
Ma (US 20220404478 A1), in regard to claim 1, discloses a system (Fig. 4, computer system 400, further detailed in Figs. 2A, 2B, 2C, and 6, implemented by Fig. 7, computing system 700; ¶¶ 53, 64-65, 69, 79) comprising:
a memory (Fig. 7, memory 720; ¶¶ 79, 83); and
one or more processors coupled to the memory (Fig. 7, processor 710 communicatively coupled to memory 720; ¶¶ 79, 81),
the one or more processors being configured to: [1: ...] a light detection and ranging (LIDAR) sensor configured to collect sensor data for different regions of space as the LIDAR sensor scans in different directions during each scan cycle (Fig. 2A, LIDAR sensor 200 and successive arcs 211-220; ¶ 43, each revolution scans successive spatial arcs and then repeats);
[2: ...] align with LIDAR data from a LIDAR scan associated with the sensor data collected by the LIDAR sensor during a scan cycle (Fig. 4, shared memory trigger 414, using Fig. 2A, LIDAR sensor 200 and Fig. 2B, image capturing sensor 230; ¶¶ 43, 46-47, 65, delaying image capture synchronizes the image data with LIDAR data);
based on the frequency of the scan cycle of the LIDAR sensor, a field-of-view (FOV) of the LIDAR sensor, [3: ...], determine an amount of time estimated to lapse between a reference time and an alignment time during the scan cycle when a first point within the FOV of the LIDAR sensor is aligned in space [4: …] (¶¶ 43-46, 65, calculating the time for the LIDAR beam to rotate from zero crossing point 202 to center 234 based on angular velocity);
based on the amount of time estimated to lapse between the reference time and the alignment time during the scan cycle, determine a time offset (¶¶ 47, 65, determining delay timing from the alignment time) [5: ...]; and
send [6: ...] at a time associated with the time offset (Fig. 4, camera driver 420, further detailed by Fig. 2C, checkpoints 244, 246, and 248 and delay 256, and Fig. 6, block 608; ¶¶ 52-53, 69, 73, camera driver communicates with the camera sensor and capture begins after the timing delay).
However, Ma does not teach:
(1) “determine a frequency of each scan cycle of” [a LIDAR sensor];
(2) “select an exposure from an exposure sequence generated based on data captured by a time-of-flight (TOF) sensor to” [align with LIDAR data from a LIDAR scan associated with the sensor data collected by the LIDAR sensor during a scan cycle];
(3) “a FOV of the TOF sensor, and a location of the exposure within the exposure sequence”;
(4) [determine an amount of time estimated to lapse between a reference time and an alignment time during the scan cycle when a first point within the FOV of the LIDAR sensor is aligned in space] “with a second point within the FOV of the TOF sensor”;
(5) [determine a time offset] “for triggering the TOF sensor to capture data associated with the exposure sequence”; and,
(6) [send] “to the TOF sensor, a signal configured to trigger the TOF sensor to capture the data associated with the exposure sequence” [at a time associated with the time offset].
Rather, Ma teaches sensor 230 as an image capturing camera sensor rather than a TOF sensor, and further teaches averaging of frame exposure durations rather than generating an exposure sequence, selecting an exposure, or using an exposure location within such an exposure sequence (¶¶ 44, 49).
Benemann (US 20200099872 A1) teaches triggering a rolling shutter image capture based on relative sensor FOV orientations and times a particular scan line to a particular portion of the rotating LIDAR FOV, estimating a current frame exposure duration from prior scan line timing, and identifies a TOF camera as a possible sensor type (Figs. 5A & 10; ¶¶ 31, 47-55, 91-100, 116). Although Benemann supplies the FOV and exposure position features, the disclosed synchronization operation concerns a rolling shutter image sensor. Furthermore, the generic identification of a TOF camera does not disclose a TOF exposure sequence generated based on captured TOF data or the use of a selected exposure position from such a sequence in the timing calculation.
Subasingha (US 20210096263 A1) discloses serial TOF exposures having different illumination power, integration, or modulation settings, and uses captured sensor data to adjust subsequent sensor acquisition conditions (Fig. 2; ¶¶ 38-42, 56-62). Xu (US 20200084361 A1) discloses independently gated TOF pixel regions, exposure start times based on object distance supplied by LIDAR, and exposure adjustment based on previously captured TOF image data (Figs. 1-3; ¶¶ 13, 18-23, 27). Subasingha and Xu provide material TOF exposure control teachings, however, the exposures are used for depth acquisition, intensity acquisition, saturation control, range gating, and sensor parameter adjustment, rather than for alignment of a selected TOF exposure with a LIDAR scan cycle.
Pacala (US 20180329066 A1) teaches the use of angular position, current LIDAR revolution rate, camera FOV, and shutter duration to establish a camera trigger angle, and employs multiple camera exposures during a LIDAR revolution (Figs. 4, 8, 9 & 11; ¶¶ 44-46, 51-60, 87-90, 101-105, 135-139). However, the exposures of Pacala are color camera exposures, and remains silent towards employing the position of an exposure within a TOF sequence as an input to trigger calculation.
In sum, the cited prior art discloses: determining camera capture timing from a LIDAR scan position or angular velocity (Gassend, ¶¶ 101-115; Ma, ¶¶ 43, 46-49; Pacala, ¶¶ 101-105, 135-139); coordinating a rolling shutter row or scan line with a LIDAR pointing direction or a location within the LIDAR field of view (Gassend, ¶¶ 101-115; Benemann, ¶¶ 47-55, 91-100); and controlling TOF exposure parameters based on captured TOF data (Subasingha, ¶¶ 38-42, 56-62; Xu, ¶¶ 13, 19-23, 27). However, although the prior art addresses certain limitations of claim 1, the teachings do not disclose or suggest the claimed relationship between the selected exposure in the TOF exposure sequence and the LIDAR alignment timing. Specifically, the prior art does not teach selecting, for alignment with LIDAR scan data, an exposure from a TOF exposure sequence generated based on captured TOF data; using the sequence location of the selected exposure, together with the LIDAR scan cycle frequency and the respective FOVs of the LIDAR and TOF sensors, to determine the interval from a reference time to spatial alignment of points within those FOVs; and using that interval to determine the TOF trigger offset; and sending, to the TOF sensor, a signal configured to trigger the TOF sensor to capture data associated with the exposure sequence at a time associated with the offset.
Accordingly, claim 1 is allowed. Claim 10 is a method corresponding to the system of claim 1 and is allowed for the same reasons. Claim 20 is a computer product corresponding to the system of claim 1 and is allowed for the same reasons. Claims 2-6, 9, 11-15 and 18-19 are allowed by virtue of dependency. Claims 7-8 and 16-17 would be allowable by virtue of dependency if rewritten to overcome the rejection under 35 U.S.C. 112(b) set forth in this Office action.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZHENGQING QI whose telephone number is 571-272-1078. The examiner can normally be reached Monday - Friday 9:00 AM - 5:00 PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, YUQING XIAO can be reached on 571-270-3603. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ZHENGQING QI/Examiner, Art Unit 3645
Prosecution Insights