DETAILED ACTION
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-7, 9, 18, 20-21, 23-27 is/are rejected under 35 U.S.C. 103 as being unpatentable over Aoki in view of Aoki in view of Lee et al. US 2022/0067973 hereinafter referred to as Lee in view of Yasui et al. US 2022/0309796 hereinafter referred to as Yasui.
In regards to claim 1, Aoki teaches:
“A method comprising: receiving sensor data obtained using a sensor located within an interior of a vehicle ... the sensor data representative of at least ... a transparent component ... and a feature that is located exterior to the vehicle and is visible through the transparent component”
Aoki Figure 1 and paragraph [0048] teaches an example of the case of a calibration of a captured image that is captured by the stereo camera 10 (car-mounted stereo camera) that is installed inside the front windshield of a car. Aoki teaches in Figure 4 and paragraph [0059] the measurement tool 20 may be any member that has an area for acquiring data that is used for a calibration. The surface of the measurement tool 20 includes five marks 21. The Examiner interprets that the camera 10 is oriented towards the windshield which is at least partially interior to the vehicle and images features exterior to the vehicle.
“determining, based at least on the sensor data, a relative position between the sensor and the feature”
Aoki paragraph [0097] and Figure 10 teaches the calibration device 30 measures the relative position between the stereo camera 10 and the object (Step S102). Specifically, the calibration device 30 measures the relative positions between the optical center O.sub.0 of the first camera 1 of the stereo camera 10 and the marks 21 on the measurement tool 20 by performing Step S6 to Step S10 that are described below.
“determining information that relates ... the interior of the vehicle with the feature”
Aoki paragraph [0097] teaches Next, the calibration device 30 measures the relative position between the stereo camera 10 and the object (Step S102). Specifically, the calibration device 30 measures the relative positions between the optical center O.sub.0 of the first camera 1 of the stereo camera 10 and the marks 21 on the measurement tool 20 by performing Step S6 to Step S10 that are described below. Aoki Figure 11 teaches various information being determined such as calculating positions of first and second cameras and calculating direction of first and second cameras. The Examiner interprets this as information that relates the interior of the vehicle (i.e. position of the camera) with the feature (marks).
“and determining, based at least on the information and the relative position, one or more values for one or more calibration parameters that calibrate the sensor ...”
Aoki paragraph [0098] and Figure 10 teaches the calibration device 30 determines a calibration parameter based on the relative position (Step S103).
Aoki does not explicitly teach:
“[determining information that relates] a coordinate system associated [with the interior of the vehicle with the feature]” and “[calibrate the sensor] with respect to the coordinate system”
However, Lee teaches in paragraph [0068] the camera calibration apparatus 100 may estimate a motion parameter based on a change in feature points on the pixel coordinate system, transform the feature points of the pixel coordinate system into three-dimensional coordinates of the camera coordinate system based on the motion parameter and triangulation, and estimate an external parameter based on a transformation relationship between the camera coordinate system and the vehicle coordinate system and constraints of the feature points of the vehicle coordinate system. Lee Figure 2 teaches the camera is located within the interior of the vehicle. It would have been obvious for a person with ordinary skill in the art before the invention was effectively filed to have modified Aoki in view of Lee to have included the features of “[determining information that relates] a coordinate system associated [with the interior of the vehicle with the feature]” and “[calibrate the sensor] with respect to the coordinate system” because it is necessary that the autonomous vehicle frequently calibrate the camera in relation to external parameter(s). (Lee [0004]).
Aoki/Lee do not explicitly teach:
“[a sensor located within an interior of a vehicle] and oriented at least partially towards a rear of the vehicle, [the sensor data representative of at least] one or more seats that are located within the interior of the vehicle, [a transparent component] located at a side of the vehicle”
Yasui paragraph [0072] teaches images of the outside of the vehicle captured through the left and right windows by a camera having a wide angle of view particularly from the vicinity of the center of the front seat toward the vehicle interior direction (that is, the rear). Yasui paragraph [0055] teaches FIG. 7A shows a region where such moving objects are detected. Here, a region 701 (vehicle) and a region 702 (person) are detected as moving objects outside the vehicle, and a region 703 (handkerchief) is detected as a moving object inside the vehicle. From Figure 7A, inter alia, it is illustrated that the view is of the interior representative of interior seats and features located outside the side window, which is a transparent component. It would have been obvious for a person with ordinary skill in the art before the invention was effectively filed to have modified Aoki/Lee in view of Yasui to have included the features of “[a sensor located within an interior of a vehicle] and oriented at least partially towards a rear of the vehicle, [the sensor data representative of at least] one or more seats that are located within the interior of the vehicle, [a transparent component] located at a side of the vehicle” because it is necessary to identify a behavior of the vehicle such as a vehicle speed or turning based on information acquired independently by a driving assistance apparatus retrofitted to the vehicle (Yasui [0006]).
In regards to claim 2, Aoki/Lee/Yasui teaches all the limitations of claim 1 and further teaches:
“further comprising: determining information that associates a coordinate system associated with the vehicle with the feature that is located exterior to the vehicle”
Aoki paragraph [0053] and Figure 3 teach the coordinate system of FIG. 3 is referred to as the “camera coordinate system”. The Examiner interprets the camera coordinate system is a coordinate system associated with the vehicle because the camera is mounted on vehicle. Aoki paragraph [0101] teaches the first-camera position calculating unit 32 calculates the first camera coordinates that indicate the position of the optical center O.sub.0 of the first camera 1 by using the object coordinate system on the basis of the multiple pieces of first distance information and the three-dimensional coordinate information on the distance measurement device 22 (the measurement point 23) in the object coordinate system (Step S6).
“wherein: the determining the one or more values for the one or more calibration parameters associated with the sensor is further based at least on the information”
Aoki Figure 11 steps S4-S12 could be interpreted as information. For example, the captured images are information that is used to determine the calibration parameters as well as all the values calculated from the capture images that are then used to determine the calibration parameters which occurs in step S13.
“and the one or more values for the one or more calibration parameters calibrate the sensor with respect to the coordinate system”
Aoki Figure 11 step S13
In regards to claim 3, Aoki/Lee/Yasui teaches all the limitations of claim 1 and further teaches:
“wherein the information indicates a transformation between the coordinate system and the feature, and wherein the determining the information is based at least on one or more of: a model representing information associated with the vehicle; and a location of the feature with respect to the vehicle”
Aoki paragraph [0065] teaches the receiving unit 31 inputs, to the distance calculating unit 36, the three-dimensional coordinate information on the multiple (five in the present embodiment) marks 21 in the object coordinate system. The receiving unit 31 inputs the first captured image and the second captured image to the parallax calculating unit 38. The Examiner interprets the locations of the marks as equivalent to a location of a feature.
In regards to claim 4, Aoki/Lee/Yasui teaches all the limitations of claim 1 and further teaches:
“wherein the determining the relative position between the sensor and the feature comprises: determining that a portion of a sensor representation represented by the sensor data depicts the feature; and determining, based at least on the portion of the sensor representation, the relative position between the sensor and the feature”
Aoki paragraph [0112] teaches the captured image that is captured at Step S23 is input to the calibration device 30 (Step S25). Next, the calibration device 30 calculates the relative position between the measurement tool 20 and the optical center O.sub.0 of the stereo camera 10. Specifically, the calibration device 30 calculates the position and the direction of the stereo camera 10 (Step S6 to Step S9 of FIG. 11). Then, the calibration device 30 calculates the distance d between the measurement tool 20 (the marks 21) and the optical center O.sub.0 of the stereo camera 10 in the direction of the optical axis of the first camera 1 (Step S10 of FIG. 11).
In regards to claim 5, Aoki/Lee/Yasui teaches all the limitations of claim 1 and further teaches:
“further comprising: receiving second sensor data generated obtained using the sensor, the second sensor data representing at least the feature located exterior to the vehicle; and determining, based at least on the second sensor data, a second relative position between the sensor and the feature, wherein the determining the one or more values for one or more calibration parameters that that calibrates the sensor ... is further based at least on the second relative position”
Aoki paragraph [0112] teaches the captured image that is captured at Step S23 is input to the calibration device 30 (Step S25). Next, the calibration device 30 calculates the relative position between the measurement tool 20 and the optical center O.sub.0 of the stereo camera 10. Specifically, the calibration device 30 calculates the position and the direction of the stereo camera 10 (Step S6 to Step S9 of FIG. 11). Then, the calibration device 30 calculates the distance d between the measurement tool 20 (the marks 21) and the optical center O.sub.0 of the stereo camera 10 in the direction of the optical axis of the first camera 1 (Step S10 of FIG. 11). The Examiner interprets that plural marks exists. Therefore, performing this process on the second mark is equivalent to the claimed feature.
Aoki does not explicitly teach:
“with respect to the coordinate system”
Lee teaches in paragraph [0068] the camera calibration apparatus 100 may estimate a motion parameter based on a change in feature points on the pixel coordinate system, transform the feature points of the pixel coordinate system into three-dimensional coordinates of the camera coordinate system based on the motion parameter and triangulation, and estimate an external parameter based on a transformation relationship between the camera coordinate system and the vehicle coordinate system and constraints of the feature points of the vehicle coordinate system. Lee Figure 2 teaches the camera is located within the interior of the vehicle. It would have been obvious for a person with ordinary skill in the art before the invention was effectively filed to have modified Aoki in view of Lee to have included the features of “with respect to the coordinate system” because it is necessary that the autonomous vehicle frequently calibrate the camera in relation to external parameter(s). (Lee [0004]).
In regards to claim 6, Aoki/Lee/Yasui teaches all the limitations of claim 1 and further teaches:
“wherein: the sensor data further represents a second feature that is located exterior to the vehicle; and the determining the relative position is based at least on a first portion of the sensor data that represents the feature and a second portion of the sensor data that represents the second feature”
The Examiner interprets that plural marks exists. Therefore, a first mark is located at a first portion on the sensor and a second mark is located at a second portion on the sensor.
In regards to claim 7, Aoki/Lee/Yasui teaches all the limitations of claim 1 and further teaches:
“wherein: the sensor data represents a sensor representation; the sensor is positioned within the vehicle such that a portion of the sensor representation is associated with the transparent component of the vehicle; and the feature is positioned external to the vehicle such that the portion of the sensor representation depicts the feature through the transparent component”
Aoki Figure 1 and paragraph [0048] teaches an example of the case of a calibration of a captured image that is captured by the stereo camera 10 (car-mounted stereo camera) that is installed inside the front windshield of a car.
In regards to claim 9, Aoki/Lee/Yasui teaches all the limitations of claim 1 and further teaches:
“wherein the one or more values for the one or more calibration parameters comprise one or more of: one or more first values associated with a translation for calibrating the sensor with respect to a coordinate system associated with the interior of the vehicle; or one or more second values associated with a rotation for calibrating the sensor with respect to the coordinate system associated with the interior of the vehicle”
Aoki paragraph [0051] teaches in the case where the coordinates of the first captured image (the second captured image) are corrected by an affine transformation, as the image correction equation can be represented by using a matrix, a calibration parameter is components of the matrix.
In regards to claim 18, Aoki/Lee/Yasui teaches all the limitations of claim 1 and claim 18 contains limitations contained in claim 1. Therefore, claim 18 is rejected for similar reasoning as applied to claim 1.
In regards to claim 20, Aoki/Lee/Yasui teaches all the limitations of claim 18 and further teaches:
“wherein the processor is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations ;a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets ;a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system implementing one or more large language models (LLMs);a system for performing conversational Al operations; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs);a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources”
Aoki paragraph [0002] teaches the distance that is measured by a car-mounted stereo camera is used for giving an alarm to a driver or controlling a brake, steering, and/or the like, for the purpose of prevention of car collision, a control on a distance between cars, and/or the like. The Examiner interprets this as at least a semi-autonomous machine.
In regards to claim 21, Aoki/Lee/Yasui teaches all the limitations of claim 1 and claim 20 contains similar limitations as in claim 1. Therefore, claim 21 is rejected for similar reasoning as applied to claim 1.
In regards to claim 23, Aoki/Lee/Yasui teaches all the limitations of claim 22 and claim 23 contains similar limitations as in claim 3. Therefore, claim 23 is rejected for similar reasoning as applied to claim 3.
In regards to claim 24, Aoki/Lee/Yasui teaches all the limitations of claim 21 and claim 24 contains similar limitations as in claim 4. Therefore, claim 24 is rejected for similar reasoning as applied to claim 4.
In regards to claim 25, Aoki/Lee/Yasui teaches all the limitations of claim 21 and claim 25 contains similar limitations as in claim 5. Therefore, claim 25 is rejected for similar reasoning as applied to claim 5.
In regards to claim 26, Aoki/Lee/Yasui teaches all the limitations of claim 21 and claim 26 contains similar limitations as in claim 6. Therefore, claim 26 is rejected for similar reasoning as applied to claim 6.
In regards to claim 27, Aoki/Lee/Yasui teaches all the limitations of claim 21 and claim 27 contains similar limitations as in claim 20. Therefore, claim 27 is rejected for similar reasoning as applied to claim 20.
In regards to claim 31, Aoki/Lee/Yasui teach all the limitations of claim 18 and further teach:
“wherein the sensor is oriented at least partially towards a rear of the vehicle”
Yasui paragraph [0072] teaches images of the outside of the vehicle captured through the left and right windows by a camera having a wide angle of view particularly from the vicinity of the center of the front seat toward the vehicle interior direction (that is, the rear). Yasui paragraph [0055] teaches FIG. 7A shows a region where such moving objects are detected. Here, a region 701 (vehicle) and a region 702 (person) are detected as moving objects outside the vehicle, and a region 703 (handkerchief) is detected as a moving object inside the vehicle. From Figure 7A, inter alia, it is illustrated that the view is of the interior representative of interior seats and features located outside the side window, which is a transparent component. It would have been obvious for a person with ordinary skill in the art before the invention was effectively filed to have modified Aoki/Lee in view of Yasui to have included the features of “wherein the sensor is oriented at least partially towards a rear of the vehicle” because it is necessary to identify a behavior of the vehicle such as a vehicle speed or turning based on information acquired independently by a driving assistance apparatus retrofitted to the vehicle (Yasui [0006]).
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Aoki in view of Aoki in view of Lee in view of Yasui in view of Campos et al. US 2020/0250440 hereinafter referred to as Campos.
In regards to claim 8, Aoki/Lee teaches all the limitations of claim 1 and further teaches:
“with respect to the coordinate system”
Lee teaches in paragraph [0068] the camera calibration apparatus 100 may estimate a motion parameter based on a change in feature points on the pixel coordinate system, transform the feature points of the pixel coordinate system into three-dimensional coordinates of the camera coordinate system based on the motion parameter and triangulation, and estimate an external parameter based on a transformation relationship between the camera coordinate system and the vehicle coordinate system and constraints of the feature points of the vehicle coordinate system. Lee Figure 2 teaches the camera is located within the interior of the vehicle. It would have been obvious for a person with ordinary skill in the art before the invention was effectively filed to have modified Aoki in view of Lee to have included the features of “with respect to the coordinate system” because it is necessary that the autonomous vehicle frequently calibrate the camera in relation to external parameter(s). (Lee [0004]).
Aoki/Lee/Yasui do not explicitly teach:
“further comprising :determining one or more values for one or more intrinsic parameters that calibrate the sensor ..., wherein the determining the one or more values for the one or more calibration parameters associated with the sensor is further based at least on the one or more values for the one or more intrinsic parameters”
Calibrating a camera for intrinsic and extrinsic parameters would be considered a standard operating procedure well-known to those of ordinary skill. Campos teaches in paragraph [0100] from a camera mounted on or near the interior of a vehicle windshield, lanes may be determined. Based on a calibration of certain camera parameters, which may include intrinsic and extrinsic camera parameters, and inverse perspective transform may be determined with which lane boundaries may be projected into a world-based coordinate frame. It would have been obvious for a person with ordinary skill in the art before the invention was effectively filed to have modified Aoki/Lee/Yasui in view of Campos to have included the features of “further comprising :determining one or more values for one or more intrinsic parameters associated with the sensor, wherein the determining the one or more values for the one or more calibration parameters associated with the sensor is further based at least on the one or more values for the one or more intrinsic parameters” to improved systems and methods for curve determination, which may include systems and methods for lane detection, parameterization of detected lanes, road boundary detection, and the like (Campos [0006]).
Claim(s) 29 is/are rejected under 35 U.S.C. 103 as being unpatentable over Aoki in view of Aoki in view of Lee in view of Yasui in view of Engstle et al. US 2023/0281873 hereinafter referred to as Engstle.
In regards to claim 29, Aoki/Lee/Yasui teaches all the limitations of claim 1 but do not explicitly teach:
“wherein the coordinate system is associated with a component of the vehicle that is separate from the sensor”
Engstle paragraph [0045] teaches by formally combining the known extrinsic calibration of the reference sensor system, i.e. positioning system to reference coordinate system, and the calibration of the positioning system to the rear axle, the extrinsic calibration of the reference coordinate system to the vehicle or rear axle can be calculated. It would have been obvious for a person with ordinary skill in the art before the invention was effectively filed to have modified Aoki/Lee/Yasui in view of Engstle to have included the features of “wherein the coordinate system is associated with a component of the vehicle that is separate from the sensor” to provide a method which overcomes the disadvantages of the method for determining the position of the portable reference sensor system compared to a vehicle in the prior art (Engstle [0007]).
Claim(s) 30 is/are rejected under 35 U.S.C. 103 as being unpatentable over Aoki in view of Aoki in view of Lee in view of Yasui in view of Singh et al. US 2017/0054970 hereinafter referred to as Singh.
In regards to claim 30, Aoki/Lee/Yasui teaches all the limitations of claim 1 but do not explicitly teach:
“wherein the sensor is oriented to capture one or more occupants of the vehicle”
Singh paragraph [0034] teaches while the test image is displayed, the head/face tracking camera 66 monitors the vehicle interior 12 and outputs a first signal indicative of the position of the head of the vehicle occupant 44. It would have been obvious for a person with ordinary skill in the art before the invention was effectively filed to have modified Aoki/Lee/Yasui in view of Singh to have included the features of “wherein the sensor is oriented to capture one or more occupants of the vehicle” for calibrating alignment of a three-dimensional display within a vehicle (Singh [0002]).
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-9, 18, 20-30 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL E TEITELBAUM, Ph.D. whose telephone number is (571)270-5996. The examiner can normally be reached 8:30AM-5:00PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Miller can be reached at 571-272-7353. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL E TEITELBAUM, Ph.D./Primary Examiner, Art Unit 2422
Prosecution Insights