DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers submitted under 35 U.S.C. 119(a)-(d), which papers have been placed of record in the file.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 06/25/2026 has been entered.
Response to Arguments
Applicant’s arguments see remarks, filed 06/25/2026, with respect to the claims 1-16 have been fully considered but are moot because the arguments do not apply to the current combinations of references being used in the current rejection.
Claim Objections
Claims 1, 12, and 16 are objected to because of the following informalities:
In claim 1, lines 15-16, the term “the vehicle that (i) has its own suspension, (ii) moves with the occupant, and (iii) is independently movable relative to a structure of the vehicle” should be changed to “the vehicle that in order to avoid typographical issue.
In claim 1, line 20, the term “the part of the vehicle moving with the occupant” should be changed to “the part of the vehicle that moves with the occupant” in order to maintain consistent terminology throughout the claim limitations.
In claim 12, lines 12-14, the term “the vehicle that (i) has its own suspension, (ii) moves with the occupant, and (iii) is independently movable relative to a structure of the vehicle” should be changed to “the vehicle that in order to avoid typographical issue.
In claim 12, lines 16-17, the term “the part of the vehicle moving with the occupant” should be changed to “the part of the vehicle that moves with the occupant” in order to maintain consistent terminology throughout the claim limitations.
In claim 16, lines 13-15, the term “the vehicle that (i) has its own suspension, (ii) moves with the occupant, and (iii) is independently movable relative to a structure of the vehicle” should be changed to “the vehicle that in order to avoid typographical issue.
In claim 16, lines 17-18, the term “the part of the vehicle moving with the occupant” should be changed to “the part of the vehicle that moves with the occupant” in order to maintain consistent terminology throughout the claim limitations.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 7-8, and 10-11 are rejected under 35 U.S.C. 103 as being unpatentable over HYUGA (US 20200065595 A1), hereinafter referenced as HYUGA, in view of UPMANUE et al. (US 20200074197 A1), hereinafter referenced as UPMANUE, and further in view of SELDEN et al. (US 20160129749 A1), hereinafter referenced as SELDEN, and further in view of JARADI et al. (US 20220388466 A1), hereinafter referenced as JARADI.
Regarding claim 1, HYUGA teaches a system for interior sensing in a vehicle (Fig. 1-2. Paragraph [0069]-HYUGA discloses the driver state estimation device 10 is established as a device wherein various kinds of programs stored in the ROM 13 are read into the RAM 14 and conducted by the CPU 12, so as to perform processing as the storage instructing section 21, reading instructing section 22, head detecting section 23, defocus amount detecting section 24, distance estimating section 25, and driving operation possibility deciding section 26.), the system comprising:
a camera (Fig. 1, #11 called a monocular camera. Paragraph [0050]) configured to capture image data relating to an occupant within the vehicle to create captured image data (Fig. 1-3. Paragraph [0052]-HYUGA discloses the monocular camera 11 as an imaging section can periodically (e.g. 30-60 times/sec) pick up images including the head of the driver sitting in the driver's seat, and comprises a lens system 11a consisting of one or more lenses, an imaging element 11b such as a CCD or a CMOS which generates imaging data of a subject, an analog-to-digital conversion section (not shown) which converts the imaging data to digital data.),
at least one processor (Fig. 1-2 and Fig. 6, #12 called a CPU. Paragraph [0054]) configured to:
identify feature points of the occupant (Fig. 1-3, #30 called a driver. Paragraph [0061]) from the captured image data and generate raw feature points data (Fig. 1-6. Paragraph [0054]-HYUGA discloses the CPU 12 is a hardware processor, which reads out a program stored in the ROM 13, and based on said program, performs various kinds of processing on image data picked up by the monocular camera 11 (wherein the raw feature points data is the camera data.). Further in paragraph [0069]-HYUGA discloses the driver state estimation device 10 is established as a device wherein various kinds of programs stored in the ROM 13 are read into the RAM 14 and conducted by the CPU 12, so as to perform processing as the storage instructing section 21, reading instructing section 22, head detecting section 23, defocus amount detecting section 24, distance estimating section 25, and driving operation possibility deciding section 26.);
compensate the raw feature points data with displacement data correlated with a second coordinate system (Fig. 3, illustrates a driver seat that moves away or towards the facing camera that is mounted behind the steering wheel. Paragraph [0089]-HYUGA discloses when estimating the distance Z, changes in size of the face area of the driver in a plurality of images (time-series images) picked up by the monocular camera 11 may be detected so as to decide in which direction, forward or backward, the driver is away from the focal position where the monocular camera 11 focuses, and with use of said decision result and the defocus amount d, the distance Z may be estimated. Further in paragraph [0090]-HYUGA discloses in step S5, with use of the distance Z, the distance A from the steering wheel 32 to the head of the driver 30 is estimated. For example, when the steering wheel 32 is on the line segment between the monocular camera 11 and the driver 30, the distance A is estimated by subtracting the distance B between the monocular camera 11 and the steering wheel 32 from the distance Z. Further read Fig. 6.);
generate processed result data (Fig. 6. Paragraph [0090]-HYUGA discloses in step S5, with use of the distance Z, the distance A from the steering wheel 32 to the head of the driver 30 is estimated. For example, when the steering wheel 32 is on the line segment between the monocular camera 11 and the driver 30, the distance A is estimated by subtracting the distance B between the monocular camera 11 and the steering wheel 32 from the distance Z (wherein distance A is result data).); and
determine, based on the processed result data, a set of occupant characteristics indicative of behavior of the occupant (Fig. 1-2 and Fig. 6, illustrates the image data captured by the monocular camera #11 is the supply to the image storing part #15a and CPU #12 to process the camera data to further send data to an HMI (Human Machine Interface), and S1-7 the camera data is analyzed and controls the HMI #40 to send an alarm about the driver and seat. Paragraph [0054]-HYUGA discloses the CPU 12 is a hardware processor, which reads out a program stored in the ROM 13, and based on said program, performs various kinds of processing on image data picked up by the monocular camera 11. A plurality of CPUs 12 may be mounted for every processing such as image processing or control signal output processing. Further in paragraph [0087]-HYUGA discloses in step S1, data of one or more images picked up by the monocular camera 11 is read from the image storing part 15a, and in step S2, in the read-out one or more images 11c, the head (face) area of the driver 30A is detected. Paragraph [0092-0093]-HYUGA discloses when it is judged that the distance A is not within the range wherein the steering wheel can be appropriately operated, the operation goes to step S7. In step S7, a driving operation impossible signal is output to the HMI 40 and the automatic vehicle operation control device 50, and thereafter, the processing is ended. The HMI 40, when the driving operation impossible signal is input thereto, for example, performs a display giving an alarm about the driving attitude or seat position on the display section 41, and an announcement giving an alarm about the driving attitude or seat position by the voice output section 42.), wherein:
the displacement data is obtained from at least one of (Fig. 3, illustrates a driver seat that moves away or towards the facing camera that is mounted behind the steering wheel. Paragraph [0089]-HYUGA discloses when estimating the distance Z, changes in size of the face area of the driver in a plurality of images (time-series images) picked up by the monocular camera 11 may be detected so as to decide in which direction, forward or backward, the driver is away from the focal position where the monocular camera 11 focuses, and with use of said decision result and the defocus amount d, the distance Z may be estimated. Further in paragraph [0090]-HYUGA discloses in step S5, with use of the distance Z, the distance A from the steering wheel 32 to the head of the driver 30 is estimated. For example, when the steering wheel 32 is on the line segment between the monocular camera 11 and the driver 30, the distance A is estimated by subtracting the distance B between the monocular camera 11 and the steering wheel 32 from the distance Z. Further read Fig. 6.):
Although, HYUGA explicitly teaches the camera is located in the vehicle. HYUGA fails to explicitly teach wherein the camera is subject to a first coordinate system of the vehicle.
However, UPMANUE explicitly teaches wherein the camera is subject to a first coordinate system of the vehicle (Fig. 1. Paragraph [0013]-UPMANUE discloses the system 100 may include such sensors, such as various cameras, a LIDAR sensor, a radar sensor, an ultrasonic sensor, or other sensor for detecting information about the surroundings of the vehicle, including, for example, other vehicles, lane lines, guard rails, objects in the roadway, buildings, pedestrians, etc. In the example shown in FIG. 1, the system 100 may include an in-vehicle camera 103, a transceiver 105, a vehicle-to-vehicle transceiver 109, a GPS module 113 (wherein the camera in the vehicle has the GPS module #113 first coordinate system, location), an HMI display as well as other sensors, controllers, and modules); and
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA of a system for interior sensing in a vehicle, the system comprising: a camera configured to capture image data relating to an occupant within the vehicle to create captured image data, with the teachings of UPMANUE of wherein the camera is subject to a first coordinate system of the vehicle.
Wherein having HYUGA’s system for interior sensing in a vehicle having wherein the camera is subject to a first coordinate system of the vehicle.
The motivation behind the modification would have been to obtain a system for interior sensing in a vehicle that enhances the location detection of the driver head, face, or eye accurately, since both HYUGA and UPMANUE relate to driver monitoring systems in a vehicle, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while UPMANUE the radar sensors may be utilized to help or enhance various vehicle safety systems. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and UPMANUE et al. (US 20200074197 A1), Paragraph [0029].
HYUGA in view of UPMANUE fail to explicitly teach the second coordinate system is associated with a part of the vehicle that (i) has its own suspension, (ii) moves with the occupant, and (iii) is independently movable relative to a structure of the vehicle, and a sensor, or.
However, SELDEN explicitly teaches the second coordinate system is associated with a part of the vehicle that (i) has its own suspension (Fig. 2. Paragraph [0019]-SELDEN discloses truck seat suspension system 40 is schematically depicted in FIG. 2. System 40 is a non-limiting example of a variable tracking active suspension system of the present disclosure. Truck seat 42 (which together with the driver sitting on the seat is the sprung mass) has accelerometer 44 coupled to it. The active suspension system comprises dynamically adjustable variable force spring 46 and electromagnetic actuator (motor) 48.), (ii) moves with the occupant (Fig. 2. Paragraph [0017]-SELDEN discloses one particular non-limiting example of a sprung mass is a seat for the driver of a truck, with an active suspension that in part is designed to cancel or at least minimize the seat vibrations caused by the running engine and by movement of the truck over roadways), and (iii) is independently movable relative to a structure of the vehicle (Fig. 2. Paragraph [0019]-SELDEN discloses position sensor 50 measures the position of truck seat 42 relative to truck floor 51 (which is the unsprung mass).), and
a sensor (Fig. 2, #50 called position sensor. Paragraph [0019]-SELDEN discloses position sensor 50 measures the position of truck seat 42 relative to truck floor 51 (which is the unsprung mass).), or
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA in view of UPMANUE of a system for interior sensing in a vehicle, the system comprising: a camera configured to capture image data relating to an occupant within the vehicle to create captured image data, with the teachings of SELDEN of the second coordinate system is associated with a part of the vehicle that (i) has its own suspension, (ii) moves with the occupant, and (iii) is independently movable relative to a structure of the vehicle, and a sensor, or.
Wherein having HYUGA’s system for interior sensing in a vehicle having the second coordinate system is associated with a part of the vehicle that (i) has its own suspension, (ii) moves with the occupant, and (iii) is independently movable relative to a structure of the vehicle, and a sensor, or.
The motivation behind the modification would have been to obtain a system for interior sensing in a vehicle that enhances the location detection of the driver head, face, or eye accurately, since both HYUGA and SELDEN relate to sensing and analyzing data related to drivers in a vehicle, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while SELDEN allows the driver to choose an amount of seat motion versus road-induced floor motion that is comfortable, and/or that is best for a particular roadway or a particular driving situation. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and SELDEN et al. (US 20160129749 A1), Paragraph [0015].
HYUGA in view of UPMANUE and further in view of SELDEN fail to explicitly teach processing of image data to track movement of the part of the vehicle moving with the occupant.
However, JARADI explicitly teaches processing of image data (Fig. 4A-4B. Paragraph [0088]-JARADI discloses the seat position sensors 50 may be any suitable sensor in the seat 14 (e.g., rotary encoders, Hall-effect sensors, etc.) or exterior to the seat 14 (including cameras, image sensors, etc.). The computer 34 may receive one or more signals from the seat position sensors 50 indicating the position of the seat 14 along the seat track 58.) to track movement of the part of the vehicle moving with the occupant (Fig. 4A-4B. Paragraph [0048]-JARADI discloses the position of the seats 14 relative to the floor 12 may be adjustable by an occupant. The seat 14 may be selectively slidable relative to the seat track 58 (see FIGS. 4A and 4B). In other words, the occupant may slide the seat 14 along the seat track 58 and may secure the seat 14 to the seat track 58 at selected positions. For example, the occupant may actuate a motor (not shown) that moves the seat 14 along the seat track 58.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA in view of UPMANUE and further in view of SELDEN of a system for interior sensing in a vehicle, the system comprising: a camera configured to capture image data relating to an occupant within the vehicle to create captured image data, with the teachings of JARADI of processing of image data to track movement of the part of the vehicle moving with the occupant.
Wherein having HYUGA’s system for interior sensing in a vehicle having processing of image data to track movement of the part of the vehicle moving with the occupant.
The motivation behind the modification would have been to obtain a system for interior sensing in a vehicle that enhances the location detection of the driver head, face, or eye accurately, since both HYUGA and JARADI relate to sensing an occupant in a seat in a vehicle, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while JARADI improves occupant safety by translating the airbag 18a relative to the track 36a and/or to translate the second airbag 18b relative to the second track 36b. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and JARADI et al. (US 20220388466 A1), Paragraph [0095].
Regarding claim 2, HYUGA in view of UPMANUE and further in view of SELDEN and further in view of JARADI explicitly teach the system of claim 1, further comprising:
HYUGA in view of UPMANUE fail to explicitly teach the sensor from which the displacement data is obtained.
However, SELDEN explicitly teaches the sensor from which the displacement data is obtained (Fig. 2, #50 called position sensor. Paragraph [0019]-SELDEN discloses position sensor 50 measures the position of truck seat 42 relative to truck floor 51 (which is the unsprung mass).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA in view of UPMANUE of a system for interior sensing in a vehicle, the system comprising: a camera configured to capture image data relating to an occupant within the vehicle to create captured image data, with the teachings of SELDEN of the sensor from which the displacement data is obtained.
Wherein having HYUGA’s system for interior sensing in a vehicle having the sensor from which the displacement data is obtained.
The motivation behind the modification would have been to obtain a system for interior sensing in a vehicle that enhances the location detection of the driver head, face, or eye accurately, since both HYUGA and SELDEN relate to sensing and analyzing data related to drivers in a vehicle, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while SELDEN allows the driver to choose an amount of seat motion versus road-induced floor motion that is comfortable, and/or that is best for a particular roadway or a particular driving situation. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and SELDEN et al. (US 20160129749 A1), Paragraph [0015].
Regarding claim 3, HYUGA in view of UPMANUE and further in view of SELDEN and further in view of JARADI explicitly teach the system of claim 2,
HYUGA further explicitly teaches wherein the sensor is located in proximity to the occupant (Fig. 1-3, #11 called a camera. Paragraph [0052]-HYUGA discloses the monocular camera 11 as an imaging section can periodically (e.g. 30-60 times/sec) pick up images including the head of the driver sitting in the driver's seat, and comprises a lens system 11a consisting of one or more lenses, an imaging element 11b such as a CCD (wherein the CCD is a sensor, camera uses a Charge-Coupled Device (CCD) sensor to capture images.) or a CMOS which generates imaging data of a subject, an analog-to-digital conversion section (not shown) which converts the imaging data to digital data, an infrared irradiation unit (not shown) such as a near infrared LED which irradiates near infrared light, and associated parts.).
Regarding claim 7, HYUGA in view of UPMANUE and further in view of SELDEN and further in view of JARADI explicitly teach the system of claim 1,
HYUGA further explicitly teaches wherein the displacement data is time-correlated with the captured image data (Fig. 3, illustrates a driver seat that moves away or towards the facing camera that is mounted behind the steering wheel. Paragraph [0089]-HYUGA discloses when estimating the distance Z, changes in size of the face area of the driver in a plurality of images (time-series images) picked up by the monocular camera 11 may be detected so as to decide in which direction, forward or backward, the driver is away from the focal position where the monocular camera 11 focuses, and with use of said decision result and the defocus amount d, the distance Z may be estimated. Further in paragraph [0090]-HYUGA discloses in step S5, with use of the distance Z, the distance A from the steering wheel 32 to the head of the driver 30 is estimated. For example, when the steering wheel 32 is on the line segment between the monocular camera 11 and the driver 30, the distance A is estimated by subtracting the distance B between the monocular camera 11 and the steering wheel 32 from the distance Z. Further read Fig. 6.).
Regarding claim 8, HYUGA in view of UPMANUE and further in view of SELDEN and further in view of JARADI explicitly teach the system of claim 1,
HYUGA further explicitly teaches wherein an action is generated based on the set of occupant characteristics (Fig. 1-2 and Fig. 6, illustrates the image data captured by the monocular camera #11 is supply to the image storing part #15a and CPU #12 to process the camera data to further send data to an HMI (Human Machine Interface). And S1-7 the camera data is analyzed and controls the HMI #40 to send an alarm about the driver and seat. Paragraph [0054]-HYUGA discloses The CPU 12 is a hardware processor, which reads out a program stored in the ROM 13, and based on said program, performs various kinds of processing on image data picked up by the monocular camera 11. A plurality of CPUs 12 may be mounted for every processing such as image processing or control signal output processing. Further in paragraph [0087]-HYUGA discloses In step S1, data of one or more images picked up by the monocular camera 11 is read from the image storing part 15a, and in step S2, in the read-out one or more images 11c, the head (face) area of the driver 30A is detected. Paragraph [0092-0093]-HYUGA discloses when it is judged that the distance A is not within the range wherein the steering wheel can be appropriately operated, the operation goes to step S7. In step S7, a driving operation impossible signal is output to the HMI 40 and the automatic vehicle operation control device 50, and thereafter, the processing is ended. The HMI 40, when the driving operation impossible signal is input thereto, for example, performs a display giving an alarm about the driving attitude or seat position on the display section 41, and an announcement giving an alarm about the driving attitude or seat position by the voice output section 42.).
Regarding claim 10, HYUGA in view of UPMANUE and further in view of SELDEN and further in view of JARADI explicitly teach a vehicle comprising: the system of claim 1; and
HYUGA further explicitly teaches a seat (Fig. 3, illustrates a driver seat that moves away or towards the facing camera that is mounted behind the steering wheel. Paragraph [0061]), wherein the seat is part of the vehicle (Fig. 3. Paragraph [0061]-HYUGA discloses as shown in FIG. 3, it is a situation in which a driver 30 is sitting in a driver's seat 31. A steering wheel 32 is located in front of the driver's seat 31. The position of the driver's seat 31 can be rearwardly and forwardly adjusted, and the adjustable range of the seat is set to be S. Further in paragraph [0093]-HYUGA discloses the HMI 40, when the driving operation impossible signal is input thereto, for example, performs a display giving an alarm about the driving attitude or seat position on the display section 41, and an announcement giving an alarm about the driving attitude or seat position by the voice output section 42. The automatic vehicle operation control device 50, when the driving operation impossible signal is input thereto, for example, performs speed reduction control (wherein the seat 31 is part of the vehicle).).
Regarding claim 11, HYUGA in view of UPMANUE and further in view of SELDEN and further in view of JARADI explicitly teach the vehicle of claim 10,
Although HYUGA teaches a vehicle with seat, HYUGA in view of UPMANUE fails to explicitly teach wherein the seat includes at least one of a set of seat feature points or a sensor for deriving the displacement data.
However, SELDEN explicitly teaches wherein the seat includes at least one of a set of seat feature points or a sensor for deriving the displacement data (Fig. 2. Paragraph [0019]-SELDEN discloses position sensor 50 measures the position of truck seat 42 relative to truck floor 51 (which is the unsprung mass).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA in view of UPMANUE of a system for interior sensing in a vehicle, the system comprising: a camera configured to capture image data relating to an occupant within the vehicle to create captured image data, with the teachings of SELDEN of wherein the seat includes at least one of a set of seat feature points or a sensor for deriving the displacement data.
Wherein having HYUGA’s system for interior sensing in a vehicle having wherein the seat includes at least one of a set of seat feature points or a sensor for deriving the displacement data.
The motivation behind the modification would have been to obtain a system for interior sensing in a vehicle that enhances the location detection of the driver head, face, or eye accurately, since both HYUGA and SELDEN relate to sensing and analyzing data related to drivers in a vehicle, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while SELDEN allows the driver to choose an amount of seat motion versus road-induced floor motion that is comfortable, and/or that is best for a particular roadway or a particular driving situation. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and SELDEN et al. (US 20160129749 A1), Paragraph [0015].
Claims 4-6 are rejected under 35 U.S.C. 103 as being unpatentable over HYUGA (US 20200065595 A1), hereinafter referenced as HYUGA, in view of UPMANUE et al. (US 20200074197 A1), hereinafter referenced as UPMANUE, and further in view of SELDEN et al. (US 20160129749 A1), hereinafter referenced as SELDEN, and further in view of JARADI et al. (US 20220388466 A1), hereinafter referenced as JARADI, and further in view of IMAGAWA (US 20220215582 A1), hereinafter referenced as IMAGAWA.
Regarding claim 4, HYUGA in view of UPMANUE and further in view of SELDEN and further in view of JARADI explicitly teach the system of claim 1,
HYUGA further explicitly teaches the captured image data (Fig. 3. illustrates a driver seat that moves away or towards the facing camera that is mounted behind the steering wheel. Paragraph [0089]-HYUGA discloses when estimating the distance Z, changes in size of the face area of the driver in a plurality of images (time-series images) picked up by the monocular camera 11 may be detected so as to decide in which direction, forward or backward, the driver is away from the focal position where the monocular camera 11 focuses, and with use of said decision result and the defocus amount d, the distance Z may be estimated. Further in paragraph [0090]-HYUGA discloses in step S5, with use of the distance Z, the distance A from the steering wheel 32 to the head of the driver 30 is estimated. For example, when the steering wheel 32 is on the line segment between the monocular camera 11 and the driver 30, the distance A is estimated by subtracting the distance B between the monocular camera 11 and the steering wheel 32 from the distance Z. Further read Fig. 6.), or
HYUGA in view of UPMANUE and further in view of SELDEN and further in view of JARADI explicitly teach fail to explicitly teach wherein the processing of image data includes processing at least one of: second captured image data collected from a second camera.
However, IMAGAWA explicitly teaches wherein the processing of image data includes processing at least one of (Fig. 1. Paragraph [0066]-IMAGAWA discloses displacement measurement system 1 according to Embodiment 1 is an information processing system that measures an actual value of a displacement of object 60, using two image capturing devices disposed in different positions.):
second captured image data collected from a second camera (Fig. 1. Paragraph [0066]-IMAGAWA discloses displacement measurement system 1 according to Embodiment 1 is an information processing system that measures an actual value of a displacement of object 60, using two image capturing devices disposed in different positions. First image capturing device 10 and second image capturing device 20 capture images of object 60 from different viewpoints).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA in view of UPMANUE and further in view of SELDEN and further in view of JARADI explicitly teach of having a system for interior sensing in a vehicle, the system comprising: a camera configured to capture image data relating to an occupant within the vehicle to create captured image data, wherein the camera is subject to a first coordinate system of the vehicle, with the teachings of IMAGAWA of wherein the processing of image data includes processing at least one of: second captured image data collected from a second camera.
Wherein having HYUGA’s system for interior sensing in a vehicle having wherein the processing of image data includes processing at least one of: second captured image data collected from a second camera.
The motivation behind the modification would have been to obtain a system for interior sensing in a vehicle that enhances the location of the driver head, face eye accurately, since both HYUGA and IMAGAWA relate to object displacement detection, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while IMAGAWA the degree of freedom of selecting a measurement point is enhanced with conversion parameter calculator 110. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and IMAGAWA et al. (US 20220215582 A1), Paragraph [0127].
Regarding claim 5, HYUGA in view of UPMANUE and further in view of SELDEN and further in view of JARADI and further in view of IMAGAWA explicitly teach the system of claim 4,
HYUGA further explicitly teaches wherein: at least one of the camera or the second camera has a wide angle to monitor a seat of a cabin structure of the vehicle, and the seat is the part of the vehicle (Fig. 3, illustrates the camera #11 have a wide angle α to monitor the driver and driver seat #31. Paragraph [0061]-HYUGA discloses a steering wheel 32 is located in front of the driver's seat 31. The position of the driver's seat 31 can be rearwardly and forwardly adjusted, and the adjustable range of the seat is set to be S. The monocular camera 11 is mounted behind the steering wheel 32 (on a steering column, or at the front of a dashboard or an instrument panel, none of them shown), that is, on a place where images 11c including a head (face) of the driver 30A can be picked up thereby.).
Regarding claim 6, HYUGA in view of UPMANUE and further in view of SELDEN and further in view of JARADI and further in view of IMAGAWA explicitly teach the system of claim 5,
HYUGA further explicitly teaches wherein the displacement data is obtained from monitoring movement of the seat (Fig. 3, illustrates a driver seat that moves away or towards the facing camera that is mounted behind the steering wheel. Paragraph [0061]-HYUGA discloses as shown in FIG. 3, it is a situation in which a driver 30 is sitting in a driver's seat 31. A steering wheel 32 is located in front of the driver's seat 31. The position of the driver's seat 31 can be rearwardly and forwardly adjusted, and the adjustable range of the seat is set to be S. The monocular camera 11 is mounted behind the steering wheel 32 (on a steering column, or at the front of a dashboard or an instrument panel, none of them shown), that is, on a place where images 11c including a head (face) of the driver 30A can be picked up thereby.).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over HYUGA (US 20200065595 A1), hereinafter referenced as HYUGA, in view of UPMANUE et al. (US 20200074197 A1), hereinafter referenced as UPMANUE, and further in view of SELDEN et al. (US 20160129749 A1), hereinafter referenced as SELDEN, and further in view of JARADI et al. (US 20220388466 A1), hereinafter referenced as JARADI, and further in view of CHANDUPATLA (US 20210318135 A1), hereinafter referenced as CHANDUPATLA and further in view of PASZKOWICZ (US 20170043712 A1), hereinafter referenced as PASZKOWICZ.
Regarding claim 9, HYUGA in view of UPMANUE and further in view of SELDEN and further in view of JARADI explicitly teach the system of claim 8,
HYUGA further explicitly teach wherein the action includes at least one of: generating at least one of an audible alert, a visual alert, or a tactile alert associated with an occupant straying outside predetermined safety parameters (Fig. 1-6. Paragraph [0093]-HYUGA discloses in step S7, a driving operation impossible signal is output to the HMI 40 and the automatic vehicle operation control device 50, and thereafter, the processing is ended. The HMI 40, when the driving operation impossible signal is input thereto, for example, performs a display giving an alarm (Wherein display giving an alarm is a visual alert) about the driving attitude or seat position on the display section 41, and an announcement giving an alarm about the driving attitude or seat position by the voice output section 42. The automatic vehicle operation control device 50, when the driving operation impossible signal is input thereto, for example, performs speed reduction control.);
HYUGA in view of UPMANUE and further in view of SELDEN and further in view of JARADI fail to explicitly teach modifying an augmented reality display/projection to correct for driver's perspective.
However, CHANDUPATLA explicitly teaches modifying an augmented reality display/projection to correct for driver's perspective (Fig. 8-9. Paragraph [0081]-CHANDUPATLA discloses in some examples, the augmentations may [be] displayed in a fixed region of the windshield that is assumed to be out of the driver FOV when the driver is looking straight ahead at the road. In other examples, the augmentations may be displayed based on the actual driver FOV, such that the display coordinates of the augmentations may change based on vehicle orientation (e.g., which may indicate the vehicle is turning or traversing a curve, and thus the driver FOV may have shifted) or the gaze direction of the driver.).; and
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA in view of UPMANUE and further in view of SELDEN and further in view of JARADI of having a system for interior sensing in a vehicle, the system comprising: a camera configured to capture image data relating to an occupant within a vehicle, wherein the camera is subject to a first coordinate system of the vehicle, having wherein the action includes at least one of: generating at least one of an audible alert, a visual alert, or a tactile alert associated with an occupant straying outside predetermined safety parameters, with the teachings of CHANDUPATLA modifying an augmented reality display/projection to correct for driver's perspective
Wherein having HYUGA’s system of actions wherein modifying an augmented reality display/projection to correct for driver's perspective
The motivation behind the modification would have been to obtain a system for interior sensing in a vehicle that enhances the location of the driver head, face eye accurately, safety and occupant experience, since both HYUGA and CHANDUPATLA relate to driver monitoring system in a vehicle, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while CHANDUPATLA shows displaying augmentations on a windshield is that vehicle occupants may be informed of upcoming road conditions, points of interest, communication outages, and the like, which may improve vehicle safety and/or occupant experience with the vehicle. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and CHANDUPATLA (US 20210318135 A1), Paragraph [0103].
HYUGA in view of UPMANUE and further in view of SELDEN and further in view of JARADI and further in view of CHANDUPATLA fail to explicitly teach activating a cabin illumination based on driver's line of sight.
However, PASZKOWICZ explicitly teaches activating a cabin illumination based on driver's line of sight (Fig. 1. Paragraph [0048]- PASZKOWICZ discloses for example, if the driver D scans the interior of the cabin C and the driver's gaze passes several visual zones (and hence the virtual projection PV intersects several associated areas of interest An), each visual zone will be highlighted in turn as the driver's gaze crosses that area.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA in view of UPMANUE and further in view of SELDEN and further in view of JARADI and further in view of CHANDUPATLA of having a system for interior sensing in a vehicle, the system comprising: a camera configured to capture image data relating to an occupant within a vehicle, wherein the camera is subject to a first coordinate system of the vehicle, wherein the action includes at least one of: generating at least one of an audible alert, a visual alert, or a tactile alert associated with an occupant straying outside predetermined safety parameters, modifying an augmented reality display/projection to correct for driver's perspective with the teachings of PASZKOWICZ activating a cabin illumination based on driver's line of sight
Wherein having HYUGA’s system for interior sensing in a vehicle wherein activating a cabin illumination based on driver's line of sight
The motivation behind the modification would have been to obtain a system for interior sensing in a vehicle that enhances the location of the driver head, face eye accurately, safety and occupant experience, since both HYUGA and PASZKOWICZ relate to driver monitoring system in a vehicle, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while PASZKOWICZ the selective highlighting can enhance functionality of the vehicle and improve the vehicle occupant's visual acuity. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and PASZKOWICZ (US 20170043712 A1), Paragraph [0009].
Claims 12-16 are rejected under 35 U.S.C. 103 as being unpatentable over HYUGA (US 20200065595 A1), hereinafter referenced as HYUGA, in view of SELDEN et al. (US 20160129749 A1), hereinafter referenced as SELDEN, and further in view of JARADI et al. (US 20220388466 A1), hereinafter referenced as JARADI.
Regarding claim 12, HYUGA teaches a method comprising (Fig. 2. Paragraph [0047]-HYUGA discloses the embodiments of the driver state estimation device and the driver state estimation method according to the present invention are described below by reference to the Figures.):
capturing image data relating to an occupant within a vehicle to create captured image data (Fig. 1-3. Paragraph [0052]-HYUGA discloses the monocular camera 11 as an imaging section can periodically (e.g. 30-60 times/sec) pick up images including the head of the driver sitting in the driver's seat, and comprises a lens system 11a consisting of one or more lenses, an imaging element 11b such as a CCD or a CMOS which generates imaging data of a subject, an analog-to-digital conversion section (not shown) which converts the imaging data to digital data.);
identifying feature points of the occupant (Fig. 1-3, #30 called a driver. Paragraph [0061]) from the captured image data (Fig. 1-6. Paragraph [0054]-HYUGA discloses the CPU 12 is a hardware processor, which reads out a program stored in the ROM 13, and based on said program, performs various kinds of processing on image data picked up by the monocular camera 11 (wherein the raw feature points data is the camera data.). Further in paragraph [0069]-HYUGA discloses the driver state estimation device 10 is established as a device wherein various kinds of programs stored in the ROM 13 are read into the RAM 14 and conducted by the CPU 12, so as to perform processing as the storage instructing section 21, reading instructing section 22, head detecting section 23, defocus amount detecting section 24, distance estimating section 25, and driving operation possibility deciding section 26.);
generating raw feature points data (Fig. 1-6. Paragraph [0054]-HYUGA discloses The CPU 12 is a hardware processor, which reads out a program stored in the ROM 13, and based on said program, performs various kinds of processing on image data picked up by the monocular camera 11 (wherein the raw feature points data is the camera data.). Further in paragraph [0069]-HYUGA discloses the driver state estimation device 10 is established as a device wherein various kinds of programs stored in the ROM 13 are read into the RAM 14 and conducted by the CPU 12, so as to perform processing as the storage instructing section 21, reading instructing section 22, head detecting section 23, defocus amount detecting section 24, distance estimating section 25, and driving operation possibility deciding section 26.);
compensating the raw feature points data with displacement data time- correlated with the captured image data (Fig. 3, illustrates a driver seat that moves away or towards the facing camera that is mounted behind the steering wheel. Paragraph [0089]-HYUGA discloses when estimating the distance Z, changes in size of the face area of the driver in a plurality of images (time-series images) picked up by the monocular camera 11 may be detected so as to decide in which direction, forward or backward, the driver is away from the focal position where the monocular camera 11 focuses, and with use of said decision result and the defocus amount d, the distance Z may be estimated. Please also read Paragraph [0064]);
generating processed result data (Fig. 3, Paragraph [0090]-HYUGA discloses in step S5, with use of the distance Z, the distance A from the steering wheel 32 to the head of the driver 30 is estimated. For example, when the steering wheel 32 is on the line segment between the monocular camera 11 and the driver 30, the distance A is estimated by subtracting the distance B between the monocular camera 11 and the steering wheel 32 from the distance Z. Further read Fig. 6.); and
determining, based on the processed result data, a set of occupant characteristics indicative of behavior of the occupant (Fig. 1-2 and Fig. 6, illustrates the image data captured by the monocular camera #11 is supply to the image storing part #15a and CPU #12 to process the camera data to further send data to an HMI (Human Machine Interface). And S1-7 the camera data is analyzed and controls the HMI #40 to send an alarm about the driver and seat. Paragraph [0054]-HYUGA discloses The CPU 12 is a hardware processor, which reads out a program stored in the ROM 13, and based on said program, performs various kinds of processing on image data picked up by the monocular camera 11. A plurality of CPUs 12 may be mounted for every processing such as image processing or control signal output processing. Further in paragraph [0087]-HYUGA discloses in step S1, data of one or more images picked up by the monocular camera 11 is read from the image storing part 15a, and in step S2, in the read-out one or more images 11c, the head (face) area of the driver 30A is detected. Paragraph [0092-0093]-HYUGA discloses when it is judged that the distance A is not within the range wherein the steering wheel can be appropriately operated, the operation goes to step S7. In step S7, a driving operation impossible signal is output to the HMI 40 and the automatic vehicle operation control device 50, and thereafter, the processing is ended. The HMI 40, when the driving operation impossible signal is input thereto, for example, performs a display giving an alarm about the driving attitude or seat position on the display section 41, and an announcement giving an alarm about the driving attitude or seat position by the voice output section 42.), wherein:
HYUGA fails to explicitly teach the displacement data is correlated with a second coordinate system that is associated with a part of the vehicle that (i) has its own suspension, (ii) moves with the occupant, and (iii) is independently movable relative to a structure of the vehicle, and the displacement data is obtained from at least one of a sensor or.
However, SELDEN explicitly teaches the displacement data is correlated with a second coordinate system that is associated with a part of the vehicle that (i) has its own suspension (Fig. 2. Paragraph [0019]-SELDEN discloses truck seat suspension system 40 is schematically depicted in FIG. 2. System 40 is a non-limiting example of a variable tracking active suspension system of the present disclosure. Truck seat 42 (which together with the driver sitting on the seat is the sprung mass) has accelerometer 44 coupled to it. The active suspension system comprises dynamically adjustable variable force spring 46 and electromagnetic actuator (motor) 48.), (ii) moves with the occupant (Fig. 2. Paragraph [0017]-SELDEN discloses one particular non-limiting example of a sprung mass is a seat for the driver of a truck, with an active suspension that in part is designed to cancel or at least minimize the seat vibrations caused by the running engine and by movement of the truck over roadways), and (iii) is independently movable relative to a structure of the vehicle (Fig. 2. Paragraph [0019]-SELDEN discloses position sensor 50 measures the position of truck seat 42 relative to truck floor 51 (which is the unsprung mass).), and
the displacement data is obtained from at least one of a sensor (Fig. 2, #50 called position sensor. Paragraph [0019]-SELDEN discloses position sensor 50 measures the position of truck seat 42 relative to truck floor 51 (which is the unsprung mass).) or.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA of a method comprising: capturing image data relating to an occupant within a vehicle to create captured image data; identifying feature points of the occupant from the captured image data; generating raw feature points data; compensating the raw feature points data with displacement data time- correlated with the captured image data; generating processed result data; and determining, based on the processed result data, a set of occupant characteristics indicative of behavior of the occupant, wherein:, with the teachings of SELDEN of the displacement data is correlated with a second coordinate system that is associated with a part of the vehicle that (i) has its own suspension, (ii) moves with the occupant, and (iii) is independently movable relative to a structure of the vehicle, and the displacement data is obtained from at least one of a sensor or.
Wherein having HYUGA’s system for interior sensing in a vehicle having the displacement data is correlated with a second coordinate system that is associated with a part of the vehicle that (i) has its own suspension, (ii) moves with the occupant, and (iii) is independently movable relative to a structure of the vehicle, and the displacement data is obtained from at least one of a sensor or.
The motivation behind the modification would have been to obtain a system for interior sensing in a vehicle that enhances the location detection of the driver head, face, or eye accurately, since both HYUGA and SELDEN relate to sensing and analyzing data related to drivers in a vehicle, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while SELDEN allows the driver to choose an amount of seat motion versus road-induced floor motion that is comfortable, and/or that is best for a particular roadway or a particular driving situation. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and SELDEN et al. (US 20160129749 A1), Paragraph [0015].
HYUGA in view of SELDEN fail to explicitly teach processing of image data to track movement of the part of the vehicle moving with the occupant.
However, JARADI explicitly teaches processing of image data (Fig. 4A-4B. Paragraph [0088]-JARADI discloses the seat position sensors 50 may be any suitable sensor in the seat 14 (e.g., rotary encoders, Hall-effect sensors, etc.) or exterior to the seat 14 (including cameras, image sensors, etc.). The computer 34 may receive one or more signals from the seat position sensors 50 indicating the position of the seat 14 along the seat track 58.) to track movement of the part of the vehicle moving with the occupant (Fig. 4A-4B. Paragraph [0048]-JARADI discloses the position of the seats 14 relative to the floor 12 may be adjustable by an occupant. The seat 14 may be selectively slidable relative to the seat track 58 (see FIGS. 4A and 4B). In other words, the occupant may slide the seat 14 along the seat track 58 and may secure the seat 14 to the seat track 58 at selected positions. For example, the occupant may actuate a motor (not shown) that moves the seat 14 along the seat track 58.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA in view of SELDEN of a method comprising: capturing image data relating to an occupant within a vehicle to create captured image data; identifying feature points of the occupant from the captured image data; generating raw feature points data; compensating the raw feature points data with displacement data time- correlated with the captured image data; generating processed result data; and determining, based on the processed result data, a set of occupant characteristics indicative of behavior of the occupant, wherein:, with the teachings of JARADI of processing of image data to track movement of the part of the vehicle moving with the occupant.
Wherein having HYUGA’s system for interior sensing in a vehicle having processing of image data to track movement of the part of the vehicle moving with the occupant.
The motivation behind the modification would have been to obtain a system for interior sensing in a vehicle that enhances the location detection of the driver head, face, or eye accurately, since both HYUGA and JARADI relate to sensing an occupant in a seat in a vehicle, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while JARADI improves occupant safety by translating the airbag 18a relative to the track 36a and/or to translate the second airbag 18b relative to the second track 36b. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and JARADI et al. (US 20220388466 A1), Paragraph [0095].
Regarding claim 13, HYUGA in view of SELDEN and further in view of JADADI explicitly teach the method of claim 12,
HYUGA further explicitly teach wherein the displacement data is obtained from a sensor (Fig. 1-3, #11 called a camera. Paragraph [0052]-HYUGA discloses the monocular camera 11 as an imaging section can periodically (e.g. 30-60 times/sec) pick up images including the head of the driver sitting in the driver's seat, and comprises a lens system 11a consisting of one or more lenses, an imaging element 11b such as a CCD (wherein the CCD is a sensor, camera uses a Charge-Coupled Device (CCD) sensor to capture images.) or a CMOS which generates imaging data of a subject, an analog-to-digital conversion section (not shown) which converts the imaging data to digital data, an infrared irradiation unit (not shown) such as a near infrared LED which irradiates near infrared light, and associated parts. Fig. 3, illustrates a driver seat that moves away or towards the facing camera that is mounted behind the steering wheel. Paragraph [0089]-HYUGA discloses when estimating the distance Z, changes in size of the face area of the driver in a plurality of images (time-series images) picked up by the monocular camera 11 may be detected so as to decide in which direction, forward or backward, the driver is away from the focal position where the monocular camera 11 focuses, and with use of said decision result and the defocus amount d, the distance Z may be estimated. Further in paragraph [0090]-HYUGA discloses in step S5, with use of the distance Z, the distance A from the steering wheel 32 to the head of the driver 30 is estimated. For example, when the steering wheel 32 is on the line segment between the monocular camera 11 and the driver 30, the distance A is estimated by subtracting the distance B between the monocular camera 11 and the steering wheel 32 from the distance Z. Further read Fig. 6.).
Regarding claim 14, HYUGA in view of SELDEN and further in view of JADADI explicitly teach the method of claim 12,
HYUGA further teaches wherein the displacement data is obtained from monitoring movement of a seat in the vehicle (Fig. 3, illustrates a driver seat that moves away or towards the facing camera that is mounted behind the steering wheel. Paragraph [0061]-HYUGA discloses as shown in FIG. 3, it is a situation in which a driver 30 is sitting in a driver's seat 31. A steering wheel 32 is located in front of the driver's seat 31. The position of the driver's seat 31 can be rearwardly and forwardly adjusted, and the adjustable range of the seat is set to be S. The monocular camera 11 is mounted behind the steering wheel 32 (on a steering column, or at the front of a dashboard or an instrument panel, none of them shown), that is, on a place where images 11c including a head (face) of the driver 30A can be picked up thereby.).
Regarding claim 15, HYUGA in view of SELDEN and further in view of JADADI explicitly teach the method of claim 12,
HYUGA further teaches wherein an action is generated dependent on the set of occupant characteristics (Fig. 1-2 and Fig. 6, illustrates the image data captured by the monocular camera #11 is supply to the image storing part #15a and CPU #12 to process the camera data to further send data to an HMI (Human Machine Interface). And S1-7 the camera data is analyzed and controls the HMI #40 to send an alarm about the driver and seat. Paragraph [0054]-HYUGA discloses The CPU 12 is a hardware processor, which reads out a program stored in the ROM 13, and based on said program, performs various kinds of processing on image data picked up by the monocular camera 11. A plurality of CPUs 12 may be mounted for every processing such as image processing or control signal output processing. Further in paragraph [0087]-HYUGA discloses in step S1, data of one or more images picked up by the monocular camera 11 is read from the image storing part 15a, and in step S2, in the read-out one or more images 11c, the head (face) area of the driver 30A is detected. Paragraph [0092-0093]-HYUGA discloses when it is judged that the distance A is not within the range wherein the steering wheel can be appropriately operated, the operation goes to step S7. In step S7, a driving operation impossible signal is output to the HMI 40 and the automatic vehicle operation control device 50, and thereafter, the processing is ended. The HMI 40, when the driving operation impossible signal is input thereto, for example, performs a display giving an alarm about the driving attitude or seat position on the display section 41, and an announcement giving an alarm about the driving attitude or seat position by the voice output section 42.).
Regarding claim 16, HYUGA explicitly teaches a non-transitory computer-readable medium (Fig. 1-2. Paragraph [0054-56]-HYUGA discloses the CPU 12 is a hardware processor, which reads out a program stored in the ROM 13, and based on said program, performs various kinds of processing on image data picked up by the monocular camera 11. A plurality of CPUs 12 may be mounted for every processing such as image processing or control signal output processing. In the ROM 13, programs for allowing the CPU 12 to perform processing as a storage instructing section 21, a reading instructing section 22, the head detecting section 23, a defocus amount detecting section 24, a distance estimating section 25, and a driving operation possibility deciding section 26 shown in FIG. 2, and the like are stored. All or part of the programs performed by the CPU 12 may be stored in the storage section 15 or a storing medium (not shown) other than the ROM 13. In the RAM 14, data required for various kinds of processing performed by the CPU 12, programs read from the ROM 13, and the like are temporarily stored) comprising instructions including:
capturing image data relating to an occupant within a vehicle to create captured image data(Fig. 1-3. Paragraph [0052]-HYUGA discloses the monocular camera 11 as an imaging section can periodically (e.g. 30-60 times/sec) pick up images including the head of the driver sitting in the driver's seat, and comprises a lens system 11a consisting of one or more lenses, an imaging element 11b such as a CCD or a CMOS which generates imaging data of a subject, an analog-to-digital conversion section (not shown) which converts the imaging data to digital data.);
identifying feature points of the occupant (Fig. 1-3, #30 called a driver. Paragraph [0061]) from the captured image data (Fig. 1-6. Paragraph [0054]-HYUGA discloses the CPU 12 is a hardware processor, which reads out a program stored in the ROM 13, and based on said program, performs various kinds of processing on image data picked up by the monocular camera 11 (wherein the raw feature points data is the camera data.). Further in paragraph [0069]-HYUGA discloses the driver state estimation device 10 is established as a device wherein various kinds of programs stored in the ROM 13 are read into the RAM 14 and conducted by the CPU 12, so as to perform processing as the storage instructing section 21, reading instructing section 22, head detecting section 23, defocus amount detecting section 24, distance estimating section 25, and driving operation possibility deciding section 26.);
generating raw feature points data (Fig. 1-6. Paragraph [0054]-HYUGA discloses The CPU 12 is a hardware processor, which reads out a program stored in the ROM 13, and based on said program, performs various kinds of processing on image data picked up by the monocular camera 11 (wherein the raw feature points data is the camera data.). Further in paragraph [0069]-HYUGA discloses the driver state estimation device 10 is established as a device wherein various kinds of programs stored in the ROM 13 are read into the RAM 14 and conducted by the CPU 12, so as to perform processing as the storage instructing section 21, reading instructing section 22, head detecting section 23, defocus amount detecting section 24, distance estimating section 25, and driving operation possibility deciding section 26.);
compensating the raw feature points data with displacement data time- correlated with the captured image data (Fig. 3, illustrates a driver seat that moves away or towards the facing camera that is mounted behind the steering wheel. Paragraph [0089]-HYUGA discloses when estimating the distance Z, changes in size of the face area of the driver in a plurality of images (time-series images) picked up by the monocular camera 11 may be detected so as to decide in which direction, forward or backward, the driver is away from the focal position where the monocular camera 11 focuses, and with use of said decision result and the defocus amount d, the distance Z may be estimated. Please also read Paragraph [0064]);
generating processed result data (Fig. 3, Paragraph [0090]-HYUGA discloses in step S5, with use of the distance Z, the distance A from the steering wheel 32 to the head of the driver 30 is estimated. For example, when the steering wheel 32 is on the line segment between the monocular camera 11 and the driver 30, the distance A is estimated by subtracting the distance B between the monocular camera 11 and the steering wheel 32 from the distance Z. Further read Fig. 6.); and
determining, based on the processed result data, a set of occupant characteristics indicative of behavior of the occupant (Fig. 1-2 and Fig. 6, illustrates the image data captured by the monocular camera #11 is supply to the image storing part #15a and CPU #12 to process the camera data to further send data to an HMI (Human Machine Interface). And S1-7 the camera data is analyzed and controls the HMI #40 to send an alarm about the driver and seat. Paragraph [0054]-HYUGA discloses The CPU 12 is a hardware processor, which reads out a program stored in the ROM 13, and based on said program, performs various kinds of processing on image data picked up by the monocular camera 11. A plurality of CPUs 12 may be mounted for every processing such as image processing or control signal output processing. Further in paragraph [0087]-HYUGA discloses in step S1, data of one or more images picked up by the monocular camera 11 is read from the image storing part 15a, and in step S2, in the read-out one or more images 11c, the head (face) area of the driver 30A is detected. Paragraph [0092-0093]-HYUGA discloses when it is judged that the distance A is not within the range wherein the steering wheel can be appropriately operated, the operation goes to step S7. In step S7, a driving operation impossible signal is output to the HMI 40 and the automatic vehicle operation control device 50, and thereafter, the processing is ended. The HMI 40, when the driving operation impossible signal is input thereto, for example, performs a display giving an alarm about the driving attitude or seat position on the display section 41, and an announcement giving an alarm about the driving attitude or seat position by the voice output section 42.), wherein:
HYUGA fails to explicitly teach the displacement data is correlated with a second coordinate system that is associated with a part of the vehicle that (i) has its own suspension, (ii) moves with the occupant, and (iii) is independently movable relative to a structure of the vehicle; and the displacement data is obtained from at least one of a sensor or.
However, SELDEN explicitly teaches the displacement data is correlated with a second coordinate system that is associated with a part of the vehicle that (i) has its own suspension (Fig. 2. Paragraph [0019]-SELDEN discloses truck seat suspension system 40 is schematically depicted in FIG. 2. System 40 is a non-limiting example of a variable tracking active suspension system of the present disclosure. Truck seat 42 (which together with the driver sitting on the seat is the sprung mass) has accelerometer 44 coupled to it. The active suspension system comprises dynamically adjustable variable force spring 46 and electromagnetic actuator (motor) 48.), (ii) moves with the occupant (Fig. 2. Paragraph [0017]-SELDEN discloses one particular non-limiting example of a sprung mass is a seat for the driver of a truck, with an active suspension that in part is designed to cancel or at least minimize the seat vibrations caused by the running engine and by movement of the truck over roadways), and (iii) is independently movable relative to a structure of the vehicle (Fig. 2. Paragraph [0019]-SELDEN discloses position sensor 50 measures the position of truck seat 42 relative to truck floor 51 (which is the unsprung mass).); and
the displacement data is obtained from at least one of a sensor (Fig. 2, #50 called position sensor. Paragraph [0019]-SELDEN discloses position sensor 50 measures the position of truck seat 42 relative to truck floor 51 (which is the unsprung mass).) or.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA of a non-transitory computer-readable medium comprising instructions including: capturing image data relating to an occupant within a vehicle to create captured image data; identifying feature points of the occupant from the captured image data; generating raw feature points data; compensating the raw feature points data with displacement data time- correlated with the captured image data; generating processed result data; and determining, based on the processed result data, a set of occupant characteristics indicative of behavior of the occupant, wherein, with the teachings of SELDEN of the displacement data is correlated with a second coordinate system that is associated with a part of the vehicle that (i) has its own suspension, (ii) moves with the occupant, and (iii) is independently movable relative to a structure of the vehicle; and the displacement data is obtained from at least one of a sensor or.
Wherein having HYUGA’s system for interior sensing in a vehicle having the displacement data is correlated with a second coordinate system that is associated with a part of the vehicle that (i) has its own suspension, (ii) moves with the occupant, and (iii) is independently movable relative to a structure of the vehicle; and the displacement data is obtained from at least one of a sensor or.
The motivation behind the modification would have been to obtain a system for interior sensing in a vehicle that enhances the location detection of the driver head, face, or eye accurately, since both HYUGA and SELDEN relate to sensing and analyzing data related to drivers in a vehicle, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while SELDEN allows the driver to choose an amount of seat motion versus road-induced floor motion that is comfortable, and/or that is best for a particular roadway or a particular driving situation. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and SELDEN et al. (US 20160129749 A1), Paragraph [0015].
HYUGA in view of SELDEN fail to explicitly teach processing of image data to track movement of the part of the vehicle moving with the occupant.
However, JARADI explicitly teaches processing of image data (Fig. 4A-4B. Paragraph [0088]-JARADI discloses the seat position sensors 50 may be any suitable sensor in the seat 14 (e.g., rotary encoders, Hall-effect sensors, etc.) or exterior to the seat 14 (including cameras, image sensors, etc.). The computer 34 may receive one or more signals from the seat position sensors 50 indicating the position of the seat 14 along the seat track 58.) to track movement of the part of the vehicle moving with the occupant (Fig. 4A-4B. Paragraph [0048]-JARADI discloses the position of the seats 14 relative to the floor 12 may be adjustable by an occupant. The seat 14 may be selectively slidable relative to the seat track 58 (see FIGS. 4A and 4B). In other words, the occupant may slide the seat 14 along the seat track 58 and may secure the seat 14 to the seat track 58 at selected positions. For example, the occupant may actuate a motor (not shown) that moves the seat 14 along the seat track 58.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA in view of SELDEN of a non-transitory computer-readable medium comprising instructions including: capturing image data relating to an occupant within a vehicle to create captured image data; identifying feature points of the occupant from the captured image data; generating raw feature points data; compensating the raw feature points data with displacement data time- correlated with the captured image data; generating processed result data; and determining, based on the processed result data, a set of occupant characteristics indicative of behavior of the occupant, wherein, with the teachings of JARADI of processing of image data to track movement of the part of the vehicle moving with the occupant.
Wherein having HYUGA’s system for interior sensing in a vehicle having processing of image data to track movement of the part of the vehicle moving with the occupant.
The motivation behind the modification would have been to obtain a system for interior sensing in a vehicle that enhances the location detection of the driver head, face, or eye accurately, since both HYUGA and JARADI relate to sensing an occupant in a seat in a vehicle, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while JARADI improves occupant safety by translating the airbag 18a relative to the track 36a and/or to translate the second airbag 18b relative to the second track 36b. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and JARADI et al. (US 20220388466 A1), Paragraph [0095].
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant’s disclosure.
O'BOYLE et al. (US 20050111700 A1) - A camera (32) captures successive images of light stripes (22) projected onto an object by a light curtain (18) positioned by a light source positioner (28). A background image is subtracted (616) therefrom, and the resulting image is boosted by binning (618), binarized with a thresholding algorithm (620), skeletonized (622), interpolated (624) and stored (626). Interpolated images are acquired for a plurality of light stripes (22). A processor (30) generates (1604) a 3-D surface model from Cartesian coordinates computed for non-zero camera pixels. A volumetric representation is determined (1610) from the offset of the object surface model relative to a model of a proximate surface, e.g. a seating surface (24). The object is classified (1614), e.g. by a trainable pattern recognitions system, responsive to 3-D shape descriptors (1606) of the 3-D surface model and to the volumetric representation (1610) or portions (1612) thereof. The detection of an occupant (14) in a vehicle (12) may be used to control a safety restraint system (36)…Abstract, Fig. 11.
NONG (US 20200317018 A1) - The present disclosure relates to autonomous driving vehicles and methods for improving stability and occupant comfort of the same. The vehicle includes: a frame member; a cabin, movable with respect to and independent from the frame member; wheels; at least one suspension between the wheels and frame member; actuation device configured to control at least the orientation of the cabin with respect to the frame member; a perception module comprising perception sensors and algorithm configured to at least identify road boundaries and obstacles in the vicinity of the vehicle; and a planning module configured to plan the motions of the steering means using information from at least the perception module…Abstract, Fig. 1.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ETHAN N WOLFSON whose telephone number is (571)272-1898. The examiner can normally be reached Monday - Friday 8:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ETHAN N WOLFSON/Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673
Prosecution Insights