DETAILED ACTION
Notice of AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers submitted under 35 U.S.C. 119(a)-(d), which papers have been placed of record in the file.
Response to Amendment
Applicant’s remarks filed 03/02/2026 regarding the objections made to the claims submitted in the non-final office action dated 12/04/2025 are withdrawn due to the amendments made to the claims.
Response to Arguments
Applicant’s arguments, see remarks, filed 03/02/2026, with respect to claims 1-16 regarding the prior art rejection under 35 U.S.C. 102 and 35 U.S.C. 103, have been considered, but are moot because the arguments do not apply to the current reference, or combinations of references being used in the current rejection.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 7-8, and 10-11 are rejected under 35 U.S.C. 103 as being unpatentable over HYUGA (US 20200065595 A1), hereinafter referenced as HYUGA, in view of UPMANUE et al. (US 20200074197 A1), hereinafter referenced as UPMANUE, and further in view of AVILA et al. (US 20230179744 A1), hereinafter referenced as AVILA, and further in view of ABHAU et al. (US 20190279008 A1), hereinafter referenced as ABHAU.
Regarding claim 1, HYUGA teaches a system for interior sensing in a vehicle (Fig. 1. Paragraph [0049]), the system comprising:
a camera (Fig. 1, #11 called a monocular camera. Paragraph [0050]) configured to capture image data relating to an occupant within the vehicle to create captured image data (Fig. 1-3. Paragraph [0052]-HYUGA discloses the monocular camera 11 as an imaging section can periodically (e.g. 30-60 times/sec) pick up images including the head of the driver sitting in the driver's seat, and comprises a lens system 11a consisting of one or more lenses, an imaging element 11b such as a CCD or a CMOS which generates imaging data of a subject, an analog-to-digital conversion section (not shown) which converts the imaging data to digital data.),
at least one processor (Fig. 1-2 and Fig. 6, #12 called a CPU. Paragraph [0054]) configured to:
identify feature points of the occupant (Fig. 1-3, #30 called a driver. Paragraph [0061]) from the captured image data and generate raw feature points data (Fig. 1-6. Paragraph [0054]-HYUGA discloses the CPU 12 is a hardware processor, which reads out a program stored in the ROM 13, and based on said program, performs various kinds of processing on image data picked up by the monocular camera 11 (wherein the raw feature points data is the camera data.). Further in paragraph [0069]-HYUGA discloses the driver state estimation device 10 is established as a device wherein various kinds of programs stored in the ROM 13 are read into the RAM 14 and conducted by the CPU 12, so as to perform processing as the storage instructing section 21, reading instructing section 22, head detecting section 23, defocus amount detecting section 24, distance estimating section 25, and driving operation possibility deciding section 26.);
compensate the raw feature points data with displacement data correlated with a second coordinate system (Fig. 3, illustrates a driver seat that moves away or towards the facing camera that is mounted behind the steering wheel. Paragraph [0089]-HYUGA discloses when estimating the distance Z, changes in size of the face area of the driver in a plurality of images (time-series images) picked up by the monocular camera 11 may be detected so as to decide in which direction, forward or backward, the driver is away from the focal position where the monocular camera 11 focuses, and with use of said decision result and the defocus amount d, the distance Z may be estimated. Further in paragraph [0090]-HYUGA discloses in step S5, with use of the distance Z, the distance A from the steering wheel 32 to the head of the driver 30 is estimated. For example, when the steering wheel 32 is on the line segment between the monocular camera 11 and the driver 30, the distance A is estimated by subtracting the distance B between the monocular camera 11 and the steering wheel 32 from the distance Z. Further read Fig. 6.);
generate processed result data (Fig. 6. Paragraph [0090]-HYUGA discloses in step S5, with use of the distance Z, the distance A from the steering wheel 32 to the head of the driver 30 is estimated. For example, when the steering wheel 32 is on the line segment between the monocular camera 11 and the driver 30, the distance A is estimated by subtracting the distance B between the monocular camera 11 and the steering wheel 32 from the distance Z (wherein distance A is result data).); and
determine, based on the processed result data, a set of occupant characteristics indicative of behavior of the occupant (Fig. 1-2 and Fig. 6, illustrates the image data captured by the monocular camera #11 is the supply to the image storing part #15a and CPU #12 to process the camera data to further send data to an HMI (Human Machine Interface), and S1-7 the camera data is analyzed and controls the HMI #40 to send an alarm about the driver and seat. Paragraph [0054]-HYUGA discloses the CPU 12 is a hardware processor, which reads out a program stored in the ROM 13, and based on said program, performs various kinds of processing on image data picked up by the monocular camera 11. A plurality of CPUs 12 may be mounted for every processing such as image processing or control signal output processing. Further in paragraph [0087]-HYUGA discloses in step S1, data of one or more images picked up by the monocular camera 11 is read from the image storing part 15a, and in step S2, in the read-out one or more images 11c, the head (face) area of the driver 30A is detected. Paragraph [0092-0093]-HYUGA discloses when it is judged that the distance A is not within the range wherein the steering wheel can be appropriately operated, the operation goes to step S7. In step S7, a driving operation impossible signal is output to the HMI 40 and the automatic vehicle operation control device 50, and thereafter, the processing is ended. The HMI 40, when the driving operation impossible signal is input thereto, for example, performs a display giving an alarm about the driving attitude or seat position on the display section 41, and an announcement giving an alarm about the driving attitude or seat position by the voice output section 42.), wherein:
the displacement data is obtained from at least one of (Fig. 3, illustrates a driver seat that moves away or towards the facing camera that is mounted behind the steering wheel. Paragraph [0089]-HYUGA discloses when estimating the distance Z, changes in size of the face area of the driver in a plurality of images (time-series images) picked up by the monocular camera 11 may be detected so as to decide in which direction, forward or backward, the driver is away from the focal position where the monocular camera 11 focuses, and with use of said decision result and the defocus amount d, the distance Z may be estimated. Further in paragraph [0090]-HYUGA discloses in step S5, with use of the distance Z, the distance A from the steering wheel 32 to the head of the driver 30 is estimated. For example, when the steering wheel 32 is on the line segment between the monocular camera 11 and the driver 30, the distance A is estimated by subtracting the distance B between the monocular camera 11 and the steering wheel 32 from the distance Z. Further read Fig. 6.):
Although, HYUGA explicitly teaches the camera is located in the vehicle. HYUGA fails to explicitly teach wherein the camera is subject to a first coordinate system of the vehicle.
However, UPMANUE explicitly teaches wherein the camera is subject to a first coordinate system of the vehicle (Fig. 1. Paragraph [0013]-UPMANUE discloses the system 100 may include such sensors, such as various cameras, a LIDAR sensor, a radar sensor, an ultrasonic sensor, or other sensor for detecting information about the surroundings of the vehicle, including, for example, other vehicles, lane lines, guard rails, objects in the roadway, buildings, pedestrians, etc. In the example shown in FIG. 1, the system 100 may include an in-vehicle camera 103, a transceiver 105, a vehicle-to-vehicle transceiver 109, a GPS module 113 (wherein the camera in the vehicle has the GPS module #113 first coordinate system, location), an HMI display as well as other sensors, controllers, and modules); and
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA of a system for interior sensing in a vehicle, the system comprising: a camera configured to capture image data relating to an occupant within the vehicle to create captured image data, with the teachings of UPMANUE of wherein the camera is subject to a first coordinate system of the vehicle.
Wherein having HYUGA’s system for interior sensing in a vehicle wherein the camera is subject to a first coordinate system of the vehicle.
The motivation behind the modification would have been to obtain a system for interior sensing in a vehicle that enhances the location detection of the driver head, face, or eye accurately, since both HYUGA and UPMANUE relate to driver monitoring systems in a vehicle, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while UPMANUE the radar sensors may be utilized to help or enhance various vehicle safety systems. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and UPMANUE et al. (US 20200074197 A1), Paragraph [0029].
HYUGA in view of UPMANUE fail to explicitly teach the second coordinate system is associated with a part of the vehicle moving with the occupant and independently movable relative to a structure of the vehicle, and a sensor.
However, AVILA explicitly teaches the second coordinate system is associated with a part of the vehicle moving with the occupant and independently movable relative to a structure of the vehicle (Fig. 1. Paragraph [0016]-Avila discloses a system 10 for tracking a child car seat 12 (wherein a car seat is a portable seat that attaches to a seat of a vehicle—i.e. a part of the vehicle moving with the occupant and independently movable relative to the vehicle) according to the present invention includes a seat sensing device 14 that is equipped with one or more proximity sensors 20. The seat sensing device 14 can communicate wirelessly with a vehicle onboard GPS tracker 16, which can communicate with a vehicle onboard camera 18. The seat sensing device 14 also includes a Bluetooth tracker or tag (also referred to as a key finder or smart tracker), which is a small electronic device that allows a user to monitor the location of an item to which it is attached.), and
a sensor (Fig. 1. Paragraph [0016]-Avila discloses a system 10 for tracking a child car seat 12 (wherein the car seat is a portable seat that attaches to a seat of a vehicle) according to the present invention includes a seat sensing device 14 that is equipped with one or more proximity sensors 20. The seat sensing device 14 can communicate wirelessly with a vehicle onboard GPS tracker 16, which can communicate with a vehicle onboard camera 18. The seat sensing device 14 also includes a Bluetooth tracker or tag (also referred to as a key finder or smart tracker), which is a small electronic device that allows a user to monitor the location of an item to which it is attached. Further in paragraph [0017]-Avila discloses a proximity sensor is a sensor that can detect the presence of nearby objects without any physical contact. In some embodiment, the proximity sensors can, for example, be optical proximity sensors. An optical proximity sensor is a device that uses the principle of triangulation of reflected infrared or visible light to measure small distances in robotic systems. Such sensors can be utilized to check for the child's presence in the car seat 12 by measuring the distance from the sensor to the child and calibrating that distance to avoid false positive readings.), or
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA in view of UPMANUE of having a system for interior sensing in a vehicle, the system comprising: a camera configured to capture image data relating to an occupant within the vehicle to create captured image data, wherein the camera is subject to a first coordinate system of the vehicle, with the teachings of AVILA of the second coordinate system is associated with a part of the vehicle moving with the occupant and independently movable relative to a structure of the vehicle, and a sensor.
Wherein having HYUGA’s system for interior sensing in a vehicle the second coordinate system is associated with a part of the vehicle moving with the occupant and independently movable relative to a structure of the vehicle, and a sensor.
The motivation behind the modification would have been to obtain system for interior sensing in a vehicle that enhances occupant in seat even when ignition is off, the location of the driver head, face eye accurately, safety and occupant experience, since both HYUGA and AVILA relate to driver monitoring system in a vehicle, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while AVILA the seat sensing device is configured so that if the proximity sensor senses a child's presence in the car seat when the vehicle ignition is off, the seat sensing device will send an alert signal to the GPS tracker. In response to the alert signal, the GPS tracker is configured to cause the camera to send an alert to a user, capture an image of the car seat in the vehicle, and send the image to the user. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and AVILA et al. (US 20230179744 A1), Abstract.
HYUGA in view of UPMANUE and further in view of AVILA fail to explicitly teach processing of image data to track movement of the part of the vehicle moving with the occupant.
However, ABHAU explicitly teaches processing of image data to track movement of the part of the vehicle moving with the occupant (Fig. 8 and 10. Paragraph [0026-0027]-ABHAU discloses the 3D reference model can comprise seats, tables, interior walls, fixtures, etc. The reference model preferably also takes the vehicle interior into account resulting from rotating, moving, or tilting the seats and other static interior elements of the vehicle. As a result, when a vehicle seat has been moved from a reference position or the backrest has been adjusted, this can be derived from a change in the three dimensional coordinates of a seat determined from the camera images. Further in paragraph [0056]-ABHAU discloses FIG. 8 shows an example in which two vehicle seats S1 and S2 defined in the reference model are identified by comparing the collection of three dimensional coordinates of pixels identified in the correlation process with the reference model. A displacement Dis of the vehicle seat S1 from its standard position S1sdt can be detected from the deviations of the three dimensional coordinates S1 of the first seat from the camera images. In the same manner, an adjustment of the backrests of seats S1 and S2 from their standard tilt can also be detected (wherein the seat is the part of the vehicle moving with the occupant).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA in view of UPMANUE and further in view of AVILA of having a system for interior sensing in a vehicle, the system comprising: a camera configured to capture image data relating to an occupant within the vehicle to create captured image data, wherein the camera is subject to a first coordinate system of the vehicle, with the teachings of ABHAU of the second coordinate system is associated with a part of the vehicle moving with the occupant and independently movable relative to a structure of the vehicle, and a sensor.
Wherein having HYUGA’s system for interior sensing in a vehicle the second coordinate system is associated with a part of the vehicle moving with the occupant and independently movable relative to a structure of the vehicle, and a sensor.
The motivation behind the modification would have been to obtain system for interior sensing in a vehicle that enhances occupant in seat even when ignition is off, the location of the driver head, face eye accurately, safety and occupant experience, since both HYUGA and ABHAU relate to interior monitoring systems in a vehicle, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while ABHAU the image can be shown to the driver, for example, such that said driver can easily see that the vehicle interior is in a safe state. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and ABHAU et al. (US 20190279008 A1), Paragraph [0057].
Regarding claim 2, HYUGA in view of UPMANUE and further in view of AVILA and further in view of ABHAU explicitly teach the system of claim 1, further comprising:
HYUGA in view of UPMANUE fail to explicitly teach the sensor from which the displacement data is obtained.
However, AVILA explicitly teaches the sensor from which the displacement data is obtained (Fig. 1. Paragraph [0016]-Avila discloses a system 10 for tracking a child car seat 12 (wherein the car seat is a portable seat that attaches to a seat of a vehicle) according to the present invention includes a seat sensing device 14 that is equipped with one or more proximity sensors 20. The seat sensing device 14 can communicate wirelessly with a vehicle onboard GPS tracker 16, which can communicate with a vehicle onboard camera 18. The seat sensing device 14 also includes a Bluetooth tracker or tag (also referred to as a key finder or smart tracker), which is a small electronic device that allows a user to monitor the location of an item to which it is attached. Further in paragraph [0017]-Avila discloses a proximity sensor is a sensor that can detect the presence of nearby objects without any physical contact. In some embodiment, the proximity sensors can, for example, be optical proximity sensors. An optical proximity sensor is a device that uses the principle of triangulation of reflected infrared or visible light to measure small distances in robotic systems. Such sensors can be utilized to check for the child's presence in the car seat 12 by measuring the distance from the sensor to the child and calibrating that distance to avoid false positive readings.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA in view of UPMANUE of having a system for interior sensing in a vehicle, the system comprising: a camera configured to capture image data relating to an occupant within the vehicle to create captured image data, wherein the camera is subject to a first coordinate system of the vehicle, with the teachings of AVILA of the sensor from which the displacement data is obtained.
Wherein having HYUGA’s system for interior sensing in a vehicle the sensor from which the displacement data is obtained.
The motivation behind the modification would have been to obtain system for interior sensing in a vehicle that enhances occupant in seat even when ignition is off, the location of the driver head, face eye accurately, safety and occupant experience, since both HYUGA and AVILA relate to driver monitoring system in a vehicle, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while AVILA the seat sensing device is configured so that if the proximity sensor senses a child's presence in the car seat when the vehicle ignition is off, the seat sensing device will send an alert signal to the GPS tracker. In response to the alert signal, the GPS tracker is configured to cause the camera to send an alert to a user, capture an image of the car seat in the vehicle, and send the image to the user. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and AVILA et al. (US 20230179744 A1), Abstract.
Regarding claim 3, HYUGA in view of UPMANUE and further in view of AVILA and further in view of ABHAU explicitly teach the system of claim 2,
HYUGA further explicitly teaches wherein the sensor is located in proximity to the occupant (Fig. 1-3, #11 called a camera. Paragraph [0052]-HYUGA discloses the monocular camera 11 as an imaging section can periodically (e.g. 30-60 times/sec) pick up images including the head of the driver sitting in the driver's seat, and comprises a lens system 11a consisting of one or more lenses, an imaging element 11b such as a CCD (wherein the CCD is a sensor, camera uses a Charge-Coupled Device (CCD) sensor to capture images.) or a CMOS which generates imaging data of a subject, an analog-to-digital conversion section (not shown) which converts the imaging data to digital data, an infrared irradiation unit (not shown) such as a near infrared LED which irradiates near infrared light, and associated parts.).
Regarding claim 7, HYUGA in view of UPMANUE and further in view of AVILA and further in view of ABHAU explicitly teach the system of claim 1, HYUGA further explicitly teaches wherein the displacement data is time-correlated with the captured image data (Fig. 3, illustrates a driver seat that moves away or towards the facing camera that is mounted behind the steering wheel. Paragraph [0089]-HYUGA discloses when estimating the distance Z, changes in size of the face area of the driver in a plurality of images (time-series images) picked up by the monocular camera 11 may be detected so as to decide in which direction, forward or backward, the driver is away from the focal position where the monocular camera 11 focuses, and with use of said decision result and the defocus amount d, the distance Z may be estimated. Further in paragraph [0090]-HYUGA discloses in step S5, with use of the distance Z, the distance A from the steering wheel 32 to the head of the driver 30 is estimated. For example, when the steering wheel 32 is on the line segment between the monocular camera 11 and the driver 30, the distance A is estimated by subtracting the distance B between the monocular camera 11 and the steering wheel 32 from the distance Z. Further read Fig. 6.).
Regarding claim 8, HYUGA in view of UPMANUE and further in view of AVILA and further in view of ABHAU explicitly teach the system of claim 1,
HYUGA further explicitly teaches wherein an action is generated based on the set of occupant characteristics (Fig. 1-2 and Fig. 6, illustrates the image data captured by the monocular camera #11 is supply to the image storing part #15a and CPU #12 to process the camera data to further send data to an HMI (Human Machine Interface). And S1-7 the camera data is analyzed and controls the HMI #40 to send an alarm about the driver and seat. Paragraph [0054]-HYUGA discloses The CPU 12 is a hardware processor, which reads out a program stored in the ROM 13, and based on said program, performs various kinds of processing on image data picked up by the monocular camera 11. A plurality of CPUs 12 may be mounted for every processing such as image processing or control signal output processing. Further in paragraph [0087]-HYUGA discloses In step S1, data of one or more images picked up by the monocular camera 11 is read from the image storing part 15a, and in step S2, in the read-out one or more images 11c, the head (face) area of the driver 30A is detected. Paragraph [0092-0093]-HYUGA discloses when it is judged that the distance A is not within the range wherein the steering wheel can be appropriately operated, the operation goes to step S7. In step S7, a driving operation impossible signal is output to the HMI 40 and the automatic vehicle operation control device 50, and thereafter, the processing is ended. The HMI 40, when the driving operation impossible signal is input thereto, for example, performs a display giving an alarm about the driving attitude or seat position on the display section 41, and an announcement giving an alarm about the driving attitude or seat position by the voice output section 42.).
Regarding claim 10, HYUGA in view of UPMANUE and further in view of AVILA and further in view of ABHAU explicitly teach a vehicle comprising: the system of claim 1; and
HYUGA further explicitly teaches a seat (Fig. 3, illustrates a driver seat that moves away or towards the facing camera that is mounted behind the steering wheel. Paragraph [0061]), wherein the seat is (Fig. 3. Paragraph [0061]-HYUGA discloses as shown in FIG. 3, it is a situation in which a driver 30 is sitting in a driver's seat 31. A steering wheel 32 is located in front of the driver's seat 31. The position of the driver's seat 31 can be rearwardly and forwardly adjusted, and the adjustable range of the seat is set to be S.):
the part of the vehicle (Fig. 3. Paragraph [0061]-HYUGA discloses as shown in FIG. 3, it is a situation in which a driver 30 is sitting in a driver's seat 31. A steering wheel 32 is located in front of the driver's seat 31. The position of the driver's seat 31 can be rearwardly and forwardly adjusted, and the adjustable range of the seat is set to be S. Further in paragraph [0093]-HYUGA discloses the HMI 40, when the driving operation impossible signal is input thereto, for example, performs a display giving an alarm about the driving attitude or seat position on the display section 41, and an announcement giving an alarm about the driving attitude or seat position by the voice output section 42. The automatic vehicle operation control device 50, when the driving operation impossible signal is input thereto, for example, performs speed reduction control (wherein the seat 31 is part of the vehicle).):,
Although HYUGA teaches a seat, HYUGA in view of UPMANUE fails to explicitly teach associated with the second coordinate systems, and independently suspended from the structure of the vehicle.
However, AVILA explicitly teaches associated with the second coordinate systems (Fig. 1. Paragraph [0016]-Avila discloses a system 10 for tracking a child car seat 12 according to the present invention includes a seat sensing device 14 that is equipped with one or more proximity sensors 20. The seat sensing device 14 can communicate wirelessly with a vehicle onboard GPS tracker 16, which can communicate with a vehicle onboard camera 18. The seat sensing device 14 also includes a Bluetooth tracker or tag (also referred to as a key finder or smart tracker), which is a small electronic device that allows a user to monitor the location of an item to which it is attached. Further in paragraph [0017]-Avila discloses a proximity sensor is a sensor that can detect the presence of nearby objects without any physical contact. In some embodiment, the proximity sensors can, for example, be optical proximity sensors. An optical proximity sensor is a device that uses the principle of triangulation of reflected infrared or visible light to measure small distances in robotic systems. Such sensors can be utilized to check for the child's presence in the car seat 12 by measuring the distance from the sensor to the child and calibrating that distance to avoid false positive readings.), and
independently suspended from the structure of the vehicle (Fig. 1. Paragraph [0016]-Avila discloses a system 10 for tracking a child car seat 12 (wherein the car seat is a portable seat that attaches to a seat of a vehicle—i.e. a seat independently suspended from the structure of the vehicle) according to the present invention includes a seat sensing device 14 that is equipped with one or more proximity sensors 20. The seat sensing device 14 can communicate wirelessly with a vehicle onboard GPS tracker 16, which can communicate with a vehicle onboard camera 18. The seat sensing device 14 also includes a Bluetooth tracker or tag (also referred to as a key finder or smart tracker), which is a small electronic device that allows a user to monitor the location of an item to which it is attached. Further in paragraph [0017]-Avila discloses a proximity sensor is a sensor that can detect the presence of nearby objects without any physical contact. In some embodiment, the proximity sensors can, for example, be optical proximity sensors. An optical proximity sensor is a device that uses the principle of triangulation of reflected infrared or visible light to measure small distances in robotic systems. Such sensors can be utilized to check for the child's presence in the car seat 12 by measuring the distance from the sensor to the child and calibrating that distance to avoid false positive readings.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA in view of UPMANUE of having a system for interior sensing in a vehicle, the system comprising: a camera configured to capture image data relating to an occupant within a vehicle, wherein the camera is subject to a first coordinate system of the vehicle, with the teachings of AVILA having associated with the second coordinate systems, and independently suspended from the structure of the vehicle.
Wherein having HYUGA’s system for interior sensing in a vehicle associated with the second coordinate systems, and independently suspended from the structure of the vehicle.
The motivation behind the modification would have been to obtain system for interior sensing in a vehicle that enhances occupant in seat even when ignition is off, the location of the driver head, face eye accurately, safety and occupant experience, since both HYUGA and AVILA relate to driver monitoring system in a vehicle, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while AVILA the seat sensing device is configured so that if the proximity sensor senses a child's presence in the car seat when the vehicle ignition is off, the seat sensing device will send an alert signal to the GPS tracker. In response to the alert signal, the GPS tracker is configured to cause the camera to send an alert to a user, capture an image of the car seat in the vehicle, and send the image to the user. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and AVILA et al. (US 20230179744 A1), Abstract.
Regarding claim 11, HYUGA in view of UPMANUE and further in view of AVILA and further in view of ABHAU explicitly teach the vehicle of claim 10,
Although HYUGA teaches a vehicle with seat, HYUGA in view of UPMANUE fails to explicitly teach wherein the seat includes at least one of a set of seat feature points or a sensor for deriving the displacement data.
However, AVILA explicitly teaches wherein the seat includes at least one of a set of seat feature points or a sensor for deriving the displacement data (Fig. 1. Paragraph [0016]-Avila discloses a system 10 for tracking a child car seat 12 (wherein the car seat is a portable seat that attaches to a seat of a vehicle) according to the present invention includes a seat sensing device 14 that is equipped with one or more proximity sensors 20. The seat sensing device 14 can communicate wirelessly with a vehicle onboard GPS tracker 16, which can communicate with a vehicle onboard camera 18. The seat sensing device 14 also includes a Bluetooth tracker or tag (also referred to as a key finder or smart tracker), which is a small electronic device that allows a user to monitor the location of an item to which it is attached. Further in paragraph [0017]-Avila discloses a proximity sensor is a sensor that can detect the presence of nearby objects without any physical contact. In some embodiment, the proximity sensors can, for example, be optical proximity sensors. An optical proximity sensor is a device that uses the principle of triangulation of reflected infrared or visible light to measure small distances in robotic systems. Such sensors can be utilized to check for the child's presence in the car seat 12 by measuring the distance from the sensor to the child and calibrating that distance to avoid false positive readings.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA in view of UPMANUE of having a system for interior sensing in a vehicle, the system comprising: a camera configured to capture image data relating to an occupant within a vehicle, wherein the camera is subject to a first coordinate system of the vehicle, with the teachings of AVILA wherein the seat includes at least one of a set of seat feature points or a sensor for deriving the displacement data
Wherein having HYUGA’s system for interior sensing in a vehicle wherein a seat associated with the second coordinate system and independently suspended from a structure of the vehicle.
The motivation behind the modification would have been to obtain system for interior sensing in a vehicle that enhances occupant in seat even when ignition is off, the location of the driver head, face eye accurately, safety and occupant experience, since both HYUGA and AVILA relate to driver monitoring system in a vehicle, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while AVILA the seat sensing device is configured so that if the proximity sensor senses a child's presence in the car seat when the vehicle ignition is off, the seat sensing device will send an alert signal to the GPS tracker. In response to the alert signal, the GPS tracker is configured to cause the camera to send an alert to a user, capture an image of the car seat in the vehicle, and send the image to the user. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and AVILA et al. (US 20230179744 A1), Abstract.
Claims 4-6 are rejected under 35 U.S.C. 103 as being unpatentable over HYUGA (US 20200065595 A1), hereinafter referenced as HYUGA, in view of UPMANUE et al. (US 20200074197 A1), hereinafter referenced as UPMANUE, and further in view of AVILA et al. (US 20230179744 A1), hereinafter referenced as AVILA, and further in view of ABHAU et al. (US 20190279008 A1), hereinafter referenced as ABHAU, and further in view of IWAGAWA (US 20220215582 A1), hereinafter referenced as IWAGAWA.
Regarding claim 4, HYUGA in view of UPMANUE and further in view of AVILA and further in view of ABHAU explicitly teach the system of claim 1,
HYUGA further explicitly teaches the captured image data (Fig. 3. illustrates a driver seat that moves away or towards the facing camera that is mounted behind the steering wheel. Paragraph [0089]-HYUGA discloses when estimating the distance Z, changes in size of the face area of the driver in a plurality of images (time-series images) picked up by the monocular camera 11 may be detected so as to decide in which direction, forward or backward, the driver is away from the focal position where the monocular camera 11 focuses, and with use of said decision result and the defocus amount d, the distance Z may be estimated. Further in paragraph [0090]-HYUGA discloses in step S5, with use of the distance Z, the distance A from the steering wheel 32 to the head of the driver 30 is estimated. For example, when the steering wheel 32 is on the line segment between the monocular camera 11 and the driver 30, the distance A is estimated by subtracting the distance B between the monocular camera 11 and the steering wheel 32 from the distance Z. Further read Fig. 6.), or
HYUGA in view of UPMANUE fails to explicitly teach wherein the processing of image data includes processing at least one of:
However, ABHAU explicitly teaches wherein the processing of image data includes processing at least one of: (Fig. 8 and 10. Paragraph [0026-0027]-ABHAU discloses the 3D reference model can comprise seats, tables, interior walls, fixtures, etc. The reference model preferably also takes the vehicle interior into account resulting from rotating, moving, or tilting the seats and other static interior elements of the vehicle. As a result, when a vehicle seat has been moved from a reference position or the backrest has been adjusted, this can be derived from a change in the three dimensional coordinates of a seat determined from the camera images. Further in paragraph [0056]-ABHAU discloses FIG. 8 shows an example in which two vehicle seats S1 and S2 defined in the reference model are identified by comparing the collection of three dimensional coordinates of pixels identified in the correlation process with the reference model. A displacement Dis of the vehicle seat S1 from its standard position S1sdt can be detected from the deviations of the three dimensional coordinates S1 of the first seat from the camera images. In the same manner, an adjustment of the backrests of seats S1 and S2 from their standard tilt can also be detected (wherein the seat is the part of the vehicle moving with the occupant).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA in view of UPMANUE and further in view of AVILA of having a system for interior sensing in a vehicle, the system comprising: a camera configured to capture image data relating to an occupant within the vehicle to create captured image data, wherein the camera is subject to a first coordinate system of the vehicle, with the teachings of ABHAU of wherein the processing of image data includes processing at least one of:
Wherein having HYUGA’s system for interior sensing in a vehicle wherein the processing of image data includes processing at least one of:
The motivation behind the modification would have been to obtain system for interior sensing in a vehicle that enhances occupant in seat even when ignition is off, the location of the driver head, face eye accurately, safety and occupant experience, since both HYUGA and ABHAU relate to interior monitoring systems in a vehicle, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while ABHAU the image can be shown to the driver, for example, such that said driver can easily see that the vehicle interior is in a safe state. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and ABHAU et al. (US 20190279008 A1), Paragraph [0057].
Although, UPMANUE teaches the use of multiple cameras (Fig. 1. Paragraph [0028])
HYUGA in view of UPMANUE and further in view of AVILA and further in view of ABHAU fail to explicitly teach second captured image data collected from a second camera.
However, IMAGAWA explicitly teaches second captured image data collected from a second camera (Fig. 1-2. Paragraph [0070]-IMAGAWA discloses a second image capturing device 20 includes an image capturer for capturing a second image and a distance measurer for measuring a distance.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of UPMANUE and further in view of AVILA and further in view of ABHAU of having a system for interior sensing in a vehicle, the system comprising: a camera configured to capture image data relating to an occupant within the vehicle to create captured image data, wherein the camera is subject to a first coordinate system of the vehicle, with the teachings of IWAGAWA of second captured image data collected from a second camera.
Wherein having HYUGA’s system for interior sensing in a vehicle second captured image data collected from a second camera.
The motivation behind the modification would have been to obtain a system for interior sensing in a vehicle that enhances the location of the driver head, face eye accurately, since both HYUGA and IMAGAWA relate to object displacement detection, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while IMAGAWA the degree of freedom of selecting a measurement point is enhanced with conversion parameter calculator 110. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and IWAGAWA (US 20220215582 A1), Paragraph [0127].
Regarding claim 5, HYUGA in view of UPMANUE and further in view of AVILA and further in view of ABHAU and further in view IMAGAWA explicitly teach the system of claim 4,
HYUGA further explicitly teaches wherein at least one of the camera or the second camera has a wide angle to monitor a seat of a cabin structure of the vehicle (Fig. 3, illustrates the camera #11 have a wide angle α to monitor the driver and driver seat #31. Paragraph [0062]).
Regarding claim 6, HYUGA in view of UPMANUE and further in view of AVILA and further in view of ABHAU and further in view IMAGAWA explicitly teach the system of claim 5,
HYUGA further explicitly teaches wherein the displacement data is obtained from monitoring movement of the seat (Fig. 3, illustrates a driver seat that moves away or towards the facing camera that is mounted behind the steering wheel. Paragraph [0061]-HYUGA discloses as shown in FIG. 3, it is a situation in which a driver 30 is sitting in a driver's seat 31. A steering wheel 32 is located in front of the driver's seat 31. The position of the driver's seat 31 can be rearwardly and forwardly adjusted, and the adjustable range of the seat is set to be S. The monocular camera 11 is mounted behind the steering wheel 32 (on a steering column, or at the front of a dashboard or an instrument panel, none of them shown), that is, on a place where images 11c including a head (face) of the driver 30A can be picked up thereby.).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over HYUGA (US 20200065595 A1), hereinafter referenced as HYUGA, in view of UPMANUE et al. (US 20200074197 A1), hereinafter referenced as UPMANUE, and further in view of AVILA et al. (US 20230179744 A1), hereinafter referenced as AVILA, and further in view of ABHAU et al. (US 20190279008 A1), hereinafter referenced as ABHAU, and further in view of CHANDUPATLA (US 20210318135 A1), hereinafter referenced as CHANDUPATLA and further in view of PASZKOWICZ (US 20170043712 A1), hereinafter referenced as PASZKOWICZ.
Regarding claim 9, HYUGA in view of UPMANUE and further in view of AVILA and further in view of ABHAU explicitly teach the system of claim 8,
HYUGA further explicitly teach wherein the action includes at least one of: generating at least one of an audible alert, a visual alert, or a tactile alert associated with an occupant straying outside predetermined safety parameters (Fig. 1-6. Paragraph [0093]-HYUGA discloses in step S7, a driving operation impossible signal is output to the HMI 40 and the automatic vehicle operation control device 50, and thereafter, the processing is ended. The HMI 40, when the driving operation impossible signal is input thereto, for example, performs a display giving an alarm (Wherein display giving an alarm is a visual alert) about the driving attitude or seat position on the display section 41, and an announcement giving an alarm about the driving attitude or seat position by the voice output section 42. The automatic vehicle operation control device 50, when the driving operation impossible signal is input thereto, for example, performs speed reduction control.);
HYUGA in view of UPMANUE and further in view of AVILA and further in view of ABHAU fail to explicitly teach modifying an augmented reality display/projection to correct for driver's perspective.
However, CHANDUPATLA explicitly teaches modifying an augmented reality display/projection to correct for driver's perspective (Fig. 8-9. Paragraph [0081]-CHANDUPATLA discloses in some examples, the augmentations may [be] displayed in a fixed region of the windshield that is assumed to be out of the driver FOV when the driver is looking straight ahead at the road. In other examples, the augmentations may be displayed based on the actual driver FOV, such that the display coordinates of the augmentations may change based on vehicle orientation (e.g., which may indicate the vehicle is turning or traversing a curve, and thus the driver FOV may have shifted) or the gaze direction of the driver.).; and
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA in view of UPMANUE and further in view of AVILA and further in view of ABHAU of having a system for interior sensing in a vehicle, the system comprising: a camera configured to capture image data relating to an occupant within a vehicle, wherein the camera is subject to a first coordinate system of the vehicle, having wherein the action includes at least one of: generating at least one of an audible alert, a visual alert, or a tactile alert associated with an occupant straying outside predetermined safety parameters, with the teachings of CHANDUPATLA modifying an augmented reality display/projection to correct for driver's perspective
Wherein having HYUGA in view of UPMANUE’s system of actions wherein modifying an augmented reality display/projection to correct for driver's perspective
The motivation behind the modification would have been to obtain a system for interior sensing in a vehicle that enhances the location of the driver head, face eye accurately, safety and occupant experience, since both HYUGA and CHANDUPATLA relate to driver monitoring system in a vehicle, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while CHANDUPATLA shows displaying augmentations on a windshield is that vehicle occupants may be informed of upcoming road conditions, points of interest, communication outages, and the like, which may improve vehicle safety and/or occupant experience with the vehicle. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and CHANDUPATLA (US 20210318135 A1), Paragraph [0103].
HYUGA in view of UPMANUE and further in view of AVILA and further in view of ABHAU and further in view of CHANDUPATLA fail to explicitly teach activating a cabin illumination based on driver's line of sight.
However, PASZKOWICZ explicitly teaches activating a cabin illumination based on driver's line of sight (Fig. 1. Paragraph [0048]- PASZKOWICZ discloses for example, if the driver D scans the interior of the cabin C and the driver's gaze passes several visual zones (and hence the virtual projection PV intersects several associated areas of interest An), each visual zone will be highlighted in turn as the driver's gaze crosses that area.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA in view of UPMANUE and further in view of AVILA and further in view of ABHAU and further in view of CHANDUPATLA of having a system for interior sensing in a vehicle, the system comprising: a camera configured to capture image data relating to an occupant within a vehicle, wherein the camera is subject to a first coordinate system of the vehicle, wherein the action includes at least one of: generating at least one of an audible alert, a visual alert, or a tactile alert associated with an occupant straying outside predetermined safety parameters, modifying an augmented reality display/projection to correct for driver's perspective with the teachings of PASZKOWICZ activating a cabin illumination based on driver's line of sight
Wherein having HYUGA’s system for interior sensing in a vehicle wherein activating a cabin illumination based on driver's line of sight
The motivation behind the modification would have been to obtain a system for interior sensing in a vehicle that enhances the location of the driver head, face eye accurately, safety and occupant experience, since both HYUGA and PASZKOWICZ relate to driver monitoring system in a vehicle, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while PASZKOWICZ the selective highlighting can enhance functionality of the vehicle and improve the vehicle occupant's visual acuity. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and PASZKOWICZ (US 20170043712 A1), Paragraph [0009].
Claims 12-16 are rejected under 35 U.S.C. 103 as being unpatentable over HYUGA (US 20200065595 A1), hereinafter referenced as HYUGA, in view of AVILA et al. (US 20230179744 A1), hereinafter referenced as AVILA, and further in view of ABHAU et al. (US 20190279008 A1), hereinafter referenced as ABHAU.
Regarding claim 12, HYUGA teaches a method comprising:
capturing image data relating to an occupant within a vehicle to create captured image data (Fig. 1-3. Paragraph [0052]-HYUGA discloses the monocular camera 11 as an imaging section can periodically (e.g. 30-60 times/sec) pick up images including the head of the driver sitting in the driver's seat, and comprises a lens system 11a consisting of one or more lenses, an imaging element 11b such as a CCD or a CMOS which generates imaging data of a subject, an analog-to-digital conversion section (not shown) which converts the imaging data to digital data.);
identifying feature points of the occupant (Fig. 1-3, #30 called a driver. Paragraph [0061]) from the captured image data (Fig. 1-6. Paragraph [0054]-HYUGA discloses the CPU 12 is a hardware processor, which reads out a program stored in the ROM 13, and based on said program, performs various kinds of processing on image data picked up by the monocular camera 11 (wherein the raw feature points data is the camera data.). Further in paragraph [0069]-HYUGA discloses the driver state estimation device 10 is established as a device wherein various kinds of programs stored in the ROM 13 are read into the RAM 14 and conducted by the CPU 12, so as to perform processing as the storage instructing section 21, reading instructing section 22, head detecting section 23, defocus amount detecting section 24, distance estimating section 25, and driving operation possibility deciding section 26.);
generating raw feature points data (Fig. 1-6. Paragraph [0054]-HYUGA discloses The CPU 12 is a hardware processor, which reads out a program stored in the ROM 13, and based on said program, performs various kinds of processing on image data picked up by the monocular camera 11 (wherein the raw feature points data is the camera data.). Further in paragraph [0069]-HYUGA discloses the driver state estimation device 10 is established as a device wherein various kinds of programs stored in the ROM 13 are read into the RAM 14 and conducted by the CPU 12, so as to perform processing as the storage instructing section 21, reading instructing section 22, head detecting section 23, defocus amount detecting section 24, distance estimating section 25, and driving operation possibility deciding section 26.);
compensating the raw feature points data with displacement data time- correlated with the captured image data (Fig. 3, illustrates a driver seat that moves away or towards the facing camera that is mounted behind the steering wheel. Paragraph [0089]-HYUGA discloses when estimating the distance Z, changes in size of the face area of the driver in a plurality of images (time-series images) picked up by the monocular camera 11 may be detected so as to decide in which direction, forward or backward, the driver is away from the focal position where the monocular camera 11 focuses, and with use of said decision result and the defocus amount d, the distance Z may be estimated. Please also read Paragraph [0064]);
generating processed result data (Fig. 3, Paragraph [0090]-HYUGA discloses in step S5, with use of the distance Z, the distance A from the steering wheel 32 to the head of the driver 30 is estimated. For example, when the steering wheel 32 is on the line segment between the monocular camera 11 and the driver 30, the distance A is estimated by subtracting the distance B between the monocular camera 11 and the steering wheel 32 from the distance Z. Further read Fig. 6.); and
determining, based on the processed result data, a set of occupant characteristics indicative of behavior of the occupant (Fig. 1-2 and Fig. 6, illustrates the image data captured by the monocular camera #11 is supply to the image storing part #15a and CPU #12 to process the camera data to further send data to an HMI (Human Machine Interface). And S1-7 the camera data is analyzed and controls the HMI #40 to send an alarm about the driver and seat. Paragraph [0054]-HYUGA discloses The CPU 12 is a hardware processor, which reads out a program stored in the ROM 13, and based on said program, performs various kinds of processing on image data picked up by the monocular camera 11. A plurality of CPUs 12 may be mounted for every processing such as image processing or control signal output processing. Further in paragraph [0087]-HYUGA discloses in step S1, data of one or more images picked up by the monocular camera 11 is read from the image storing part 15a, and in step S2, in the read-out one or more images 11c, the head (face) area of the driver 30A is detected. Paragraph [0092-0093]-HYUGA discloses when it is judged that the distance A is not within the range wherein the steering wheel can be appropriately operated, the operation goes to step S7. In step S7, a driving operation impossible signal is output to the HMI 40 and the automatic vehicle operation control device 50, and thereafter, the processing is ended. The HMI 40, when the driving operation impossible signal is input thereto, for example, performs a display giving an alarm about the driving attitude or seat position on the display section 41, and an announcement giving an alarm about the driving attitude or seat position by the voice output section 42.), wherein:
HYUGA fails to explicitly teach the displacement data is correlated with a second coordinate system that is associated with a part of the vehicle moving with the occupant and independently movable relative to a structure of the vehicle, and the displacement data is obtained from at least one of a sensor.
However, AVILA explicitly teaches the displacement data is correlated with a second coordinate system that is associated with a part of the vehicle moving with the occupant and independently movable relative to a structure of the vehicle (Fig. 1. Paragraph [0016]-Avila discloses a system 10 for tracking a child car seat 12 (wherein a car seat is a portable seat that attaches to a seat of a vehicle—i.e. a part of the vehicle moving with the occupant and independently movable relative to the vehicle) according to the present invention includes a seat sensing device 14 that is equipped with one or more proximity sensors 20. The seat sensing device 14 can communicate wirelessly with a vehicle onboard GPS tracker 16, which can communicate with a vehicle onboard camera 18. The seat sensing device 14 also includes a Bluetooth tracker or tag (also referred to as a key finder or smart tracker), which is a small electronic device that allows a user to monitor the location of an item to which it is attached. Further in paragraph [0017]-Avila discloses a proximity sensor is a sensor that can detect the presence of nearby objects without any physical contact. In some embodiment, the proximity sensors can, for example, be optical proximity sensors. An optical proximity sensor is a device that uses the principle of triangulation of reflected infrared or visible light to measure small distances in robotic systems. Such sensors can be utilized to check for the child's presence in the car seat 12 by measuring the distance from the sensor to the child and calibrating that distance to avoid false positive readings.), and
the displacement data is obtained from at least one of a sensor (Fig. 1. Paragraph [0016]-Avila discloses a system 10 for tracking a child car seat 12 (wherein a car seat is a portable seat that attaches to a seat of a vehicle—i.e. a part of the vehicle moving with the occupant and independently movable relative to the vehicle) according to the present invention includes a seat sensing device 14 that is equipped with one or more proximity sensors 20. The seat sensing device 14 can communicate wirelessly with a vehicle onboard GPS tracker 16, which can communicate with a vehicle onboard camera 18. The seat sensing device 14 also includes a Bluetooth tracker or tag (also referred to as a key finder or smart tracker), which is a small electronic device that allows a user to monitor the location of an item to which it is attached. Further in paragraph [0017]-Avila discloses a proximity sensor is a sensor that can detect the presence of nearby objects without any physical contact. In some embodiment, the proximity sensors can, for example, be optical proximity sensors. An optical proximity sensor is a device that uses the principle of triangulation of reflected infrared or visible light to measure small distances in robotic systems. Such sensors can be utilized to check for the child's presence in the car seat 12 by measuring the distance from the sensor to the child and calibrating that distance to avoid false positive readings.)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA of a method comprising: capturing image data relating to an occupant within a vehicle to create captured image data; identifying feature points of the occupant from the captured image data; generating raw feature points data, with the teachings of AVILA of the displacement data is correlated with a second coordinate system that is associated with a part of the vehicle moving with the occupant and independently movable relative to a structure of the vehicle, and the displacement data is obtained from at least one of a sensor.
Wherein having HYUGA’s system for interior sensing in a vehicle the displacement data is correlated with a second coordinate system that is associated with a part of the vehicle moving with the occupant and independently movable relative to a structure of the vehicle, and the displacement data is obtained from at least one of a sensor.
The motivation behind the modification would have been to obtain system for interior sensing in a vehicle that enhances occupant in seat even when ignition is off, the location of the driver head, face eye accurately, safety and occupant experience, since both HYUGA and AVILA relate to driver monitoring system in a vehicle, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while AVILA the seat sensing device is configured so that if the proximity sensor senses a child's presence in the car seat when the vehicle ignition is off, the seat sensing device will send an alert signal to the GPS tracker. In response to the alert signal, the GPS tracker is configured to cause the camera to send an alert to a user, capture an image of the car seat in the vehicle, and send the image to the user. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and AVILA et al. (US 20230179744 A1), Abstract.
HYUGA in view of AVILA fail to explicitly teach or processing of image data to track movement of the part of the vehicle moving with the occupant.
However, ABHAU explicitly teaches or processing of image data to track movement of the part of the vehicle moving with the occupant (Fig. 8 and 10. Paragraph [0026-0027]-ABHAU discloses the 3D reference model can comprise seats, tables, interior walls, fixtures, etc. The reference model preferably also takes the vehicle interior into account resulting from rotating, moving, or tilting the seats and other static interior elements of the vehicle. As a result, when a vehicle seat has been moved from a reference position or the backrest has been adjusted, this can be derived from a change in the three dimensional coordinates of a seat determined from the camera images. Further in paragraph [0056]-ABHAU discloses FIG. 8 shows an example in which two vehicle seats S1 and S2 defined in the reference model are identified by comparing the collection of three dimensional coordinates of pixels identified in the correlation process with the reference model. A displacement Dis of the vehicle seat S1 from its standard position S1sdt can be detected from the deviations of the three dimensional coordinates S1 of the first seat from the camera images. In the same manner, an adjustment of the backrests of seats S1 and S2 from their standard tilt can also be detected (wherein the seat is the part of the vehicle moving with the occupant).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA in view of AVILA of a method comprising: capturing image data relating to an occupant within a vehicle to create captured image data; identifying feature points of the occupant from the captured image data; generating raw feature points data, with the teachings of ABHAU of or processing of image data to track movement of the part of the vehicle moving with the occupant.
Wherein having HYUGA’s system for interior sensing in a vehicle or processing of image data to track movement of the part of the vehicle moving with the occupant.
The motivation behind the modification would have been to obtain system for interior sensing in a vehicle that enhances occupant in seat even when ignition is off, the location of the driver head, face eye accurately, safety and occupant experience, since both HYUGA and ABHAU relate to interior monitoring systems in a vehicle, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while ABHAU the image can be shown to the driver, for example, such that said driver can easily see that the vehicle interior is in a safe state. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and ABHAU et al. (US 20190279008 A1), Paragraph [0057].
Regarding claim 13, HYUGA in view of AVILA and further in view of ABHAU explicitly teach the method of claim 12,
HYUGA further explicitly teach wherein the displacement data is obtained from a sensor (Fig. 1-3, #11 called a camera. Paragraph [0052]-HYUGA discloses the monocular camera 11 as an imaging section can periodically (e.g. 30-60 times/sec) pick up images including the head of the driver sitting in the driver's seat, and comprises a lens system 11a consisting of one or more lenses, an imaging element 11b such as a CCD (wherein the CCD is a sensor, camera uses a Charge-Coupled Device (CCD) sensor to capture images.) or a CMOS which generates imaging data of a subject, an analog-to-digital conversion section (not shown) which converts the imaging data to digital data, an infrared irradiation unit (not shown) such as a near infrared LED which irradiates near infrared light, and associated parts. Fig. 3, illustrates a driver seat that moves away or towards the facing camera that is mounted behind the steering wheel. Paragraph [0089]-HYUGA discloses when estimating the distance Z, changes in size of the face area of the driver in a plurality of images (time-series images) picked up by the monocular camera 11 may be detected so as to decide in which direction, forward or backward, the driver is away from the focal position where the monocular camera 11 focuses, and with use of said decision result and the defocus amount d, the distance Z may be estimated. Further in paragraph [0090]-HYUGA discloses in step S5, with use of the distance Z, the distance A from the steering wheel 32 to the head of the driver 30 is estimated. For example, when the steering wheel 32 is on the line segment between the monocular camera 11 and the driver 30, the distance A is estimated by subtracting the distance B between the monocular camera 11 and the steering wheel 32 from the distance Z. Further read Fig. 6.)..
Regarding claim 14, HYUGA in view of AVILA and further in view of ABHAU explicitly teach the method of claim 12,
HYUGA further teaches wherein the displacement data is obtained from monitoring movement of a seat in the vehicle (Fig. 3, illustrates a driver seat that moves away or towards the facing camera that is mounted behind the steering wheel. Paragraph [0061]-HYUGA discloses as shown in FIG. 3, it is a situation in which a driver 30 is sitting in a driver's seat 31. A steering wheel 32 is located in front of the driver's seat 31. The position of the driver's seat 31 can be rearwardly and forwardly adjusted, and the adjustable range of the seat is set to be S. The monocular camera 11 is mounted behind the steering wheel 32 (on a steering column, or at the front of a dashboard or an instrument panel, none of them shown), that is, on a place where images 11c including a head (face) of the driver 30A can be picked up thereby.).
Regarding claim 15, HYUGA in view of AVILA and further in view of ABHAU explicitly teach the method of claim 12, HYUGA further teaches wherein an action is generated dependent on the set of occupant characteristics (Fig. 1-2 and Fig. 6, illustrates the image data captured by the monocular camera #11 is supply to the image storing part #15a and CPU #12 to process the camera data to further send data to an HMI (Human Machine Interface). And S1-7 the camera data is analyzed and controls the HMI #40 to send an alarm about the driver and seat. Paragraph [0054]-HYUGA discloses The CPU 12 is a hardware processor, which reads out a program stored in the ROM 13, and based on said program, performs various kinds of processing on image data picked up by the monocular camera 11. A plurality of CPUs 12 may be mounted for every processing such as image processing or control signal output processing. Further in paragraph [0087]-HYUGA discloses in step S1, data of one or more images picked up by the monocular camera 11 is read from the image storing part 15a, and in step S2, in the read-out one or more images 11c, the head (face) area of the driver 30A is detected. Paragraph [0092-0093]-HYUGA discloses when it is judged that the distance A is not within the range wherein the steering wheel can be appropriately operated, the operation goes to step S7. In step S7, a driving operation impossible signal is output to the HMI 40 and the automatic vehicle operation control device 50, and thereafter, the processing is ended. The HMI 40, when the driving operation impossible signal is input thereto, for example, performs a display giving an alarm about the driving attitude or seat position on the display section 41, and an announcement giving an alarm about the driving attitude or seat position by the voice output section 42.).
Regarding claim 16, HYUGA explicitly teaches a non-transitory computer-readable medium (Fig. 1-2. Paragraph [0054-56]-HYUGA discloses the CPU 12 is a hardware processor, which reads out a program stored in the ROM 13, and based on said program, performs various kinds of processing on image data picked up by the monocular camera 11. A plurality of CPUs 12 may be mounted for every processing such as image processing or control signal output processing. In the ROM 13, programs for allowing the CPU 12 to perform processing as a storage instructing section 21, a reading instructing section 22, the head detecting section 23, a defocus amount detecting section 24, a distance estimating section 25, and a driving operation possibility deciding section 26 shown in FIG. 2, and the like are stored. All or part of the programs performed by the CPU 12 may be stored in the storage section 15 or a storing medium (not shown) other than the ROM 13. In the RAM 14, data required for various kinds of processing performed by the CPU 12, programs read from the ROM 13, and the like are temporarily stored) comprising instructions including:
capturing image data relating to an occupant within a vehicle to create captured image data(Fig. 1-3. Paragraph [0052]-HYUGA discloses the monocular camera 11 as an imaging section can periodically (e.g. 30-60 times/sec) pick up images including the head of the driver sitting in the driver's seat, and comprises a lens system 11a consisting of one or more lenses, an imaging element 11b such as a CCD or a CMOS which generates imaging data of a subject, an analog-to-digital conversion section (not shown) which converts the imaging data to digital data.);
identifying feature points of the occupant (Fig. 1-3, #30 called a driver. Paragraph [0061]) from the captured image data (Fig. 1-6. Paragraph [0054]-HYUGA discloses the CPU 12 is a hardware processor, which reads out a program stored in the ROM 13, and based on said program, performs various kinds of processing on image data picked up by the monocular camera 11 (wherein the raw feature points data is the camera data.). Further in paragraph [0069]-HYUGA discloses the driver state estimation device 10 is established as a device wherein various kinds of programs stored in the ROM 13 are read into the RAM 14 and conducted by the CPU 12, so as to perform processing as the storage instructing section 21, reading instructing section 22, head detecting section 23, defocus amount detecting section 24, distance estimating section 25, and driving operation possibility deciding section 26.);
generating raw feature points data (Fig. 1-6. Paragraph [0054]-HYUGA discloses The CPU 12 is a hardware processor, which reads out a program stored in the ROM 13, and based on said program, performs various kinds of processing on image data picked up by the monocular camera 11 (wherein the raw feature points data is the camera data.). Further in paragraph [0069]-HYUGA discloses the driver state estimation device 10 is established as a device wherein various kinds of programs stored in the ROM 13 are read into the RAM 14 and conducted by the CPU 12, so as to perform processing as the storage instructing section 21, reading instructing section 22, head detecting section 23, defocus amount detecting section 24, distance estimating section 25, and driving operation possibility deciding section 26.);
compensating the raw feature points data with displacement data time- correlated with the captured image data (Fig. 3, illustrates a driver seat that moves away or towards the facing camera that is mounted behind the steering wheel. Paragraph [0089]-HYUGA discloses when estimating the distance Z, changes in size of the face area of the driver in a plurality of images (time-series images) picked up by the monocular camera 11 may be detected so as to decide in which direction, forward or backward, the driver is away from the focal position where the monocular camera 11 focuses, and with use of said decision result and the defocus amount d, the distance Z may be estimated. Please also read Paragraph [0064]);
generating processed result data (Fig. 3, Paragraph [0090]-HYUGA discloses in step S5, with use of the distance Z, the distance A from the steering wheel 32 to the head of the driver 30 is estimated. For example, when the steering wheel 32 is on the line segment between the monocular camera 11 and the driver 30, the distance A is estimated by subtracting the distance B between the monocular camera 11 and the steering wheel 32 from the distance Z. Further read Fig. 6.); and
determining, based on the processed result data, a set of occupant characteristics indicative of behavior of the occupant (Fig. 1-2 and Fig. 6, illustrates the image data captured by the monocular camera #11 is supply to the image storing part #15a and CPU #12 to process the camera data to further send data to an HMI (Human Machine Interface). And S1-7 the camera data is analyzed and controls the HMI #40 to send an alarm about the driver and seat. Paragraph [0054]-HYUGA discloses The CPU 12 is a hardware processor, which reads out a program stored in the ROM 13, and based on said program, performs various kinds of processing on image data picked up by the monocular camera 11. A plurality of CPUs 12 may be mounted for every processing such as image processing or control signal output processing. Further in paragraph [0087]-HYUGA discloses in step S1, data of one or more images picked up by the monocular camera 11 is read from the image storing part 15a, and in step S2, in the read-out one or more images 11c, the head (face) area of the driver 30A is detected. Paragraph [0092-0093]-HYUGA discloses when it is judged that the distance A is not within the range wherein the steering wheel can be appropriately operated, the operation goes to step S7. In step S7, a driving operation impossible signal is output to the HMI 40 and the automatic vehicle operation control device 50, and thereafter, the processing is ended. The HMI 40, when the driving operation impossible signal is input thereto, for example, performs a display giving an alarm about the driving attitude or seat position on the display section 41, and an announcement giving an alarm about the driving attitude or seat position by the voice output section 42.), wherein:
HYUGA fails to explicitly teach the displacement data is correlated with a second coordinate system that is associated with a part of the vehicle moving with the occupant and independently movable relative to a structure of the vehicle, and the displacement data is obtained from at least one of a sensor.
However, AVILA explicitly teaches the displacement data is correlated with a second coordinate system that is associated with a part of the vehicle moving with the occupant and independently movable relative to a structure of the vehicle (Fig. 1. Paragraph [0016]-Avila discloses a system 10 for tracking a child car seat 12 (wherein a car seat is a portable seat that attaches to a seat of a vehicle—i.e. a part of the vehicle moving with the occupant and independently movable relative to the vehicle) according to the present invention includes a seat sensing device 14 that is equipped with one or more proximity sensors 20. The seat sensing device 14 can communicate wirelessly with a vehicle onboard GPS tracker 16, which can communicate with a vehicle onboard camera 18. The seat sensing device 14 also includes a Bluetooth tracker or tag (also referred to as a key finder or smart tracker), which is a small electronic device that allows a user to monitor the location of an item to which it is attached. Further in paragraph [0017]-Avila discloses a proximity sensor is a sensor that can detect the presence of nearby objects without any physical contact. In some embodiment, the proximity sensors can, for example, be optical proximity sensors. An optical proximity sensor is a device that uses the principle of triangulation of reflected infrared or visible light to measure small distances in robotic systems. Such sensors can be utilized to check for the child's presence in the car seat 12 by measuring the distance from the sensor to the child and calibrating that distance to avoid false positive readings.); and
the displacement data is obtained from at least one of a sensor the displacement data is obtained from at least one of a sensor (Fig. 1. Paragraph [0016]-Avila discloses a system 10 for tracking a child car seat 12 (wherein a car seat is a portable seat that attaches to a seat of a vehicle—i.e. a part of the vehicle moving with the occupant and independently movable relative to the vehicle) according to the present invention includes a seat sensing device 14 that is equipped with one or more proximity sensors 20. The seat sensing device 14 can communicate wirelessly with a vehicle onboard GPS tracker 16, which can communicate with a vehicle onboard camera 18. The seat sensing device 14 also includes a Bluetooth tracker or tag (also referred to as a key finder or smart tracker), which is a small electronic device that allows a user to monitor the location of an item to which it is attached. Further in paragraph [0017]-Avila discloses a proximity sensor is a sensor that can detect the presence of nearby objects without any physical contact. In some embodiment, the proximity sensors can, for example, be optical proximity sensors. An optical proximity sensor is a device that uses the principle of triangulation of reflected infrared or visible light to measure small distances in robotic systems. Such sensors can be utilized to check for the child's presence in the car seat 12 by measuring the distance from the sensor to the child and calibrating that distance to avoid false positive readings.)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA a method comprising: capturing image data relating to an occupant within a vehicle to create captured image data; identifying feature points of the occupant from the captured image data; generating raw feature points data, with the teachings of AVILA of the displacement data is correlated with a second coordinate system that is associated with a part of the vehicle moving with the occupant and independently movable relative to a structure of the vehicle, and the displacement data is obtained from at least one of a sensor.
Wherein having HYUGA’s system for interior sensing in a vehicle the displacement data is correlated with a second coordinate system that is associated with a part of the vehicle moving with the occupant and independently movable relative to a structure of the vehicle, and the displacement data is obtained from at least one of a sensor.
The motivation behind the modification would have been to obtain system for interior sensing in a vehicle that enhances occupant in seat even when ignition is off, the location of the driver head, face eye accurately, safety and occupant experience, since both HYUGA and AVILA relate to driver monitoring system in a vehicle, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while AVILA the seat sensing device is configured so that if the proximity sensor senses a child's presence in the car seat when the vehicle ignition is off, the seat sensing device will send an alert signal to the GPS tracker. In response to the alert signal, the GPS tracker is configured to cause the camera to send an alert to a user, capture an image of the car seat in the vehicle, and send the image to the user. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and AVILA et al. (US 20230179744 A1), Abstract.
HYUGA in view of AVILA fail to explicitly teach or processing of image data to track movement of the part of the vehicle moving with the occupant.
However, ABHAU explicitly teaches or processing of image data to track movement of the part of the vehicle moving with the occupant (Fig. 8 and 10. Paragraph [0026-0027]-ABHAU discloses the 3D reference model can comprise seats, tables, interior walls, fixtures, etc. The reference model preferably also takes the vehicle interior into account resulting from rotating, moving, or tilting the seats and other static interior elements of the vehicle. As a result, when a vehicle seat has been moved from a reference position or the backrest has been adjusted, this can be derived from a change in the three dimensional coordinates of a seat determined from the camera images. Further in paragraph [0056]-ABHAU discloses FIG. 8 shows an example in which two vehicle seats S1 and S2 defined in the reference model are identified by comparing the collection of three dimensional coordinates of pixels identified in the correlation process with the reference model. A displacement Dis of the vehicle seat S1 from its standard position S1sdt can be detected from the deviations of the three dimensional coordinates S1 of the first seat from the camera images. In the same manner, an adjustment of the backrests of seats S1 and S2 from their standard tilt can also be detected (wherein the seat is the part of the vehicle moving with the occupant).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HYUGA in view of AVILA of a method comprising: capturing image data relating to an occupant within a vehicle to create captured image data; identifying feature points of the occupant from the captured image data; generating raw feature points data, with the teachings of ABHAU of or processing of image data to track movement of the part of the vehicle moving with the occupant.
Wherein having HYUGA’s system for interior sensing in a vehicle or processing of image data to track movement of the part of the vehicle moving with the occupant.
The motivation behind the modification would have been to obtain system for interior sensing in a vehicle that enhances occupant in seat even when ignition is off, the location of the driver head, face eye accurately, safety and occupant experience, since both HYUGA and ABHAU relate to interior monitoring systems in a vehicle, wherein HYUGA the driver is away from a focal position where the imaging section focuses, leading to an enhanced estimation accuracy of the distance, while ABHAU the image can be shown to the driver, for example, such that said driver can easily see that the vehicle interior is in a safe state. Please see HYUGA (US 20200065595 A1), Paragraph [0028], and ABHAU et al. (US 20190279008 A1), Paragraph [0057].
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant`s disclosure.
ADDISON et al. (US 12374128 B2) - System and methods for non-contact monitoring in vehicles are described. The systems and methods may use depth sensing cameras as part of the non-contact monitoring system. In some embodiments, depth data from at least one depth sensing device that has a field of view of at least part of the interior of the vehicle is received, wherein the depth data represents depth information as a function of position across the field of view. The depth data is then processed to obtain further information related to the occupant within the vehicle…Abstract, Fig. 1.
RAKSHIT et al. (US 10600390 B2) - A computer-implemented method includes: monitoring, by a computing device, a driver's eye gaze direction; receiving, by the computing device, a notification; determining, by a computing device, a location to display the notification based on the driver's eye gaze direction; and generating, by the computing device, display instructions to display the notification in the determined location.…Abstract, Fig. 4B.
BREED et al. (US 8948442 B2) - Vehicle includes a structure defining an interior space for containing or designed to contain one or more objects, and an arrangement mounted on, joined or coupled to the structure for determining whether an object is present in the interior space, or the presence of multiple objects. One arrangement includes at least one optical imaging device arranged to receive images of the interior space defined by the structure, and a processor coupled to the optical imaging device(s) and arranged to determine whether one or more objects are present in the interior space and when one or more objects are determined to be present, to obtain information about the object or objects. The processor is preferably arranged to distinguish between different objects and/or different arrangements of objects. A related method for obtaining information about objects in interior spaces of vehicles is also disclosed..…Abstract, Fig. 1.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ETHAN N WOLFSON whose telephone number is (571)272-1898. The examiner can normally be reached Monday - Friday 8:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ETHAN N WOLFSON/Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673
Prosecution Insights