DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant's claim for foreign priority based on an application filed in South Korea on 12/13/2023. It is noted, however, that applicant has not filed a certified copy of the 10-2023-0180738 application as required by 37 CFR 1.55.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 10 and 20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term “at least some of…” in claims 10 and 20 is a relative term which renders the claim indefinite. The term “at least some of…” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-2 and 11-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Arora et al. (US 2021/0397859 A1) in view of Liang et al. (US 2020/0324784 A1).
Consider claim 11, Arora teaches an electronic device for monitoring a driver's state (computing device that is configured to determine driver engagement using 3D eye gaze vectors. [0041] and Fig. 5), comprising: an interior camera configured to capture a vehicle interior (Cameras 102 may capture images of an occupant of vehicle 100 as the occupant is driving vehicle 100 (e.g., the driver of vehicle 100). [0012] – [0015]); and a processor configured to: detect feature points of a driver's face in a vehicle interior image captured by the interior camera (n calculating the facial plane of the occupant, vehicle computing system 104 may identify a plurality of facial landmarks in one or more images captured by one or more of cameras 102. Facial landmarks may include edges of a mouth, eyes, nose, ears, eyebrows, jaw, or other facial features. [0015] – [0019]), and determine the driver's state using distances between coordinates of at least two of the (Using the identified facial landmarks, vehicle computing system 104 may determine if the occupant's face included in image exhibits any pitch, roll, or yaw based on a geometric consistency between the various facial landmarks. For example, if the distances between the occupant's two eyes relative to the overall distance between the occupant's mouth and eyes is smaller than when the occupant is looking straight ahead, vehicle computing system 104 determines that the occupant is looking to the left or right. [0015] – [0019]).
However, Arora does not explicitly teach transform coordinates of the detected feature points into coordinates in a frontal coordinate system, wherein the frontal coordinate system is a coordinate system of an image captured by the interior camera from directly in front of the driver's face.
Liang teaches transform coordinates of the detected feature points into coordinates in a frontal coordinate system (converting the coordinates of the face reference point from the camera coordinate system to an on-board unit coordinate system. [0118]. FIG. 8 is a schematic diagram of rotating coordinate points (x.sub.1, z.sub.1) in a camera coordinate system to coordinate points (x.sub.0, z.sub.0) in an on-board unit coordinate system. As shown in FIG. 8, assuming that a it is detected in the camera coordinate system that the coordinate point of the driver's head is (y.sub.1, z.sub.1), the coordinate point is rotated by an angle α, i.e., the installation angle of a camera, to obtain a coordinate point (x.sub.0, z.sub.0) in the on-board unit coordinate system. [0121] – [0128]. Fig. 5-Fig. 8.), wherein the frontal coordinate system is a coordinate system of an image captured by the interior camera from directly in front of the driver's face (converting the coordinates of the face reference point from the camera coordinate system to an on-board unit coordinate system. [0118]. FIG. 8 is a schematic diagram of rotating coordinate points (x.sub.1, z.sub.1) in a camera coordinate system to coordinate points (x.sub.0, z.sub.0) in an on-board unit coordinate system. As shown in FIG. 8, assuming that a it is detected in the camera coordinate system that the coordinate point of the driver's head is (y.sub.1, z.sub.1), the coordinate point is rotated by an angle α, i.e., the installation angle of a camera, to obtain a coordinate point (x.sub.0, z.sub.0) in the on-board unit coordinate system. [0121] – [0128]. Fig. 5-Fig. 8.
Fig. 8 shows that the on-board unit coordinate system is a frontal coordinate system because with the x-axis being parallel to the ground and the z-axis being perpendicular to the x-axis, the image captured with this coordinate is directly in front of the driver’s face).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the known technique of transforming coordinates of the detected feature points into a frontal coordinate system because such incorporation would help improve driving comfort and user experience. [0053].
Consider claim 12, the processor is configured to: acquire a mounting angle of the interior camera (converting the coordinates of the face reference point from the camera coordinate system to an on-board unit coordinate system. [0118]. FIG. 8 is a schematic diagram of rotating coordinate points (x.sub.1, z.sub.1) in a camera coordinate system to coordinate points (x.sub.0, z.sub.0) in an on-board unit coordinate system. As shown in FIG. 8, assuming that a it is detected in the camera coordinate system that the coordinate point of the driver's head is (y.sub.1, z.sub.1), the coordinate point is rotated by an angle α, i.e., the installation angle of a camera, to obtain a coordinate point (x.sub.0, z.sub.0) in the on-board unit coordinate system. [0121] – [0128]. Fig. 5-Fig. 8.); and transform the coordinates of the feature points into coordinates in the frontal coordinate system using the mounting angle of the interior camera (converting the coordinates of the face reference point from the camera coordinate system to an on-board unit coordinate system. [0118]. FIG. 8 is a schematic diagram of rotating coordinate points (x.sub.1, z.sub.1) in a camera coordinate system to coordinate points (x.sub.0, z.sub.0) in an on-board unit coordinate system. As shown in FIG. 8, assuming that a it is detected in the camera coordinate system that the coordinate point of the driver's head is (y.sub.1, z.sub.1), the coordinate point is rotated by an angle α, i.e., the installation angle of a camera, to obtain a coordinate point (x.sub.0, z.sub.0) in the on-board unit coordinate system. [0121] – [0128]. Fig. 5-Fig. 8.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the known technique of transforming coordinates of the detected feature points into a frontal coordinate system because such incorporation would help improve driving comfort and user experience. [0053].
Consider claim 1, claim 1 recites the method implemented by the device recited in claim 11. Thus, it is rejected for the same reasons.
Consider claim 2, claim 2 recites the method implemented by the device recited in claim 12. Thus, it is rejected for the same reasons.
Claim(s) 5-6 and 15-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Arora et al. (US 2021/0397859 A1) in view of Liang et al. (US 2020/0324784 A1) and Li et al. (US 2025/0005961 A1).
Consider claim 15, the combination of Arora and Liang teaches all the limitations in claim 11 and determine the driver's state based on a ratio of the distance between the first and third feature points to the distance between the second and third feature points (Using the identified facial landmarks, vehicle computing system 104 may determine if the occupant's face included in image exhibits any pitch, roll, or yaw based on a geometric consistency between the various facial landmarks. For example, if the distances between the occupant's two eyes relative to the overall distance between the occupant's mouth and eyes is smaller than when the occupant is looking straight ahead, vehicle computing system 104 determines that the occupant is looking to the left or right. [0015] – [0019]).
However, the combination does not explicitly teach the processor is configured to: detect the feature points of the driver's face by detecting a center point of a left eye as a first feature point, a center point of a right eye as a second feature point, and a center point of a nose as a third feature point.
Li teaches the processor is configured to: detect the feature points of the driver's face by detecting a center point of a left eye as a first feature point, a center point of a right eye as a second feature point, and a center point of a nose as a third feature point (the facial feature area may include an area corresponding to preset feature parts (e.g., the left eye center, the right eye center, the nose tip, the left mouth corner, and the right mouth corner) of the face. The area other than the area corresponding to the preset feature parts of the face may be the face contour area. The keypoints in the face contour area may correspond to the contour keypoints. The area 220 in the input face image 200 of FIG. 2 may include feature keypoints corresponding to preset feature parts of the face. [0047]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings from Li to the combination of Arora and Liang because such incorporation would help determine whether the face image is an invalid face image or not. [0048].
Consider claim 16, Arora teaches the processor is configured to: determine the driver's state as not gazing forward when the ratio of distances is greater than or equal to a first reference value, or less than a second reference value (Using the identified facial landmarks, vehicle computing system 104 may determine if the occupant's face included in image exhibits any pitch, roll, or yaw based on a geometric consistency between the various facial landmarks. For example, if the distances between the occupant's two eyes relative to the overall distance between the occupant's mouth and eyes is smaller than when the occupant is looking straight ahead, vehicle computing system 104 determines that the occupant is looking to the left or right. [0015] – [0019]); and determine the driver's state as gazing forward when the ratio of distances is greater than or equal to the second reference value and less than the first reference value (Using the identified facial landmarks, vehicle computing system 104 may determine if the occupant's face included in image exhibits any pitch, roll, or yaw based on a geometric consistency between the various facial landmarks. For example, if the distances between the occupant's two eyes relative to the overall distance between the occupant's mouth and eyes is smaller than when the occupant is looking straight ahead, vehicle computing system 104 determines that the occupant is looking to the left or right. [0015] – [0019]).
Consider claim 5, claim 5 recites the method implemented by the device recited in claim 15. Thus, it is rejected for the same reasons.
Consider claim 6, claim 6 recites the method implemented by the device recited in claim 16. Thus, it is rejected for the same reasons.
Claim(s) 7-10 and 17-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Arora et al. (US 2021/0397859 A1) in view of Liang et al. (US 2020/0324784 A1) and Dewi, C., Chen, R. C., Chang, C. W., Wu, S. H., Jiang, X., & Yu, H. (2022). Eye aspect ratio for real-time drowsiness detection to improve driver safety. Electronics, 11(19), 3183 (hereinafter “Dewi”).
Consider claim 17, the combination of Arora and Liang teaches all the limitations in claim 11 but does not explicitly teach at least multiple feature points among the detected feature points of the driver's face are detected from one feature portion among feature portions of the driver's face; and the feature portions of the driver's face include at least one of eyes, nose, mouth, eyebrows, and facial contour lines.
Dewi teaches at least multiple feature points among the detected feature points of the driver's face are detected from one feature portion among feature portions of the driver's face (we made use of the 68 facial landmarks from Dlib [42]. Estimating the 68 (x,y)-coordinates corresponding to the facial structure on the face was carried out with the help of a pre-trained facial landmark detector found in the Dlib library. Figure 1 displays that the jaw points range from 0 to 16, the right brow points range from 17 to 21, and the left brow points range from 22 to 26. The nose points range from 27 to 35, the right eye points range from 36 to 41, and the left eye points range from 42 to 47. The mouth points range from 48 to 60, and the lip points range from 61 to 67.p. 3 and Fig. 1); and the feature portions of the driver's face include at least one of eyes, nose, mouth, eyebrows, and facial contour lines (we made use of the 68 facial landmarks from Dlib [42]. Estimating the 68 (x,y)-coordinates corresponding to the facial structure on the face was carried out with the help of a pre-trained facial landmark detector found in the Dlib library. Figure 1 displays that the jaw points range from 0 to 16, the right brow points range from 17 to 21, and the left brow points range from 22 to 26. The nose points range from 27 to 35, the right eye points range from 36 to 41, and the left eye points range from 42 to 47. The mouth points range from 48 to 60, and the lip points range from 61 to 67.p. 3 and Fig. 1).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings from Dewi to the combination of Arora and Liang because such incorporation would help detect driver’s drowsiness. Abstract.
Consider claim 18, Dewi teaches the processor is configured to: determine the driver's state based on a ratio (C), where (C) is the ratio of the distance between end feature points of one eye among multiple eye feature points detected from at least one eye among the driver's left eye and right eye, to a sum of distances between at least one upper eye feature point and its corresponding lower eye feature point of the one eye (section 2.2 on p. 4-5 and Figure 2).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings from Dewi to the combination of Arora and Liang because such incorporation would help detect driver’s drowsiness. Abstract.
Consider claim 19, Dewi teaches the processor is configured to: determine the driver's state as drowsy when the ratio (C) remains less than or equal to a third reference value for at least a predetermined time period (section 2.2 on p. 4-5 and Figure 2).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings from Dewi to the combination of Arora and Liang because such incorporation would help detect driver’s drowsiness. Abstract.
Consider claim 20, the combination of Arora, Liang, and Dewi teaches the processor is configured to: determine at least one of the driver's drowsiness (section 2 and 2.1 on p. 3 of Dewi) and the driver's forward gaze state by a machine learning model trained with at least some of the detected feature points of the driver's face ([0020], [0058], [0064] – [0068], [0077], [0154] of Arora).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings from Dewi to the combination of Arora and Liang because such incorporation would help detect driver’s drowsiness. Abstract.
Consider claim 7, claim 7 recites the method implemented by the device recited in claim 17. Thus, it is rejected for the same reasons.
Consider claim 8, claim 8 recites the method implemented by the device recited in claim 18. Thus, it is rejected for the same reasons.
Consider claim 9, claim 9 recites the method implemented by the device recited in claim 19. Thus, it is rejected for the same reasons.
Consider claim 10, claim 10 recites the method implemented by the device recited in claim 20. Thus, it is rejected for the same reasons.
Allowable Subject Matter
Claims 3-4 and 13-14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAT CHI CHIO whose telephone number is (571)272-9563. The examiner can normally be reached Monday-Thursday 10am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JAMIE J ATALA can be reached at 571-272-7384. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TAT C CHIO/Primary Examiner, Art Unit 2486
Prosecution Insights