DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 02/03/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
Claims 1 & 2, which both recite the limitation “an image acquisition unit configured to acquire…”
Claim 1, which recites the limitation “a determination unit configured to determine…”
Claim 2, which recites the limitation “a determination unit configured to determine…”
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
“an image acquisition unit” (¶0025-27, 45, 64, 78 & 82; element 21 of Fig. 2; steps S11 and S31 of Figs. 5 & 8, respectively) according to claims 1 and 2.
Structure: the image acquisition unit, also referred to as the feature detection function 21 [¶0078,82], is included as part of the main control unit 20, which is implemented by a microcomputer [¶0025-26; Fig. 2].
Algorithm: The feature detection function 21 (i.e., image acquisition unit) detects features of each part of the upper half of the body and hand area of the driver 50 by performing image processing on image data obtained from the first camera 10A [¶0027]. The feature detection function 21 detects coordinate values for feature points (A,B, and C) representing the posture of the drive for each frame [¶0045; S11 of Fig. 5]. These coordinate values correspond to the position of the wrist, arm joint, and shoulder [¶0064; S31 of Fig. 8].
“a determination unit” (¶0025-26, 31, 35, 46-53, 78 & 82; element 24d of Fig. 2; steps S12-S20 of Fig. 5) according to claim 1.
Structure: the determination unit, also referred to as the action detection function 24d [¶0078, 82], is included as part of the main control unit 20, which is implemented by a microcomputer [¶0025-26; Fig. 2].
Algorithm: The action detection function 24d (i.e., determination unit) detects whether the driver 50 is performing a "distracted driving" action such as operating a smartphone based on information on the upper half of the driver's body [¶0031]. The action detection function 24d can trigger the object detection control function 27 when it detects a specific condition indicating a high possibility of distracted driving being performed [¶0035]. The determination unit extracts a first feature point (B or C) of the head or neck of the driver, respectively, and a second feature point (A) of the arm or hand and performs processing steps S12, S13, S17 and S18 [¶0078; Fig. 5]. The object detection control function 27 acquires the coordinate values of the feature points (A and C) from the feature detection function and calculates the distance delta(x) in the x-axis between feature points A and C and compares it with a threshold [¶0051; S17 of Fig. 5]. If the distance delta(x) is, then the condition is not satisfied and processing returns to step S11. If the distance delta(x) is small, then the condition is satisfied and processing proceeds to S18 [¶0051]. The object detection control function 27 then executes step S18 to detect a smartphone in an image (S19) [¶0052-53; Fig. 5]. If the smartphone is detected, the processing proceeds to step S20, wherein “an operation on a smartphone” is detected [¶0053; Fig. 5]. The object detection control function 27 acquires coordinate values of feature points (A and B) from the feature detection function 21 and calculates a distance delta(y) between feature points A and B and compares it with a threshold [¶0046; S12 of Fig. 5]. When the distance delta(y) is large, then the condition is not satisfied and the processing returns to step S11. If the distance delta(y) is small, then the condition is satisfied and processing proceeds to step S13 [¶0047]. The object detection control function 27 then executes step S13 to detect a smartphone in the image (S14) [¶0048-49; Fig. 5]. If the smartphone is detected, the processing proceeds to step S15, wherein a call is detected [¶0049; Fig. 5].
“a determination unit” (¶0025-26, 31, 35, 64-70, 78, 82; element 24d of Fig. 2; steps S32-41 of Fig. 8) according to claim 2.
Structure: the determination unit, also referred to as the action detection function 24d [¶0078, 82], is included as part of the main control unit 20, which is implemented by a microcomputer [¶0025-26; Fig. 2].
Algorithm: The action detection function 24d (i.e., determination unit) detects whether the driver 50 is performing a "distracted driving" action such as operating a smartphone based on information on the upper half of the driver's body [¶0031]. The action detection function 24d can trigger the object detection control function 27 when it detects a specific condition indicating a high possibility of distracted driving being performed [¶0035]. The determination unit detects at least three feature points (A, B and C), including a point of a joint position of a body of the driver in the input image and calculates an angle θ at the joint position based on the coordinates of these feature points, and performs processing to determine whether the calculated angle satisfies a predetermined condition in steps S32, S33, S34, S35 and S39 [¶0082-83; Fig. 8]. The object detection control function 27 acquires coordinate values of feature points from the feature detection function 21 and calculates the angle theta at a joint position of the arm (S32) and compares the angle with predetermined thresholds θ1 (S33) and θ2 (S34) [¶0064; Fig. 8]. If the angle θ at the joint position is large, then the condition is not satisfied and processing returns to S31 [¶0065; Fig. 8]. When the angle θ at the joint position is equal to or smaller than a smaller threshold θ1, then the condition is satisfied and processing proceeds to S35 for smartphone detection. If a smartphone is detected in S36, then the determination unit detects a call (S37) [¶0066-67; Fig. 8]. If the angle θ at the joint position is larger than the threshold θ1 and equal to or smaller than the threshold θ2, then the condition of S34 is satisfied and processing proceeds to S38 for smartphone detection [¶0068; Fig. 8]. If a smartphone is detected in S39, then the determination unit detects operation on the smartphone (S40) [¶0069-70; Fig. 8]
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1 & 2 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 describes two distinct scenarios for whether a user is using a mobile terminal in a vicinity of a hand of a driver:
“the driver is operating the mobile terminal, when a difference of x-coordinate… and…”
“the driver is making a call, when a difference of y-coordinate…”
The examiner understand that these two conditional limitations are intended to be mutually exclusive in light of their descriptions in the specification. Therefore, the use of “and” renders the claim indefinite as only one of the following conditions can be satisfied at a given time.
Similarly, Claim 2 also describes two distinct scenarios for whether a user is using a mobile terminal in a vicinity of a hand of a driver:
“the driver is making a call, when the angle is less… and…”
“the driver is operating the mobile terminal, when the angle is greater…”
Again, the examiner understand that these two conditional limitations are intended to be mutually exclusive in light of their descriptions in the specification. Therefore, the use of “and” renders the claim indefinite as only one of the following conditions can be satisfied at a given time.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kinoshita et al (JP 2017111508 A), hereinafter referred to as “Kinoshita” in view of Leung; Spencer Yu Cheong (US 20160026241 A1), hereinafter referred to as “Leung”.
Regarding claim 1, Kinoshita teach A driver monitoring device (Kinoshita: information processing device 1 [¶012; Fig. 1]) comprising:
an image acquisition unit† (†the examiner notes that this limitation is being mapped to its associated claim interpretation under 35 U.S.C. § 112(f) – Kinoshita: driver part detection unit 12 as part of the information processing device 1, which is implemented by a processor with a memory [¶018; Fig. 1], wherein the driver part detection unit 12 generates a skeleton model of a driver in the form of a point cloud comprising skeleton points for the wrist (P4/5), arm joint (P7/8), and shoulder (P2/3) (i.e., feature points A, B & C) from distance measuring sensor 2 or a camera [¶021- 22, 24-26, 51; Fig. 2]) configured to acquire an image of a driver; and
a determination unit† (†the examiner notes that this limitation is being mapped to its associated claim interpretation under 35 U.S.C. § 112(f), wherein the associated algorithm for the determination unit will be mapped to each subsequent portion of the claim language – Kinoshita: driver posture determination unit 13 as part of the information processing device 1, which is implemented by a processor with a memory [¶018; Fig. 1]) configured to determine, based on the acquired image, whether the driver is using a mobile terminal in a vicinity of a hand of the driver (Kinoshita: the driving posture determination unit 13, gripping object detection unit 14, and smart phone / portable operation determination unit 15 detects the posture of a driver based on the position of skeleton points to determine if they are operating a phone [¶033-34, 42]), wherein
the determination unit extracts, from the image, a first feature point included in a neck or a head of the driver and a second feature point included in an arm or a hand of the driver (Kinoshita: the driver posture detection unit 13 extracts positions of point P1 (corresponding to a head) and point P4/5, corresponding to a left or right hand, respectively [¶42-44; Fig. 4A & B]), and determines that: … and the driver is making a call, when a difference of y coordinate, corresponding to a vertical direction, between a coordinate of the first feature point and a coordinate of the second feature point is smaller than a second reference level, and when the mobile terminal is included in the image (Kinoshita: when the driving posture determination unit 13 determines the distance between P1 and P4/5 is less than a predetermined distance L2 [¶044; Figs. 4A & B] – the examiner notes that while distance between points P1 and P4/5 is not explicitly recited to be in the y-direction, one of ordinary skill in the art would recognize that only the y-component of this distance can be compared to the reference level L2, and the gripping object detection unit 14 detects a phone, the smartphone / portable operation determination unit 15 determines a user is operating a phone [¶044-47; Figs. 4A & B] – the examiner also notes that while the action of making a call is not explicitly stated, the operation of a phone as illustrated in Figs. 4A & 4B, particularly when the phone is in close range of the head (as is the case when L2 is 10 cm as described in ¶044) is analogous to the posture a driver would have when making a phone call).
Kinoshita, however, fails to recite a condition of detecting a user operating the mobile terminal using a different reference level based on the horizontal distance between two features. Leung, on the other hand, is analogous art pertinent to the field of endeavor and disclose a device for monitoring the habits a phone user. Leung teach the driver is operating the mobile terminal, when a difference of x coordinate, corresponding to a horizontal direction, between a coordinate of the first feature point and a coordinate of the second feature point is smaller than a first reference level, and when the mobile terminal is included in the image; (Leung: the monitoring system determines a user is using a smartphone within an excessively close distance Y when Y < X [¶0087-88; Fig. 13] – the examiner notes that a user holding a smartphone within a certain distance inherently implies some form of smartphone operation).
Kinoshita further describe that their system is directed to addressing the misuses of devices such as smartphones [¶0003-4]. One of ordinary skill would recognize the benefit of incorporating distance threshold taught by Leung with the base driver monitoring device of Kinoshita to detect misuse of a phone (i.e., operation) while driving, arriving at the invention of the instant application.
Claim(s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kinoshita et al (JP 2017111508 A), hereinafter referred to as “Kinoshita” in view of Xiao; Zhong-Wen (CN 111461020 A), hereinafter referred to as “Xiao”.
With respect to claim 2, Kinoshita teach A driver monitoring device (Kinoshita: information processing device 1 [¶012; Fig. 1]) comprising:
an image acquisition unit† (†the examiner notes that this limitation is being mapped to its associated claim interpretation under 35 U.S.C. § 112(f) – Kinoshita: driver part detection unit 12 as part of the information processing device 1, which is implemented by a processor with a memory [¶018; Fig. 1], wherein the driver part detection unit 12 generates a skeleton model of a driver in the form of a point cloud comprising skeleton points for the wrist (P4/5), arm joint (P7/8), and shoulder (P2/3) (i.e., feature points A, B & C) from distance measuring sensor 2 or a camera [¶021- 22, 24-26, 51; Fig. 2]) configured to acquire an image of a driver, and
a determination unit† (†the examiner notes that this limitation is being mapped to its associated claim interpretation under 35 U.S.C. § 112(f), wherein the associated algorithm for the determination unit will be mapped to each subsequent portion of the claim language – Kinoshita: driver posture determination unit 13 as part of the information processing device 1, which is implemented by a processor with a memory [¶018; Fig. 1]) configured to determine, based on the acquired image, whether the driver is using a mobile terminal in a vicinity of a hand of the driver (Kinoshita: the driving posture determination unit 13, gripping object detection unit 14, and smart phone / portable operation determination unit 15 detects the posture of a driver based on the position of skeleton points to determine if they are operating a phone [¶033-34, 42]), wherein
the determination unit detects at least three feature points including a point of a joint position of a body of the driver on the input image, (Kinoshita: the driver posture detection unit 13 extracts positions of point P1 (corresponding to a head), points P2/3 (corresponding to a left or right should) points P4/5 (corresponding to a left or right hand), and points P7/8 (corresponding to a left and right elbow), respectively [¶42-44; Fig. 4A & B]).
Kinoshita fails to describe calculating angles of the points to determine a driver is making a phone call. Xiao, per contra, describe a device and method for recognizing unsafe phone use behavior while operating a motor vehicle. Xiao teach calculates an angle at the joint position based on coordinates of these feature points (Xiao: a bending angle of a left elbow is calculated [¶015]), and determines that: the driver is making a call, when the angle is less than or equal to the first angle, and when the mobile terminal is included in the image (Xiao: when the bending angle of the elbow is less than the second angle, then the target is in a calling state [¶015]); and the driver is operating the mobile terminal, when the angle is greater than the first angle and less than or equal to a second angle greater than the first angle, and when the mobile terminal is included in the image (Xiao: when the bending angle of the left elbow is less than a first angle and not less than a second angle, the target (i.e., a driver) is in the state of playing the mobile phone by the left hand [¶015]).
Furthermore, Xiao explains the importance of identifying unsafe mobile phone usage while operating a vehicle, referencing the increased chance of traffic accidents as a result of inappropriate phone usage [¶05]. One of ordinary in the art would recognize the benefit of implementing Xiao’s angle determination system for distinguishing when a user is making a phone call vs generally operating their phone to better inform the driver monitoring system of Kinoshita to arrive at the invention of the present application.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Wang; Keyao (US 2021/0347366 A1) describe an abnormal driving behavior determination leveraging arm key-points to recognize phone usage while operating a vehicle.
Porta; Pier Paolo (US 2021/0012126 A1) teach an apparatus for monitoring a driver in an interior of a vehicle for illegal phone usage
Li et al (“Human phone usage recognition based on OpenPose”, July 18, 2022, CVIDL & ICCEA) describe using the popularize OpenPose pose estimation method for detecting postures associated with phone usage
Rangesh et al (“When Vehicles See Pedestrians with Phones: A Multi-Cue Framework for Recognizing Phone-based Activities of Pedestrians”, 2018, arXiv) describe using a variety of pose estimation paradigms for identifying pedestrians when they are texting or on a phone call.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael M. Sofroniou whose telephone number is (571)272-0287. The examiner can normally be reached M-F: 8:30 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John M. Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL M SOFRONIOU/Examiner, Art Unit 2661
/JOHN VILLECCO/Supervisory Patent Examiner, Art Unit 2661
Prosecution Insights