DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The disclosure is objected to because of the following informalities:
Paragraph 65, Line 1: The verb “simulate” should be plural to agree with the singular subject “system”.
Paragraph 88, Line 6: The article “an” should be replaced with –a--.
Appropriate correction is required.
Claim Objections
Claims 2, 12, and 14 are objected to because of the following informalities: The phrase “digital twin” does not have a common meaning in the art. Paragraph 67 of the applicant’s disclosure states that the digital twin system constitutes the “static object and the dynamic object together”. This is how the claims will be interpreted. Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 11 and 12 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claims do not fall within at least one of the four categories of patent eligible subject matter because claim 11 pertains to a computer-readable storage medium without a qualifier regarding a non-transitory medium. Furthermore, the applicant’s disclosure does not specifically exclude transitory media. Therefore, the claim pertains to all media, both transitory and non-transitory, which is considered non-statutory subject matter.
Claim 12 is rejected as being dependent on a rejected base claim.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-3 and 5-7 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Martin et al [IEEE Intelligent Vehicles Symposium, 6/4/23: “Viewpoint Invariant 3D Driver Body Pose-Based Activity Recognition”].
For claim 1, the method for predicting a driving risk taught by Martin comprises the following claimed steps, as noted, 1) determining a dynamic object for a vehicle while driving (Page 1, Right Column, second paragraph: “we use the position of dynamic 3D objects such as phones, books or bottles”; Page 3, III. METHOD: 3D body pose: “Depending on the sensor system the 3D body pose estimation method changes”), 2) determining a first position of the dynamic object in the vehicle (Page 3, 3D Object Positions; also, see Figure 2 “Sensor dependent 3D Body Pose Estimation” and picture captioned “3D interior, 3D objects”), 3) determining a first static object in the vehicle corresponding to the dynamic object (Page 1, Right Column, second paragraph: “…and the location of interior elements like the controls, seats, and storage areas as input to the driver activity recognition system”; also, see Page 3: 3D Interior Positions) depending on the structure of the vehicle, and 4) predicting the driving risk corresponding to the dynamic object based on an interactive action between the dynamic object and the first static object (Fig. 2, right-most picture: “Interaction Graph” and “Activity Recognition”; Page 3, Left Column, first paragraph: “The modular recognition pipeline combines 3D driver body pose detection, 3D interior objects and 3D dynamic object detection for driver activity recognition; Page 1, Left Column, first paragraph: “A key requirement for driver monitoring is detecting a driver’s state”, second paragraph: “Recognizing the driver’s activities is not part of the legal requirements yet but they are part of many distracting behavior patterns that in turn cause accidents”).
For claim 2, Figure 3 of Martin depicts the sensors used in the method, in this case either Drive&Act cameras or a Kinnect system. Figure 1 depicts a digital twin system in that a three-dimensional representation is created using digital representations of the driver (green), objects (orange), and interior elements (gray). Figure 2 depicts the positions of the objects from the reconstruction.
For claim 3, the data of Martin is input into a neural network model (Page 3, Right Column, last paragraph: “The resulting graph is then processed by a neural network with a graph convolution-based architecture”), which is then used in a three-dimensional reconstruction model (Page 4, Left Column, second paragraph: “To compare the modular system with an end-to-end video-based baseline approach we select the I3D model. The I3D model is an approach based on 3D convolutions”).
For claims 5-7, the method taught by Martin obtains a positional relationship between the dynamic objects and static objects (See Figure 1 and Figure 2: “3D interior, 3D objects”), determines an interactive action between the dynamic object and a static object (Fig. 2: “Interaction Graph”, “Activity recognition”), and predicts a driving risk corresponding to the dynamic object (Fig. 2, right-most picture: “Interaction Graph” and “Activity Recognition”; Page 3, Left Column, first paragraph: “The modular recognition pipeline combines 3D driver body pose detection, 3D interior objects and 3D dynamic object detection for driver activity recognition; Page 1, Left Column, first paragraph: “A key requirement for driver monitoring is detecting a driver’s state”, second paragraph: “Recognizing the driver’s activities is not part of the legal requirements yet but they are part of many distracting behavior patterns that in turn cause accidents”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Martin et al in view of Martin et al [IEEE Intelligent Transportation Systems Conference, 9/19/21: An Evaluation of Different Methods for 3D-Driver-Body-Pose Estimation”].
For claim 4, the Martin ’23 reference does mention triangulation (Page 1, Right Column, second paragraph: “using a multi-view camara system with a triangulation-based approach”) as well as two-dimensional key points from an image (Page 3, Left Column, 3D body pose: “If the multi-view system is used we use standard triangulation of each keypoint for 3D reconstruction”).
Furthermore, the Martin ’21 reference, which shares authors with the Martin ’23 reference, details an evaluation of different methods for driver body pose using both two-dimensional and three-dimensional input. For example, Page 3 (A. Multi-View Triangulation) evaluates data from a plurality of cameras that captures body joints and then triangulates each joint separately. Also, Page 5 introduces metrics (B. Metrics) such as Object Keypoint Similarity and Percentage of Correct Keypoints in order to produce a correct two-dimensional estimation result.
The Martin ’21 reference shows that more than one method can be used to deduce and estimate the locations of a person or object in a vehicle interior. Different approaches may be used depending on the number and type of sensors used (Martin ’23: Page 3, Left Column, 3D body pose: Either an OpenPose approach may be used or a DepthFix approach). This allows the system to be used in a wider variety of vehicles and sensor systems that may occur in vehicles. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to extract two-dimensional keypoints from sensor data in order to perform the method of Martin ’21 for the purpose of increasing the applicability of the system among many different types of vehicles and sensor systems.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Martin et al in view of Katz et al [US 2020/0207358].
For claim 8, the method taught by Martin teaches the claimed subject matter as discussed above. However, there is no mention of making an alert for a prohibited action.
Alerts and alarms pertaining to driver risk is not new in the prior art. The system of Katz discloses a system that monitors driver attentiveness. Figure 1 depicts the system having a sensor such as a camera (No. 130) that focuses on the driver’s behavior. The system also includes an advanced driver-assistance system (ADAS No. 150) that aids the driver while driving. This aid can include taking control of the vehicle (Paragraph 56) as well as lighting automation, adaptive cruise control, and collision avoidance.
The Katz reference presents plain evidence that alerts can and have been used to alert a driver if their behavior puts them or the vehicle at risk. The Katz reference basically uses the information gathered in Martin and gives it a real-world application and use. And this application would prevent a number of accidents and injuries. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to make an alert in the system of Martin as the reference already determines possible distracted behavior of the driver in order to prevent any adverse consequences.
Claims 11-15 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Martin et al.
For claim 11, the method for predicting a driving risk taught by Martin comprises the following claimed steps, as noted, 1) determining a dynamic object for a vehicle while driving (Page 1, Right Column, second paragraph: “we use the position of dynamic 3D objects such as phones, books or bottles”; Page 3, III. METHOD: 3D body pose: “Depending on the sensor system the 3D body pose estimation method changes”), 2) determining a first position of the dynamic object in the vehicle (Page 3, 3D Object Positions; also, see Figure 2 “Sensor dependent 3D Body Pose Estimation” and picture captioned “3D interior, 3D objects”), 3) determining a first static object in the vehicle corresponding to the dynamic object (Page 1, Right Column, second paragraph: “…and the location of interior elements like the controls, seats, and storage areas as input to the driver activity recognition system”; also, see Page 3: 3D Interior Positions) depending on the structure of the vehicle, and 4) predicting the driving risk corresponding to the dynamic object based on an interactive action between the dynamic object and the first static object (Fig. 2, right-most picture: “Interaction Graph” and “Activity Recognition”; Page 3, Left Column, first paragraph: “The modular recognition pipeline combines 3D driver body pose detection, 3D interior objects and 3D dynamic object detection for driver activity recognition; Page 1, Left Column, first paragraph: “A key requirement for driver monitoring is detecting a driver’s state”, second paragraph: “Recognizing the driver’s activities is not part of the legal requirements yet but they are part of many distracting behavior patterns that in turn cause accidents”).
The Martin reference does not mention a computer-readable storage medium storing a computer program; however, the reference does mention using a modular processing pipeline (Page 5, Right Column, fourth paragraph, V. Conclusion) for the driver activity recognition that works with the camera system. As modern processing equipment necessarily requires various software and computer programming in order to function, this is considered an obvious property as electronic devices have been using computer-readable storage media for decades.
For claim 12, Figure 3 of Martin depicts the sensors used in the method, in this case either Drive&Act cameras or a Kinnect system. Figure 1 depicts a digital twin system in that a three-dimensional representation is created using digital representations of the driver (green), objects (orange), and interior elements (gray). Figure 2 depicts the positions of the objects from the reconstruction.
For claim 13, the method for predicting a driving risk taught by Martin comprises the following claimed steps, as noted, 1) determining a dynamic object for a vehicle while driving (Page 1, Right Column, second paragraph: “we use the position of dynamic 3D objects such as phones, books or bottles”; Page 3, III. METHOD: 3D body pose: “Depending on the sensor system the 3D body pose estimation method changes”), 2) determining a first position of the dynamic object in the vehicle (Page 3, 3D Object Positions; also, see Figure 2 “Sensor dependent 3D Body Pose Estimation” and picture captioned “3D interior, 3D objects”), 3) determining a first static object in the vehicle corresponding to the dynamic object (Page 1, Right Column, second paragraph: “…and the location of interior elements like the controls, seats, and storage areas as input to the driver activity recognition system”; also, see Page 3: 3D Interior Positions) depending on the structure of the vehicle, and 4) predicting the driving risk corresponding to the dynamic object based on an interactive action between the dynamic object and the first static object (Fig. 2, right-most picture: “Interaction Graph” and “Activity Recognition”; Page 3, Left Column, first paragraph: “The modular recognition pipeline combines 3D driver body pose detection, 3D interior objects and 3D dynamic object detection for driver activity recognition; Page 1, Left Column, first paragraph: “A key requirement for driver monitoring is detecting a driver’s state”, second paragraph: “Recognizing the driver’s activities is not part of the legal requirements yet but they are part of many distracting behavior patterns that in turn cause accidents”).
The Martin reference teaches a modular processing pipeline (Page 5, Right Column, fourth paragraph, V. Conclusion) for the driver activity recognition that works with the camera system. The reference also mentions data collection and annotation (Page 1, Right Column, first paragraph). Although this is not the specific words “processor” and “memory”, these terms are very common and well-known in electronic and computing arts and are considered obvious elements used to perform the tasks included in the method.
For claim 14, Figure 3 of Martin depicts the sensors used in the method, in this case either Drive&Act cameras or a Kinnect system. Figure 1 depicts a digital twin system in that a three-dimensional representation is created using digital representations of the driver (green), objects (orange), and interior elements (gray). Figure 2 depicts the positions of the objects from the reconstruction.
For claim 15, the data of Martin is input into a neural network model (Page 3, Right Column, last paragraph: “The resulting graph is then processed by a neural network with a graph convolution-based architecture”), which is then used in a three-dimensional reconstruction model (Page 4, Left Column, second paragraph: “To compare the modular system with an end-to-end video-based baseline approach we select the I3D model. The I3D model is an approach based on 3D convolutions”).
For claim 17, the method taught by Martin obtains a positional relationship between the dynamic objects and static objects (See Figure 1 and Figure 2: “3D interior, 3D objects”), determines an interactive action between the dynamic object and a static object (Fig. 2: “Interaction Graph”, “Activity recognition”), and predicts a driving risk corresponding to the dynamic object (Fig. 2, right-most picture: “Interaction Graph” and “Activity Recognition”; Page 3, Left Column, first paragraph: “The modular recognition pipeline combines 3D driver body pose detection, 3D interior objects and 3D dynamic object detection for driver activity recognition; Page 1, Left Column, first paragraph: “A key requirement for driver monitoring is detecting a driver’s state”, second paragraph: “Recognizing the driver’s activities is not part of the legal requirements yet but they are part of many distracting behavior patterns that in turn cause accidents”).
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Martin et al in view of Martin et al.
For claim 16, the Martin ’23 reference does mention triangulation (Page 1, Right Column, second paragraph: “using a multi-view camara system with a triangulation-based approach”) as well as two-dimensional key points from an image (Page 3, Left Column, 3D body pose: “If the multi-view system is used we use standard triangulation of each keypoint for 3D reconstruction”).
The claim is interpreted and rejected for the reasons and rationale as is mentioned in the rejection of claim 4 above.
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Martin et al in view of Katz et al.
For claim 18, the method taught by Martin teaches the claimed subject matter as discussed above. However, there is no mention of making an alert for a prohibited action.
The claim is interpreted and rejected for the same reasons and rationale as is mentioned in the rejection of claim 8 above.
Claims 9, 10, 19, and 20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
As seen above, merely determining dynamic and static objects and their positions in order to predict driving risk has been taught in the prior art. Also, using driving risk to perform an alert to the driver is also very common and well-known. However, the objected claims above mention a very specific application in response to determining the interactive action between the dynamic object and static object occurs, in this case storing first relevant information for analyzing whether the interactive action is a misoperation and also the relevant information comprises external environment information about the vehicle, motion information of the vehicle, the structure of the vehicle, and the three-dimensional reconstruction result corresponding to the dynamic object. This combination is considered unobvious subject matter.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Wang et al [U.S. 7,839,292] predicts driving danger and issues alarms.
Misu et al [U.S. 11,745,744] determines object-wise situational awareness in a vehicle.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHN A. TWEEL JR whose telephone number is (571)272-2969. The examiner can normally be reached M-F 8-4.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Davetta W Goins can be reached at 571-272-2957. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JAT
7/22/2026
/JOHN A TWEEL JR/Primary Examiner, Art Unit 2689
Prosecution Insights