DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendments
The preliminary amendments to claims 1-14 are accepted and entered.
The amendment to the specification is accepted and entered.
The amendment to the abstract is accepted and entered.
Priority
The present application claims foreign priority benefits from GB2400900.3 filed on
01/24/2024. The certified copies of the priority documents were electronically retrieved on 03/07/2025.
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 01/23/2025 is considered and attached.
Claim Objections
Claim 2 is objected to because of the following informalities:
Claim 2 recites “wherein: the semantic model is further operable to generate the aggregated value in relation to at least one subject observed in the observation scene comprising:”. Since this step is already recited in claim 1, upon which claim 2 is dependent, it is clear applicant intends for this portion to act as part of the preamble. As such, the following amendment is suggested: “wherein: the semantic model further comprises analysing:”. See also the 112(b) rejection below regarding the recitation of “at least one subject” in the above portion of claim 2.
Appropriate correction is required.
Claim Rejections - 35 USC § 101 (abstract idea)
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-14 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Analysis for claim 1 is provided in the following. Claim 1 is reproduced in the following (annotation added):
An in-vehicle information processing system comprising:
at least one sensing device operable to obtain sensing data in relation to an observation scene; and
a processor including a memory having a set of instruction stored thereon, the set of instructions stored thereon retrievable by the processor, the processor further comprising:
a semantic model operable to generate an aggregated value in relation to at least one subject observed in the observation scene; and
in response to the aggregated value generated by a semantic model, the processor is further operable to analyse a motion of the at least one subject observed in the observation scene.
Step 1: Evaluating whether the claim belongs to one of the statutory categories.
Claim 1 recites at least one step or act. Thus, the claim is directed to a machine, which is one of the statutory categories of invention (Step 1: YES)
Step 2A Prong One: Evaluating whether the claim recites a judicial exception (an abstract idea enumerated in 2019 PEG, a law of nature, or a natural phenomenon). If no exception is recited, the claim is eligible. This concludes the eligibility analysis. If the claim recites an exception, go to Step 2A Prong Two.
Claim 1 recites an abstract idea of a mental process. At least steps c and d are recited at a high level of generality such that they could be practically performed by a human (The courts consider a mental process (thinking) that “can be performed in the human mind, or be a human using a pen and paper” to be an abstract idea.). These concepts fall into the “mental processes” group of abstract ideas, which is observation, evaluation and/or judgement. The process of generating an aggregated value in relation to a subject in a scene in order to analyse a motion of the subject can ultimately be done by hand. For example, one may mentally combine values associated with the speeds, poses, or positions of a subject in order to analyze the type of gesture of said subject. As such, calculating the above aggregated value using computers constitutes mere automation of manual processes. MPEP, 2106.04 (a) (2) III (C): Performing a mental process on a generic computer. An example of a case identifying a mental process performed on a generic computer as an abstract idea is Voter Verified, Inc. v. Election Systems & Software, LLC, 887 F.3d 1376, 1385, 126 USPQ2d 1498, 1504 (Fed. Cir. 2018). The limitations, interpreted under their broadest reasonable interpretation and in consistence with the specification, cover performance of the limitations in the mind or by generic computer components. See MPEP 2106.04 and the 2019 PEG. (Step 2A Prong One YES)
Step 2A Prong Two: Evaluating whether the claim recites additional elements that integrate the exception into a practical application of the exception. This evaluation is performed by (a) identifying whether there are any additional elements recited in the claim beyond judicial exception, and (b) evaluating those additional elements individually and in combination to determine whether the claim as a whole integrates the exception into practical application. If the answer to (a) is YES and (b) is NO, go to Step 2B; if the answer to (a) and (b) is YES, go to PATHWAY B, i.e., the claim is not directed to a judicial exception and the claim is eligible.
The 2019 PEG defines the phrase “integration into a practical application” to require an additional element or a combination of additional elements in the claim to apple, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that it is more than a drafting effort designed to monopolize the exception.
Limitations that are indicative of integration into a practical application when recited in a claim with a judicial exception include:
Improvements to the functioning of a computer, or to any other technology or technical field, as discussed in MPEP 2106.05(a);
Applying or using a judicial exception to affect a particular treatment or prophylaxis for disease or medical condition – see Vanda Memo
Applying the judicial exception with, or by use of, a particular machine, as discussed in MPEP 2106.05(b);
Effecting a transformation or reduction of a particular article to a different state or thing, as discussed in MPEP 2106.05©; and
Applying or using the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception, as discussed in MPEP 2106.05(e) and the Vanda Memo issued in June 2018.
Limitations that are not indicative of integration into a practical application when recited in a claim with a judicial exception include:
Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea, as discussed in MPEP 2106.05(f);
Adding insignificant extra-solution activity to the judicial exception, as discussed in MPEP 2106.05(g); and
Generally linking the use of the judicial exception to a particular technological environment or field of use, as discussed in MPEP 2106.05(h).
[Examiners should note that revised Step 2A excludes consideration of whether claim elements represent well-understood, routine, conventional activity. The question of whether claim elements represent only well-understood, routine, conventional activity is considered at Step 2B and is not a consideration in Step 2A.]
Steps a and b can be regarded as an additional element recited in claim 1. These additional elements, i.e., a sensing device capable of obtaining sensing data, a processor, a memory having a set of instructions, does not integrate the exception into a practical application of the exception. Note even if the specification discloses that the invention pertains to an improvement in the technology, the claim must be evaluated to ensure the claim itself reflects the improvement in technology. It is also important to note, the judicial exception alone cannot provide the improvement. The improvement can be provided by one or more additional elements. Therefore, the additional elements do not recite an improvement. (Step 2A Prong Two NO)
Step 2B: This part of the eligibility analysis evaluates whether the claim as a whole amounts to significantly more than the recited exception, i.e., whether any additional element, or combination of additional elements, adds an inventive concept to the claim.
Steps a and b can be regarded as additional elements recited in claim 1. These additional elements, i.e., a sensing device capable of obtaining sensing data, a processor, a memory having a set of instructions is considered insignificant extra-solution activities which amounts to automating a manual human activity.
In the instant case, the recited functional limitation in step a can be performed by a photographer (organizing human activity/ Mere automation of manual processes). Additionally, obtaining sensing data from a sensing device, and utilizing a processor including a memory having a set of instructions retrievable by the processor is a well-understood, routine, conventional activity in the field of image analysis. Furthermore, the elements of a processor and memory are recited at a high level of generality such that they amount to no more than mere generic computer system elements. Using the broadest reasonable interpretation of the claim, the additional elements, taken individually and in combination, do not result in the claim, as a whole, amounting to significantly more than the abstract idea itself. See MPEP 2106.05. (Step 2B: NO) The claim is not eligible.
Claim 2 recites
a set of observation sensing data, the set of observation sensing data comprising one or more parametric data aggregated from the observation scene,
a set of latent sensing data, the set of latent sensing data comprising one or more parametric data extracted from the observation scene, or combination thereof
Steps a-b are directed to the abstract idea of mental processes which are mere automation of manual processes. Generating/obtaining these sets of data is a process that may be done manually. The similar examination analysis as applied to claim 1 is applied to steps a-b of claim 2. No additional elements are recited. Accordingly, claim 2 does not have eligible subject matter.
Claim 3 recites “wherein: the one or more parametric data comprises: at least one certain confidence value; at least one uncertain confidence value, or combination thereof”. Claim 3 contains steps that are directed to the abstract idea of mental processes which are mere automation of manual processes, wherein determining certain/uncertain confidence values is a process which may be done manually. Accordingly, claim 3 does not have eligible subject matter.
Claim 4 recites “wherein the aggregated value generated by the semantic model comprises: an average confidence value generated from the one or more parametric data in relation to the at least one subject observed, wherein the one or more parametric data include at least one uncertain confidence value”. Claim 4 contains steps that are directed to the abstract idea of mental processes which are mere automation of manual processes, wherein determining an average confidence value is a process which may be done manually. Accordingly, claim 4 does not have eligible subject matter.
Claim 5 recites “wherein the aggregated value generated by the semantic model comprises: a spectrum of data in relation to the at least one subject observed, the spectrum of data representing the at least one uncertain confidence value of the one or more parametric data”. Claim 5 contains steps that are directed to the abstract idea of mental processes which are mere automation of manual processes, wherein determining a spectrum of data representing at least one uncertain confidence value is a process which may be done manually. Accordingly, claim 5 does not have eligible subject matter.
Claim 6 recites “wherein the aggregated value generated by the semantic model comprises: a weighted confidence value, wherein the weighted confidence value is computed from the at least one certain confidence value of the one or more parametric data in relation to the at least one subject observed”. Claim 6 contains steps that are directed to the abstract idea of mental processes which are mere automation of manual processes, wherein determining a weighted confidence value is a process which may be done manually. Accordingly, claim 6 does not have eligible subject matter.
Claim 7 recites “wherein the aggregated value generated by the semantic model comprises: a confidence error value between the set of observation sensing data and the set of latent sensing data extracted by the semantic model”. Claim 7 contains steps that are directed to the abstract idea of mental processes which are mere automation of manual processes, wherein determining a confidence error value based on two sets of data is a process which may be done manually. Accordingly, claim 7 does not have eligible subject matter.
Claim 8 recites “wherein the at least one sensing device comprises: an imaging device; an image sensing device; an in-vehicle camera of a driver monitoring system; an in-vehicle camera of a passenger cabin monitoring system; or combination thereof”. Claim 8 contains steps that are recited at a high level of generality such that they amount to no more than mere generic computer system elements. Accordingly, claim 8 does not have eligible subject matter
Independent claim 9 is directed to a process, which is a statutory category of invention. Similar analysis is applicable as applied above to the method of claim 1. Accordingly claim 9 does not have eligible subject matter.
Claim 10 recites “aggregating, by way of the semantic model, a set of observation sensing data from the sensing data obtained; extracting, by way of the semantic model, a set of latent sensing data from the sensing data obtained; or combination thereof”. Claim 10 contains steps that are directed to the abstract idea of mental processes which are mere automation of manual processes, wherein generating/obtaining these sets of data is a process that may be done manually. Accordingly, claim 10 does not have eligible subject matter.
Claim 11 recites “analysing, by way of the processor, the aggregated value generated by the semantic model, wherein the aggregated value comprises: an average confidence value in relation to the at least one subject observed; a spectrum of data in relation to the at least one subject observed; a weighted confidence value in relation to the at least one subject observed, or combination thereof”. Claim 11 contains steps that are directed to the abstract idea of mental processes which are mere automation of manual processes, wherein averaging a confidence value, generating a spectrum of data, and weighting confidence values are processes that may be done manually. Accordingly, claim 11 does not have eligible subject matter.
Claim 12 recites “generating, by way of the semantic model, a confidence error value between the set of observation sensing data and the set of latent sensing data”. Claim 12 contains steps that are directed to the abstract idea of mental processes which are mere automation of manual processes, wherein determining a confidence error value based on two sets of data is a process which may be done manually. Accordingly, claim 12 does not have eligible subject matter.
Claim 13 recites “A computer program product comprising instructions which, when the program is executed by a processor, cause the processor to carry out the method of claim 9”. Claim 13 contains steps that are recited at a high level of generality such that they amount to no more than mere generic computer system elements. Accordingly, claim 13 does not have eligible subject matter. Please note that, while claim 13 fails prong 1 (see the below 101 rejection of claim 13 as it is directed to non-statutory subject matter), if resolved, claim 13 would still be rejected under 101 (abstract idea).
Claim 14 recites “A non-transitory computer-readable medium having stored thereon the computer program product of claim 13”. Claim 14 contains steps that are recited at a high level of generality such that they amount to no more than mere generic computer system elements. Accordingly, claim 14 does not have eligible subject matter.
Claim Rejections - 35 USC § 101 (non-statutory subject matter)
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 13 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because data per se and/or computer programs do not fall into one of the four categories of statutory invention (machine, process, manufacture, composition). More specifically, claims are eligible for patent protection under § 101 if they are in one of the four statutory categories and not directed to a judicial exception to patentability (i.e., laws of nature, natural phenomena, and abstract ideas). Alice Corp. v. CLS Bank Int'l, 573 U. S. 208 (2014)
Regarding claim 13, the claim is drawn towards a “computer program product”. As described in MPEP § 2106, data per se and computer programs do not fall into one of the four statutory categories. Therefore, since claim 13 is drawn towards a program, the claim is not eligible for patent protection.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-8 and 12 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 1, claim 1 recites “a semantic model” in line 8 and “a semantic model” in line 11. As such, it is unclear whether the semantic model as claimed in line 8 is equivalent to or distinct from the semantic model as claimed in line 11. Applicant discusses the semantic model throughout the specification. However, nowhere in the specification does the applicant explain whether the two semantic models as introduced in claim 1 are equivalent or distinct. As such, claim 1 is rejected for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Claims 2-8 are rejected due to their dependency upon rejected claim 1.
Regarding claim 2, claim 2 recites “at least one subject” in line 3. However, “at least one subject” is already introduced in line 9 of claim 1, upon which claim 2 is dependent. As such, it is unclear whether the at least one subject as recited in line 3 of claim 2 is equivalent to or distinct from the at least one subject as recited in line 9 of claim 1. Applicant discusses the at least one subject throughout the specification. However, nowhere in the specification does the applicant clarify whether the at least one subject of claim 2 is equivalent to or distinct from the at least one subject of claim 1. In regards to prior art interpretation, it will be assumed that the at least one subject of claim 1 is equivalent to the at least one subject of claim 2.
Additionally, regarding claim 2, claim 2 recites “one or more parametric data” in line 6 and “one or more parametric data” in lines 8-9. As such, it is unclear whether the one or more parametric data as recited in line 6 of claim 2 is equivalent to or distinct from the one or more parametric data as recited in lines 8-9 of claim 2. Applicant discusses the one or more parametric data on page 2, line 23 through page 3, line 7 and page 8, lines 30-33. However, nowhere in these sections does the applicant clarify whether the one or more parametric data in line 6 is equivalent to or distinct from the one or more parametric data in lines 8-9. As such, claim 2 is rejected for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Claim 3 is rejected due to its dependence upon claim 2.
Regarding claim 3, claim 3 recites “the one or more parametric data” in line 3. However, as noted above in the 112(b) rejection of claim 2, there are multiple instances of “one or more parametric data” recited in claim 2 that the reference of “the one or more parametric data” as recited in claim 3 may be referring to. As such, it is unclear which parametric data is being referenced to in claim 3, and claim 3 subsequently is rejected for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Regarding claim 4, claim 4 recites “the one or more parametric data” in line 4 and “the one or more parametric data” in lines 5-6. However, there is insufficient antecedent basis for the one or more parametric data in claim 4. As such, claim 4 is rejected for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Regarding claim 5, claim 5 recites “the at least one uncertain confidence value of the one or more parametric data” in lines 5-6. However, there is insufficient antecedent basis for the at least one uncertain confidence value and the one or more parametric data in claim 5. As such, claim 5 is rejected for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Regarding claim 6, claim 6 recites “the at least one certain confidence value of the one or more parametric data” in lines 5-6. However, there is insufficient antecedent basis for the at least one certain confidence value and the one or more parametric data in claim 6. As such, claim 6 is rejected for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Regarding claim 7, claim 7 recites “the set of observation sensing data and the set of latent sensing data” in lines 4-5. However, there is insufficient antecedent basis for the set of observation sensing data and the set of latent sensing data in claim 7. As such, claim 7 is rejected for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Regarding claim 12, claim 12 recites “the set of observation sensing data and the set of latent sensing data” in line 4. However, there is insufficient antecedent basis for the set of observation sensing data and the set of latent sensing data in claim 12. As such, claim 12 is rejected for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5-10, and 12-14 are rejected under 35 U.S.C. 103 as being unpatentable over Ren et al. (“A Multi-Semantic Driver Behavior Recognition Model of Autonomous Vehicles Using Confidence Fusion Mechanism”), hereinafter Ren, in view of Porta et al. (U.S. Publication No. 2021/0012126 A1), hereinafter Porta.
Regarding claim 1, Ren teaches an
a semantic model (Ren teaches “an end-to-end model with two parallel branches, called MSRNet, is employed to perform driver behavior recognition” in Section 2 and Figure 3) operable to generate an aggregated value in relation to at least one subject observed in the observation scene (Ren teaches “an end-to-end model with two parallel branches, called MSRNet, is employed to perform driver behavior recognition” in Section 2, wherein MSRNet includes both ObjectNet and ActionNet and “the Confidence Fusion Mechanism (CFM) is introduced to aggregate predictions from both ActNet and ObjectNet based on the semantic relationships between actions and key-objects” as shown in Section 2.4. Here, the aggregated prediction values are interpreted as equivalent to the claimed aggregated value); and
in response to the aggregated value generated by a semantic model, the processor is further operable to analyse a motion of the at least one subject observed in the observation scene (Ren teaches “the CFM is a decision fusion approach that combines the decisions of multiple classifiers into a common decision about driver behavior” as shown in Section 2.4. This decision about driver behavior is interpreted as equivalent to the analyzed motion of the subject observed in the observation scene. See Figure 5).
Ren fails to teach an in-vehicle information processing system comprising: at least one sensing device operable to obtain sensing data in relation to an observation scene; and a processor including a memory having a set of instruction stored thereon, the set of instructions stored thereon retrievable by the processor.
However, Porta teaches an in-vehicle information processing system (Porta teaches “interior camera systems configured to monitor vehicle occupants” in para. [0009] and FIGs. 3 and 9) comprising: at least one sensing device operable to obtain sensing data in relation to an observation scene (Porta teaches “interior camera systems configured to monitor vehicle occupants” in para. [0009] and FIGs. 3 and 9); and a processor including a memory having a set of instruction stored thereon, the set of instructions stored thereon retrievable by the processor (Porta teaches “the processors 106a-106n may perform the computer vision operations on the video frames” in para. [0177] wherein the memory/instruction are taught in para. [0188]).
Ren and Porta are both considered to be analogous to the claimed invention because they are in the same field of tracking driver behaviors. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Ren to incorporate the teachings of Porta and include “an in-vehicle information processing system comprising: at least one sensing device operable to obtain sensing data in relation to an observation scene; and a processor including a memory having a set of instruction stored thereon, the set of instructions stored thereon retrievable by the processor”. The motivation for doing so would have been to “improve road safety by reducing distracted driving”, “prevent the driver from being fined because of illegal and/or unauthorized phone use”, and to implement the functions of the method(s) as suggested by Porta in para. [0020] and para. [0186], respectively. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Ren with Porta to obtain the invention specified in claim 1.
Regarding claim 2, Ren and Porta teach the system according to claim 1, wherein:
the semantic model is further operable to generate the aggregated value in relation to at least one subject observed in the observation scene (Ren teaches that the semantic model (MSRNet as shown in Figure 3) takes in an input video wherein the video depicts a driver engaging in activities in a vehicle environment (observation scene) in Section 2.1.) comprising:
a set of observation sensing data, the set of observation sensing data comprising one or more parametric data aggregated from the observation scene,
a set of latent sensing data, the set of latent sensing data comprising one or more parametric data extracted from the observation scene (Ren teaches “the proposed model uses 3D-CNN to extract spatiotemporal features, which is able to capture motion information encoded in multiple consecutive frames” in Section 2.2. Here, these extracted spatiotemporal features are interpreted as equivalent to the claimed set of latent sensing data),
or combination thereof.
Regarding claim 3, Ren and Porta teach the system according to claim 2, wherein:
the one or more parametric data comprises:
at least one certain confidence value;
at least one uncertain confidence value,
or combination thereof (Ren teaches that “the outputs of ActNet and ObjectNet are reshaped to the same dimension (i.e., class index, four coordinates, and confidence score)” as shown in Section 2.4. See Section 2.4 and Figure 5 wherein the confidence scores can either be high/certain (above a threshold/matching action and key-object) or low (below a threshold/unmatched action and key-object)).
Regarding claim 5, Ren and Porta teach the system according to claim 1,
wherein the aggregated value generated by the semantic model comprises:
a spectrum of data in relation to the at least one subject observed, the spectrum of data representing the at least one uncertain confidence value of the one or more parametric data (Ren teaches determining a plurality of driver behavior predictions which each have confidence scores (see section 2.4), wherein the scores can be either high/certain (above a threshold/matching action and key-object) or low (below a threshold/unmatched action and key-object). Since the plurality of confidence scores include both scores above and below the threshold and relate to the behavior of a driver (as shown in section 2.4), the confidence scores are interpreted as equivalent to the claimed spectrum of data).
Regarding claim 6, Ren and Porta teach the system according to claim 1, wherein the aggregated value generated by the semantic model comprises:
a weighted confidence value, wherein the weighted confidence value is computed from the at least one certain confidence value of the one or more parametric data in relation to the at least one subject observed (While Ren teaches that “the outputs of ActNet and ObjectNet are reshaped to the same dimension (i.e., class index, four coordinates, and confidence score)” as shown in Section 2.4. See Section 2.4 and Figure 5 wherein the confidence scores can either be high/certain (above a threshold/matching action and key-object) or low (below a threshold/unmatched action and key-object), Porta teaches determining a weighted confidence value, wherein the weighted confidence value can be derived from both certain (higher weighting factor) and uncertain (lower weighting factor) confidence values, and wherein the confidence values are derived in relation to the driver of a vehicle as shown in para. [0175]. See also para. [0169]-[0174]).
Regarding claim 7, Ren and Porta teach the system according to claim 1, wherein the aggregated value generated by the semantic model comprises:
a confidence error value between the set of observation sensing data (Ren teaches “ObjectNet is utilized to extract key-object features from key frames” in Section 2 in the form of predictions. These predictions are interpreted as equivalent to the set of observation sensing data) and the set of latent sensing data (Ren teaches “ActNet is used to extract spatiotemporal features from input clips, which can capture the action cues of driver behaviors” in Section 2 in the form of predictions. These predictions are interpreted as equivalent to the set of latent sensing data) extracted by the semantic model (Ren teaches comparing the predictions output by the ActNet (based on their confidence scores) and the predictions output by the ObjectNet to determine whether they match. An output of either yes or no is determined based on this comparison. This output is interpreted as equivalent to the claimed confidence error value). It should be noted that the set of observation sensing data and the set of latent sensing data are not present in claim 1, upon which claim 7 depends. As such, these sets may be broadly interpreted. See also that applicant’s specification recites that “The confidence error value may be a disparity between the set of observation sensing data and the set of latent sensing data extracted by the latent feature extraction model” on page 3, lines 30-35. Ren’s teachings clearly define a disparity between the two sets of data as shown in the mapping above.
Regarding claim 8, Ren and Porta teach the system according to claim 1, wherein the at least one sensing device comprises:
an imaging device;
an image sensing device;
an in-vehicle camera of a driver monitoring system (Porta teaches “interior camera systems configured to monitor vehicle occupants” in para. [0009] and FIGs. 3 and 9);
an in-vehicle camera of a passenger cabin monitoring system; or
combination thereof.
Similar motivations as applied to claim 1 can be applied here to claim 8.
Regarding claim 9, Ren teaches an
obtaining, (Ren teaches that an input video capturing a driver in a driving environment is obtained as shown in Section 1 and Figure 3); and
determining, (Ren teaches that “the predictions from both branches are fed into the CFM to perform confidence fusion and action classification based on the semantic relationships between actions and key objects” in regards to driver behavior in a vehicle environment as shown in Section 1), characterised by that the method further comprises:
generating, by way of a semantic model (Ren teaches “an end-to-end model with two parallel branches, called MSRNet, is employed to perform driver behavior recognition” in Section 2 and Figure 3), an aggregated value in relation to at least one subject observed in the observation scene (Ren teaches “an end-to-end model with two parallel branches, called MSRNet, is employed to perform driver behavior recognition” in Section 2, wherein MSRNet incudes both ObjectNet and ActionNet and “the Confidence Fusion Mechanism (CFM) is introduced to aggregate predictions from both ActNet and ObjectNet based on the semantic relationships between actions and key-objects” as shown in Section 2.4. Here, the aggregated prediction values are interpreted as equivalent to the claimed aggregated value),
wherein determining the motion of the at least one subject observed in the observation scene is in response to the aggregated value generated by the semantic model (Ren teaches “the CFM is a decision fusion approach that combines the decisions of multiple classifiers into a common decision about driver behavior” as shown in Section 2.4. This decision about driver behavior is interpreted as equivalent to the analyzed motion of the subject observed in the observation scene. See Figure 5).
Ren fails to teach an in-vehicle information processing method comprising: at least one sensing device; and a processor.
However, Porta teaches an in-vehicle information processing method (Porta teaches “interior camera systems configured to monitor vehicle occupants” in para. [0009] and FIGs. 3 and 9) comprising: at least one sensing device (Porta teaches “interior camera systems configured to monitor vehicle occupants” in para. [0009] and FIGs. 3 and 9); and a processor (Porta teaches “the processors 106a-106n may perform the computer vision operations on the video frames” in para. [0177] wherein the memory/instruction are taught in para. [0188]).
Ren and Porta are both considered to be analogous to the claimed invention because they are in the same field of tracking driver behaviors. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Ren to incorporate the teachings of Porta and include “an in-vehicle information processing system comprising: at least one sensing device operable to obtain sensing data in relation to an observation scene; and a processor including a memory having a set of instruction stored thereon, the set of instructions stored thereon retrievable by the processor”. The motivation for doing so would have been to “improve road safety by reducing distracted driving”, “prevent the driver from being fined because of illegal and/or unauthorized phone use”, and to implement the functions of the method(s) as suggested by Porta in para. [0020] and para. [0186], respectively. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Ren with Porta to obtain the invention specified in claim 9.
Regarding claim 10, Ren and Porta teach the method of claim 9, further comprising:
aggregating, by way of the semantic model, a set of observation sensing data from the sensing data obtained;
extracting, by way of the semantic model, a set of latent sensing data from the sensing data obtained (Ren teaches “the proposed model uses 3D-CNN to extract spatiotemporal features, which is able to capture motion information encoded in multiple consecutive frames” in Section 2.2. Here, these extracted spatiotemporal features are interpreted as equivalent to the claimed set of latent sensing data); or
combination thereof.
Regarding claim 12, Ren and Porta teach the method of claim 9, the method further comprising:
generating, by way of the semantic model, a confidence error value between the set of observation sensing data (Ren teaches “ObjectNet is utilized to extract key-object features from key frames” in Section 2 in the form of predictions. These predictions are interpreted as equivalent to the set of observation sensing data) and the set of latent sensing data (Ren teaches “ActNet is used to extract spatiotemporal features from input clips, which can capture the action cues of driver behaviors” in Section 2 in the form of predictions. These predictions are interpreted as equivalent to the set of latent sensing data) extracted by the semantic model (Ren teaches comparing the predictions output by the ActNet (based on their confidence scores) and the predictions output by the ObjectNet to determine whether they match. An output of either yes or no is determined based on this comparison. This output is interpreted as equivalent to the claimed confidence error value). It should be noted that the set of observation sensing data and the set of latent sensing data are not present in claim 9, upon which claim 12 depends. As such, these sets may be broadly interpreted. See also that applicant’s specification recites that “The confidence error value may be a disparity between the set of observation sensing data and the set of latent sensing data extracted by the latent feature extraction model” on page 3, lines 30-35. Ren’s teachings clearly define a disparity between the two sets of data as shown in the mapping above.
Regarding claim 13, Ren and Porta teach a computer program product comprising instructions which, when the program is executed by a processor, cause the processor to carry out the method of claim 9 (Porta teaches “the functions performed by the diagrams of FIGS. 1-12 may be implemented using one or more of a conventional general purpose processor” as shown in para. [00186], wherein “the invention thus may also include a computer product which may be a storage medium or media and/or a transmission medium or media including instructions which may be used to program a machine to perform one or more processes or methods in accordance with the invention” as shown in para. [0188]. Here, the motivation would have been to implement the functions of the method(s) as described by Porta as suggested in para. [0186]).
Regarding claim 14, Ren and Porta teach a non-transitory computer-readable medium having stored thereon the computer program product of claim 13 (Porta teaches “execution of instructions contained in the computer product by the machine, along with operations of surrounding circuitry, may transform input data into one or more files on the storage medium”, wherein the storage medium may include physical storage devices (non-transitory crms) as shown in para. [0188]).
Claims 4 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Ren et al. (“A Multi-Semantic Driver Behavior Recognition Model of Autonomous Vehicles Using Confidence Fusion Mechanism”), hereinafter Ren, in view of Porta et al. (U.S. Publication No. 2021/0012126 A1), hereinafter Porta and Sarratt et al. (U.S. Publication No. 2015/0098609 A1), hereinafter Sarratt.
Regarding claim 4, Ren and Porta teach the system according to claim 1.
Ren and Porta fail to teach wherein the aggregated value generated by the semantic model comprises: an average confidence value generated from the one or more parametric data in relation to the at least one subject observed, wherein the one or more parametric data include at least one uncertain confidence value.
However, Sarratt teaches wherein the aggregated value generated by the semantic model comprises: an average confidence value generated from the one or more parametric data in relation to the at least one subject observed, wherein the one or more parametric data include at least one uncertain confidence value (Sarratt teaches determining confidence scores for predictions based on driver actions, and combining the confidence scores for each prediction in a group, wherein “a [] technique for combining the scores may be used such as, for example, computing a weighted combination, an average, a median, etc” as shown in para. [0053]).
Ren, Porta, and Sarratt are all considered to be analogous to the claimed invention because they are in the same field of tracking driver behaviors. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Ren (as modified by Porta) to incorporate the teachings of Sarratt and include “wherein the aggregated value generated by the semantic model comprises: an average confidence value generated from the one or more parametric data in relation to the at least one subject observed, wherein the one or more parametric data include at least one uncertain confidence value”. The motivation for doing so would have been to create an action recognition system that “combines the confidence scores associated with each driver action and selects the driver action with the highest combined score”, as suggested by Sarratt in para. [0021]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Ren and Porta with Sarratt to obtain the invention specified in claim 4.
Regarding claim 11, Ren and Porta teach the method of claim 9,
the method further comprising:
analysing, by way of the processor, the aggregated value generated by the semantic model, wherein the aggregated value comprises: a spectrum of data in relation to the at least one subject observed (Ren teaches determining a plurality of driver behavior predictions which each have confidence scores (see section 2.4), wherein the scores can be either high/certain (above a threshold/matching action and key-object) or low (below a threshold/unmatched action and key-object). Since the plurality of confidence scores include both scores above and below the threshold and relate to the behavior of a driver (as shown in section 2.4), the confidence scores are interpreted as equivalent to the claimed spectrum of data);
a weighted confidence value in relation to the at least one subject observed, or combination thereof (While Ren teaches that “the outputs of ActNet and ObjectNet are reshaped to the same dimension (i.e., class index, four coordinates, and confidence score)” as shown in Section 2.4. See Section 2.4 and Figure 5 wherein the confidence scores can either be high/certain (above a threshold/matching action and key-object) or low (below a threshold/unmatched action and key-object), Porta teaches determining a weighted confidence value, wherein the weighted confidence value can be derived from both certain (higher weighting factor) and uncertain (lower weighting factor) confidence values, and wherein the confidence values are derived in relation to the driver of a vehicle as shown in para. [0175]. See also para. [0169]-[0174]).
Ren and Porta fail to teach an average confidence value in relation to the at least one subject observed.
However, Sarratt teaches an average confidence value in relation to the at least one subject observed (Sarratt teaches determining confidence scores for predictions based on driver actions, and combining the confidence scores for each prediction in a group, wherein “a [] technique for combining the scores may be used such as, for example, computing a weighted combination, an average, a median, etc” as shown in para. [0053]).
Ren, Porta, and Sarratt are all considered to be analogous to the claimed invention because they are in the same field of tracking driver behaviors. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Ren (as modified by Porta) to incorporate the teachings of Sarratt and include “an average confidence value in relation to the at least one subject observed”. The motivation for doing so would have been to create an action recognition system that “combines the confidence scores associated with each driver action and selects the driver action with the highest combined score”, as suggested by Sarratt in para. [0021]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Ren and Porta with Sarratt to obtain the invention specified in claim 11.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Quader et al. (U.S. Publication No. 2020/0366960 A1) teaches determining driver gestures using an in-vehicle camera system by combining probabilities generated by distinct action recognizers.
Taha et al. (U.S. Publication No. 2020/0234086 A1) teaches obtaining multi-modal data representing driving events and corresponding actions related to the driving events wherein a confidence loss is generated based on the modelled uncertainty of the multiple modalities.
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to KYLA G ALLEN whose telephone number is (703)756-5315. The examiner can
normally be reached M-F 7:30am - 4:30pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a
USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use
the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor,
John Villecco can be reached on (571) 272-7319. The fax phone number for the organization where this
application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from
Patent Center. Unpublished application information in Patent Center is available to registered users. To
file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit
https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and
https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional
questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like
assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or
571-272-1000.
/Kyla Guan-Ping Tiao Allen/
Examiner, Art Unit 2661
/AARON W CARTER/Primary Examiner, Art Unit 2661
Prosecution Insights