DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged that application claims priority to foreign application
with application number KR10-2023-0180920 dated 12/13/2023. Copies of certified
papers required by 37 CFR 1.55 have been retrieved.
Information Disclosure Statement
The information disclosure statement (“IDS”) filed on 10/07/2024 has been reviewed and the listed references have been considered.
Status of Claims
Claims 1-20 are pending.
Drawings
The drawings are objected to under 37 CFR 1.83(a) because they fail to show the top 25% features are used for GNC combiner and bottom 25% of features are used for MSE loss as described in the specification paragraph 55. Figure 3 does show the GNC combiner and MSE loss however the labels are not consistent with the claims and specification description. Any structural detail that is essential for a proper understanding of the disclosed invention should be shown in the drawing. MPEP § 608.02(d).
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description: Figure 4, shows reference number 392 and 56. Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The specification is objected to because of the following informalities:
In paragraph 40, "memory device may correspond to a memory 520" should be "memory device may correspond to a memory 530"
In paragraph 50, "the second-level feature (GF)" should be "the second-level feature (LF)"
According to 37 CFR 1.71, MPEP §§ 608.01, 2161, and 2162, the specification must be in such particularity as to enable any person skilled in the pertinent art or science to make and use the invention without involving extensive experimentation and must clearly convey enough information about the invention to show that applicant invented the subject matter that is claimed. An applicant is ordinarily permitted to use his or her own terminology, as long as it can be understood. Necessary grammatical corrections, are required. Reference characters must be properly applied, no single reference character being used for two different parts or for a given part and a modification of such part. See 37 CFR 1.84(p). Every feature specified in the claims must be illustrated, but there should be no superfluous illustrations.
A substitute specification in proper idiomatic English and in compliance with 37 CFR 1.52(a) and (b) is required. The substitute specification filed must be accompanied by a statement that it contains no new matter.
Claim Objections
Claim 3 is objected to because of the following informality
Claim 3 recites "includes two the basic modules" should be "includes the plurality of basic modules"
Appropriate corrections are required.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claim 2 (similarly claim 19) rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. Claim 2 recites "The apparatus of claim 1, wherein the features selected from the first-level features and the features selected from the second-level features are provided as input to the fourth neural network", however the specification and drawings are not consist with the claim language as recited. The specification paragraph 52 states the fourth neural network is applied to "each patch region" which are the segmentation of "the output of the first neural network 21". Similarly, the drawing Figure 2 shows the fourth neural network 24 input is the segmented patches of the output of the first neural network, not the first and second level features. Therefore, claim 2 and 19 are rejected under 35 U.S.C. 112(a) for written description. For examination purposes examiner is interpreting the claims in light of the specification which describes the input of the fourth neural network being the segmented output of the first neural network.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-7, 10-19 are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. (CN109131167A - Translation from Espacenet) in view of Zhou et al. ("Learning Deep Global Multi-Scale and Local Attention Features for Facial Expression Recognition in the Wild" - From IDS ), in further view of Chang et al. ("Patch attention convolutional vision transformer for facial expression recognition with occlusion" - Published 2023), and in further view of Chou et al. ("A Novel Plug-in Module for Fine-Grained Visual Classification" - Published 2022).
Regarding claim 1, Li teaches “An apparatus for recognizing a facial expression of a vehicle occupant (Li paragraph [0021] "The method and apparatus for controlling a vehicle provided in this application first acquire facial perception information, which includes at least one of the following: facial emotion detection information"), the apparatus comprising:
one or more processors and one or more memory devices operably connected to the one or more processors, wherein the one or more memory devices include a program code, and wherein the program code is executed by the one or more processors (Li paragraph [0019] "an apparatus, including: one or more processors; a storage device for storing one or more programs; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement any of the methods described above") to
prepare an input image including a facial region to perform facial expression recognition (Li paragraph [0049] "Facial attribute recognition algorithms typically align the face based on the coordinates of key facial features (after operations such as rotation, scaling, and cutout, the face is adjusted to a predetermined size and shape), and then perform attribute analysis") of the vehicle occupant (Li paragraph [0009] "acquiring facial emotion detection information, which includes the emotional characteristics of the driver and /or passengers"),
input the input image (Li paragraph [0049] "Facial attribute recognition algorithms typically align the face based on the coordinates of key facial features (after operations such as rotation, scaling, and cutout, the face is adjusted to a predetermined size and shape), and then perform attribute analysis")
wherein the at least one processor (Li paragraph [0019] "an apparatus, including: one or more processors; a storage device for storing one or more programs; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement any of the methods described above") includes the classifier (Li paragraph [0043] "facial emotion detection information can be determined from the collected images based on facial emotion recognition technology. For example, it can identify human emotions such as happiness, anger, sadness, surprise, fear, aversion, contempt, confusion, and neutrality").
However, Li is not relied on to teach “to a first neural network including a plurality of basic modules including a residual block to obtain an output of the first neural network, apply a second neural network to the output of the first neural network to extract first-level features, segment the output of the first neural network into a plurality of local regions, and apply a third neural network to each of the local regions to extract a plurality of second-level features, segment the output of the first neural network into a plurality of patch regions greater than a number of the local regions, and apply a fourth neural network to each of the patch regions to extract third-level features, perform feature combination by selecting features corresponding to a top predetermined percentage of the first-level features including high classification confidence values, perform feature combination by selecting features corresponding to a top predetermined percentage of the second-level features including high classification confidence values, select features corresponding to a top predetermined percentage of the third-level features including high classification confidence values and concatenating the selected features, and concatenate the selected and concatenated features of the first-level features, the selected and concatenated features of the second-level features, and the selected and concatenated features of the third-level features, input the concatenated features of the first-level features, the second-level features and the third-level features to a classifier, and classify an emotion through the classifier”
Zhou teaches “a first neural network including a plurality of basic modules including a residual block to obtain an output of the first neural network (Zhou Figure 1 and page 4 left hand column paragraph 1 "The feature pre-extractor is to acquire middle-level facial features, and the feature pre-extractor consists of one 2D convolution layer and four basic blocks. The basic block structure shown in Fig. 2(a) is a basic building block utilized in ResNet-18 and ResNet-34 [49]"),
PNG
media_image1.png
446
713
media_image1.png
Greyscale
Zhou Figure 1
apply a second neural network to the output of the first neural network to extract first-level features (Zhou Figure 1 and page 4 left hand column paragraph 1 "In the first branch, we utilize a multi-scale module to learn global multi-scale features. Which takes whole extracted feature maps as input"),
segment the output of the first neural network into a plurality of local regions, and apply a third neural network to each of the local regions to extract a plurality of second-level features (Zhou Figure 1 and page 4 left hand column paragraph 1 "we first divide the extracted feature maps into several regional feature maps along the spatial axis without overlap. Then, several parallel local attention networks are utilized to learn local salient features. The extracted multi-scale feature maps and local attention feature maps are followed by a global average pooling layer and a fully-connected network respectively"),
(Zhou Figure 1 and page 4 left hand column paragraph 1 "In the first branch, we utilize a multi-scale module to learn global multi-scale features. Which takes whole extracted feature maps as input") including (Zhou Figure 1 and page 4 left hand column paragraph 1 "we first divide the extracted feature maps into several regional feature maps along the spatial axis without overlap. Then, several parallel local attention networks are utilized to learn local salient features. The extracted multi-scale feature maps and local attention feature maps are followed by a global average pooling layer and a fully-connected network respectively")
concatenate the selected and concatenated features of the first-level features, the selected and concatenated features of the second-level features, and the selected and concatenated features of the third-level features (Zhou page 6 left hand column paragraph 1 "The four local feature maps are then concatenated along the spatial axis, and the GAP layer is applied on the concatenated l 4 x l 4 x 5 l 2 feature maps to obtain a feature vector with a size of 512"), input the concatenated features of the first-level features, the second-level features and the third-level features to a classifier, and classify an emotion through the classifier (Zhou Figure 1 and page 4 left hand column paragraph 1 "a decision-level fusion is utilized to obtain recognition results")”.
It would have been obvious to a person having ordinary skill in the art before
effective filing date of the claimed invention of the instant application to combine a system and method of facial emotion recognition of a vehicle occupant as taught by Li to use multi-branch analysis an image using multiple recognition modules as taught by Zhou.
The suggestion/motivation for doing so would have been “Learning facial features from various perspectives may achieve better performance under occlusion and pose variation conditions, and the study on psychology indicates that human face perception mechanisms extract both holistic and part information [32]. To this end, we propose a global multi-scale and local attention network (MA-Net) from the global and local perspective to acquire robust features, which can address both occlusion and pose variation problems” as noted by the Zhou disclosure on page 2 left hand column paragraph 2.
However, the combination of Li and Zhou is not relied on to teach “segment the output of the first neural network into a plurality of patch regions greater than a number of the local regions, and apply a fourth neural network to each of the patch regions to extract third-level features, perform feature combination by selecting features corresponding to a top predetermined percentage […] including high classification confidence values […] and concatenating the selected features”.
Chang teaches “segment the output of the first neural network into a plurality of patch regions greater than a number of the local regions (Chang page 5 paragraph 2 "To get diverse local features, the facial feature maps are cropped into multiple patches by sliding cropping"), and apply a fourth neural network to each of the patch regions to extract third-level features (Chang page 5 paragraph 3 "The PAU is designed to learn local features from facial patches and dynamically estimate their importance for recognizing expressions. The architecture of the PAU is illustrated in Fig. 4. Specifically, the PAU first uses a two-layer convolution operation to extract the local features of each patch. Note that the spatial resolution of the patches is maintained to preserve more facial information")” and “concatenating the selected features [of the third level features] (Chang page 5 paragraph 1 "Each patch is weighted in terms of its importance for recognizing facial expressions under the attention mechanism. The local features of all patches are aggregated as follows where denotes the concatenation operation. The PAU adaptively calculates the weights of local features for all concat patches, which enables the network to shift its attention to unoccluded patches with salient expression features")”.
It would have been obvious to a person having ordinary skill in the art before effective filing date of the claimed invention of the instant application to combine a system and method of multi-branch facial emotion recognition of a vehicle occupant as taught by Li and Zhou to include a branch from patch level emotion recognition as taught by Chang.
The suggestion/motivation for doing so would have been that “According to human vision, when a target is occluded, humans tend to combine the features of salient local regions and the whole face for discrimination [42). Inspired by this fact, this paper presents a Patch Attention Convolutional Vision Transformer (PACVT) for recognizing expressions with occlusion” as noted by the Chang disclosure on page 2 paragraph 4.
However, the combination of Li, Zhou, and Chang is not relied on to teach “perform feature combination by selecting features corresponding to a top predetermined percentage […] including high classification confidence values”
Chou teaches “perform feature combination by selecting features corresponding to a top predetermined percentage (Chou page 8 left hand column paragraph 4 and right hand column paragraph 1 "If the confidence score is not enough, it can be further excluded. If threshold=0.9 is used as the boundary, it can be observed that the dropped area is still generally low") […] including high classification confidence values (Chou page 5 left hand column paragraph 1 "Then, the class prediction probability of each feature point is obtained through softmax, and the first few feature points with high confidence score will be selected in the weakly supervised selector")”.
It would have been obvious to a person having ordinary skill in the art before
effective filing date of the claimed invention of the instant application to combine a
system and method of multi-branch facial emotion recognition of a vehicle occupant as taught by Li, Zhou, and Chang to use confidence values for feature selection as taught by Chou.
The suggestion/motivation for doing so would have been that “In addition, in order to allow the model to extract the features of small regions more effectively, we add FPN to the backbone network to effectively fuse features of different scales to achieve more accurate recognition results” as noted by the Chou disclosure in page 4 left hand column paragraph 3.
Therefore, it would have been obvious to combine the disclosure of Li, Zhou, and Chang with the Chou disclosure to obtain the invention as specified in claim 1 as there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 12 recites a method with steps corresponding to the apparatus with elements recited in claim 1. Therefore, the recited steps of this claim are mapped to the
proposed combination in the same manner as the corresponding elements of apparatus
claim 1. Additionally, the rationale and motivation to combine the Li, Zhou, Chang, and Chou references, presented in rejection of claim 1 apply to this claim.
Claim 18 recites an apparatus with elements corresponding to the apparatus with elements recited in claim 1. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding elements of apparatus claim 1. Additionally, the rationale and motivation to combine the Li, Zhou, Chang, and Chou references, presented in rejection of claim 1 apply to this claim.
Regarding claim 2 (similarly claim 19), the combination of Li, Zhou, Chang, and Chou teaches “The apparatus of claim 1, wherein the features selected from the first-level features and the features selected from the second-level features (Chang page 5 paragraph 2 "To get diverse local features, the facial feature maps are cropped into multiple patches by sliding cropping") are provided as input to the fourth neural network (Chang page 5 paragraph 3 "The PAU is designed to learn local features from facial patches and dynamically estimate their importance for recognizing expressions. The architecture of the PAU is illustrated in Fig. 4. Specifically, the PAU first uses a two-layer convolution operation to extract the local features of each patch. Note that the spatial resolution of the patches is maintained to preserve more facial information").”
The proposed combination as well as the motivation for combining Li, Zhou, Chang, and Chou references presented in the rejection of claim 1, applies to claim 2. Finally, the apparatus recited in claim 1 is met by Li, Zhou, Chang, and Chou.
Regarding claim 3 (similarly claim 14), the combination of Li, Zhou, Chang, and Chou teaches “The apparatus of claim 1, wherein the first neural network includes an initial layer, and the plurality of residual blocks following the initial layer, and wherein the residual block includes two the basic modules (Zhou Figure 1 and page 4 left hand column paragraph 1 "The feature pre-extractor is to acquire middle-level facial features, and the feature pre-extractor consists of one 2D convolution layer and four basic blocks. The basic block structure shown in Fig. 2(a) is a basic building block utilized in ResNet-18 and ResNet-34 [49]").”
The proposed combination as well as the motivation for combining Li, Zhou, Chang, and Chou references presented in the rejection of claim 1, applies to claim 3. Finally, the apparatus recited in claim 3 is met by Li, Zhou, Chang, and Chou.
Regarding claim 4 (similarly claim 15), the combination of Li, Zhou, Chang, and Chou teaches “The apparatus of claim 1, wherein the second neural network includes a multi-scale module including a plurality of multi-scale blocks including filters of different sizes (Zhou Figure 1 and page 2 right hand column paragraph 2 "In the first branch, a multi-scale module is devised to learn robust multi-scale features towards occlusion and non-frontal pose conditions").”
The proposed combination as well as the motivation for combining Li, Zhou, Chang, and Chou references presented in the rejection of claim 1, applies to claim 4. Finally, the apparatus recited in claim 4 is met by Li, Zhou, Chang, and Chou.
Regarding claim 5 (similarly claim 16), the combination of Li, Zhou, Chang, and Chou teaches “The apparatus of claim 1, wherein the third neural network includes a convolutional block attention module (CBAM) that includes a channel attention module and a spatial attention module, and sequentially applies the channel attention module and the spatial attention module (Zhou Figure 1 and page 5 right hand column paragraph 3 "Fig, 2(c) shows the structure of the proposed attentive block. After two 3 x 3 convolution, we can obtain feature maps denoted by
F
ε
R
H
x
W
x
C
. Then, a convolutional block attention module (CBAM) [59] is employed as our attention net. The CRAM can sequentially infers attention maps along two separate dimensions, channel and spatial, then the attention maps are multiplied to the input feature map for adaptive feature refinement").”
The proposed combination as well as the motivation for combining Li, Zhou, Chang, and Chou references presented in the rejection of claim 1, applies to claim 5. Finally, the apparatus recited in claim 5 is met by Li, Zhou, Chang, and Chou.
Regarding claim 6 (similarly claim 17), the combination of Li, Zhou, Chang, and Chou teaches “The apparatus of claim 1, wherein the fourth neural network includes a patch attention module including a first basic module, a second basic module (Chang page 5 paragraph 3 "The architecture of the PAU is illustrated in Fig. 4. Specifically, the PAU first uses a two-layer convolution operation to extract the local features of each patch. Note that the spatial resolution of the patches is maintained to preserve more facial information. Then, the attention module is used to calculate the corresponding attention scalar of local features, and the local features are reshaped into vector-shaped ones"), a first CBAM, and a second CBAM which are sequentially connected (Zhou Figure 1 and page 5 right hand column paragraph 3 "Fig, 2(c) shows the structure of the proposed attentive block. After two 3 x 3 convolution, we can obtain feature maps denoted by
F
ε
R
H
x
W
x
C
. Then, a convolutional block attention module (CBAM) [59] is employed as our attention net. The CBAM can sequentially infers attention maps along two separate dimensions, channel and spatial, then the attention maps are multiplied to the input feature map for adaptive feature refinement").”
The proposed combination as well as the motivation for combining Li, Zhou, Chang, and Chou references presented in the rejection of claim 1, applies to claim 6. Finally, the apparatus recited in claim 6 is met by Li, Zhou, Chang, and Chou.
Regarding claim 7, the combination of Li, Zhou, Chang, and Chou teaches “The apparatus of claim 6, wherein the first basic module is implemented as 3 x 3 convolution and 64 filter, wherein the second basic module is implemented as 3 x 3 convolution and 128 filter (Zhou Figure 1 and page 5 right hand column paragraph 3 "Fig, 2(c) shows the structure of the proposed attentive block. After two 3 x 3 convolution, we can obtain feature maps denoted by
F
ε
R
H
x
W
x
C
"), wherein the first CBAM is implemented as 3 x 3 convolution and 256 filter, and wherein the second CBAM has 3 x 3 convolution and 512 filter (Zhou Figure 1 and page 5 right hand column paragraph 3 "Fig, 2(c) shows the structure of the proposed attentive block. After two 3 x 3 convolution, we can obtain feature maps denoted by
F
ε
R
H
x
W
x
C
. Then, a convolutional block attention module (CBAM) [59] is employed as our attention net. The CBAM can sequentially infers attention maps along two separate dimensions, channel and spatial, then the attention maps are multiplied to the input feature map for adaptive feature refinement").”
The proposed combination as well as the motivation for combining Li, Zhou, Chang, and Chou references presented in the rejection of claim 1, applies to claim 7. Finally, the apparatus recited in claim 7 is met by Li, Zhou, Chang, and Chou.
Regarding claim 10, the combination of Li, Zhou, Chang, and Chou teaches “The apparatus of claim 1, wherein the concatenation of the first-level features and the second-level features includes: inputting each of the first-level features and the second-level features into a graph convolutional network (GCN) combiner, to perform feature combination (Chou page 5 left hand column paragraph 2 "The second architecture is implemented through graph convolution, which treats all selected feature points as a graph structure, where nodes represent features at different spatial locations and scales. The graph is input into the graph convolutional network, which can learn the relationship between different nodes. And then, the feature points are aggregated into several super nodes through the pooling layer, and finally the features of these super nodes are averaged, and a linear classifier is used to complete the prediction. The advantage of this approach is that the features of each point can be integrated more efficiently without corrupting the results output by the backbone model. Therefore, we finally use graph convolution as the feature fusion mechanism").”
The proposed combination as well as the motivation for combining Li, Zhou, Chang, and Chou references presented in the rejection of claim 1, applies to claim 10. Finally, the apparatus recited in claim 10 is met by Li, Zhou, Chang, and Chou.
Regarding claim 11, the combination of Li, Zhou, Chang, and Chou teaches “The apparatus of claim 1, wherein the preparing of the input image including the facial region includes: obtaining an image from a camera (Li paragraph [0042] "acquire images from an image acquisition device (e.g., a camera or camera, etc.)") capturing the vehicle occupant (Li paragraph [0009] "acquiring facial emotion detection information, which includes the emotional characteristics of the driver and /or passengers");
detecting the facial region to perform the facial expression recognition in the image (Li paragraph [0042] "in response to the acquisition of facial information in the image, facial perception information is determined based on the facial information");
aligning the detected facial region (Li paragraph [0049] "Facial attribute recognition algorithms typically align the face based on the coordinates of key facial features (after operations such as rotation, scaling, and cutout, the face is adjusted to a predetermined size and shape)"); and
preparing a result of the aligning as the input image (Li paragraph [0049] "Facial attribute recognition algorithms typically align the face based on the coordinates of key facial features (after operations such as rotation, scaling, and cutout, the face is adjusted to a predetermined size and shape), and then perform attribute analysis").”
The proposed combination as well as the motivation for combining Li, Zhou, Chang, and Chou references presented in the rejection of claim 1, applies to claim 11. Finally, the apparatus recited in claim 11 is met by Li, Zhou, Chang, and Chou.
Regarding claim 13, the combination of Li, Zhou, Chang, and Chou teaches “The apparatus of claim 12, wherein the classifying of the emotion includes: selecting features corresponding to a top predetermined percentage of features of the remaining portion of the non-inactivated features including high classification confidence values (Chou page 5 left hand column paragraph 1 "Then, the class prediction probability of each feature point is obtained through softmax, and the first few feature points with high confidence score will be selected in the weakly supervised selector"), performing feature combination on the selected features or concatenating the selected features (Zhou page 6 left hand column paragraph 1 "The four local feature maps are then concatenated along the spatial axis, and the GAP layer is applied on the concatenated l 4 x l 4 x 5 l 2 feature maps to obtain a feature vector with a size of 512"), inputting the combined or concatenated features to the classifier, and classify the emotion through the classifier (Zhou Figure 1 and page 4 left hand column paragraph 1 "a decision-level fusion is utilized to obtain recognition results").”
The proposed combination as well as the motivation for combining Li, Zhou, Chang, and Chou references presented in the rejection of claim 1, applies to claim 13. Finally, the apparatus recited in claim 13 is met by Li, Zhou, Chang, and Chou.
Claims 8 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Li, Zhou, Chang, and Chou, in view of Shi et al. ("Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network" - Published 2016).
Regarding claim 8 (similarly claim 20), the combination of Li, Zhou, Chang, and Chou teaches “The apparatus of claim 1, wherein the segmenting of the output of the first neural network into the plurality of patch regions includes:
(Chang page 5 paragraph 2 "To get diverse local features, the facial feature maps are cropped into multiple patches by sliding cropping").
However, the combination of Li, Zhou, Chang, and Chou is not relied on to teach “performing up-sampling by applying a pixel shuffle”.
Shi teaches “performing up-sampling by applying a pixel shuffle (Shi Figure 1 and page 4 left hand column paragraph 2 "where PS is the shuffling operator that rear-ranges the elements of a
H
×
W
×
C
⋅
r
2
tensor to a rensor of shape " r
H
×
r
W
×
C
) to the output of the first neural network (Shi page 3 left hand column paragraph 3 "we propose to increase the resolution from LR to HR only at the very end of the network and super-resolve HR data from LR feature maps")”.
It would have been obvious to a person having ordinary skill in the art before
effective filing date of the claimed invention of the instant application to combine a
system and method of multi-branch facial emotion recognition of a vehicle occupant as taught by Li, Zhou, Chang, and Chou to include upsampling as taught by Shi.
The suggestion/motivation for doing so would have been “In our network, upscaling is handled by the last layer of the network. This means each LR image is directly fed to the network and feature extraction occurs through nonlinear convolutions in LR space. Due to the reduced input resolution, we can effectively use a smaller filter size to integrate the same information while maintaining a given contextual area. The resolution and filter size reduction lower the computational and memory complexity substantially enough to allow super-resolution of high definition (HD) videos in real time” as noted by the Shi disclosure on page 3 left hand column paragraph 4.
Therefore, it would have been obvious to combine the disclosure of Li, Zhou, Chang, and Chou with the Shi disclosure to obtain the invention as specified in claim 8 as there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Li, Zhou, Chang, and Chou, in view of Ghobadzadeh et al. (US 2020/0293807 A1).
Regarding claim 9, the combination of Li, Zhou, Chang, and Chou teaches “The apparatus of claim 1, wherein features corresponding to a bottom predetermined percentage of the first-level features to the third-level features including low classification confidence values (Chou page 8 left hand column paragraph 4 and right hand column paragraph 1 "If the confidence score is not enough, it can be further excluded. If threshold=0.9 is used as the boundary, it can be observed that the dropped area is still generally low") are used as
However, the combination of Li, Zhou, Chang, and Chou is not relied on to teach “mean squared error (MSE) loss”.
In an analogous field of endeavor, Ghobadzadeh teaches “as mean squared error (MSE) loss (Ghobadzadeh paragraph [0059] "Consider the output of the side network 210 as a vector with length of four relative bounding box values, {Z1,Z2,Z3 ,Z4} mean square error (MSE) may then be applied for refining the bounding box. The MSE may be denoted as Lbx and may be used when estimating the values in Equation (5)")”.
It would have been obvious to a person having ordinary skill in the art before
effective filing date of the claimed invention of the instant application to combine a
system and method of multi-branch facial emotion recognition of a vehicle occupant as taught by Li, Zhou, Chang, and Chou to include calculating a loss function as taught by Gobazadeh.
The suggestion/motivation for doing so would have been “During training, weights and biases in use in the Joint Face Alignment and Recognition system 114 adjusting based on optimizing loss functions determined while processing training data, the image with the refined bounding box is returned to the face recognition network 200 for improved learning of face features” as noted by the Gobazadeh. One of ordinary skill in the art would have recognized that using a MSE loss would improve the facial emotion recognition.
Therefore, it would have been obvious to combine the disclosure of Li, Zhou, Chang, and Chou with the Gobazadeh disclosure to obtain the invention as specified in claim 9 as there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASPREET KAUR whose telephone number is (571)272-5534. The examiner can normally be reached Monday - Friday 7:30 am - 4:00 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached at (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JASPREET KAUR/Examiner, Art Unit 2662
/AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662
Prosecution Insights