DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. TW112132895, filed on 08/30/2023.
Response to Amendment
Applicant’s Amendments filed on 06/24/2026 have been entered and made of record.
Status of Claims
Currently pending Claim(s):
Amended claim(s):
Canceled claim(s):
1-5, 8-13, and 16
1 and 9
6-7 and 14-15
Response to Arguments
This office action is responsive to Applicant’s Arguments/Remarks made in an Amendment received on 06/24/2026.
In view of amendments filed on 06/24/2026 to the title, the objection to the specification is withdrawn.
In view of the new claim amendments and applicant arguments, Remarks filed on 06/24/2026, with respect to the 35 U.S.C. 101 claim rejections have been carefully considered and the claims rejections to claims 1-6, 8-14, and 16 under 35 U.S.C. 101 are withdrawn.
In view of Applicant’s lack of written response with respect to 35 U.S.C. 112(f) claim interpretation, the interpretation made to claims 1, 3, and 8 is maintained. Applicant has not clarified why the claimed limitations do not invoke 112(f). According to MPEP 2181, 35 USC 112(f) is applicable to claim limitation if it meets the 3-prong analysis set forth in the previous Office Action. Applicant did not specifically point out why any of these prongs have not been met, and as such, the claims continue to be treated under 112(f). Applicant is welcome to amend the claim so that the limitations no longer invoke 112(f) by, e.g., modifying the “means” or generic placeholder with specific structure, with careful consideration that no new matter is introduced.
PNG
media_image1.png
302
708
media_image1.png
Greyscale
In view of applicant’s argument, Remarks filed on 06/24/2026, with respect to independent claims 1 and 9 under 35 U.S.C. 103, claim rejections have been fully considered but they are not persuasive. The Applicant argues on pages 10-11:
PNG
media_image2.png
292
674
media_image2.png
Greyscale
The Examiner respectfully disagrees. Senechal discloses “obtain a size of a face box” at column 6 [lines 23-34] “The providing of the output of the facial detector can include generating a bounding box 152 for the face. A first bounding box can be generated for a face that is detected in a first frame…The first bounding box can be a minimum-dimension bounding box, where the dimension can include area, volume, hyper-volume, and so on. The first bounding box can be generated based on analysis, estimation, simulation, prediction, and so on.”. The dimensions of the bounding box include the area, volume, and hyper-volume which are representative of the size of a face box.
Further, the rejection does not solely rely on Senechal to disclose, teach or suggest "obtaining a judgment value based on an ordinate of the first key point coordinate, an ordinate of the second key point coordinate, and the size of the face box". Rather, the Examiner relies on combination of Senechal in view of Cheng to teach "obtaining a judgment value based on an ordinate of the first key point coordinate, an ordinate of the second key point coordinate, and the size of the face box". Senechal is relied upon to teach an ordinate of the first key point coordinate, an ordinate of the second key point coordinate, and the size of the face box, see column 15 [line 32-35] “The learning can include mapping of the x-y coordinates (locations) of the facial landmarks to the coordinates of the bounding box 1030.” and column 6 [lines 23-34] “The providing of the output of the facial detector can include generating a bounding box 152 for the face. A first bounding box can be generated for a face that is detected in a first frame…The first bounding box can be a minimum-dimension bounding box, where the dimension can include area, volume, hyper-volume, and so on. The first bounding box can be generated based on analysis, estimation, simulation, prediction, and so on.”. Whereas, Cheng is relied on to teach obtaining a judgment value, see page 2206 left column paragraph 3 “we define θ as the angle between device’s x-axis and earth’s horizontal plane, and φ as the angle between device’s y-axis. We experimentally measured the orientation threshold used by iPhone and iPad, by monitoring the accelerometer readings and rotating the devices as slowly as possible until the screen rotated. We found that the threshold is θ - φ =30, with 2 degrees of dead band, for both iPhone and iPad.”.
PNG
media_image3.png
104
672
media_image3.png
Greyscale
On page 12 of Remarks, the Applicant argues that:
The Examiner respectfully disagrees. Cheng discloses at page 2206 left column paragraph 3 “we define θ as the angle between device’s x-axis and earth’s horizontal plane, and φ as the angle between device’s y-axis. We experimentally measured the orientation threshold used by iPhone and iPad, by monitoring the accelerometer readings and rotating the devices as slowly as possible until the screen rotated. We found that the threshold is θ - φ =30, with 2 degrees of dead band, for both iPhone and iPad.” and page 2207 [left column paragraph 4] “Our functional prototype automatically rotates screens to the orientation detected by the face detection API. It counts the number of frames with detected face orientation within a 0.5-second window, and rotates to the most frequently detected orientation. The 0.5-second threshold is the average rotation delay for iPhone and iPad”. This disclosure teaches obtaining a judgment value used to determine a threshold for rotation, and not merely the performance of screen rotation display in a hardware device. The detected orientation of the face is used to determine when rotation of the screen should occur.
On page 12 of Remarks, the Applicant argues that:
PNG
media_image4.png
144
668
media_image4.png
Greyscale
The Examiner respectfully disagrees. The Applicant’s argument does not address the actual reasoning of the Examiner’s rejections. Instead, the Applicant attack the references singly for lacking teachings that the Examiner relied on a combination of references to show. It is well established that one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references (See In re Keller, 642 F.2d 413). The court requires that references must be read, not in isolation, but for what they fairly teach in combination with the prior art as a whole (See In re Keller, 642 F.2d 413, 425 (CCPA 1981); In re Merck & Co., 800 F.2d 1091 (Fed. Cir. 1986)).
Further, this Office Actions has been updated to address the added limitations to independent claims 1 and 9, which were previously presented in claims 6-7 and 14-15, using Zeng et al ("Proposal pyramid networks for fast face detection." Information Sciences 495 (2019): 136-149.).
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) are:
“feature acquisition module” in claims 1, 3, 9, and 11
“judgment module” in claims 1 and 9
“output feature tensor generation module” in claims 1 and 9
“prediction modules” in claim 1 and 9
“display module” in claim 8 and 16
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
Claim 1, 3, 9, and 11: “feature acquisition module” corresponds to figure 1 element 101 “The judgment system 100 comprises a feature acquisition module 101 and a judgment module 102.” (Paragraph [0012].); “The processing units 1001-1 through 1001-R read corresponding computer programs…Such process forms the judgment system 100 and the electronic system 800…each module of the judgment system 100 and the electronic system 800 may also be implemented using hardware…processing units 1001-1 through 1001-R may be an integrated circuit chip with signal processing capability…The processing units 1001-1 through 1001-R may be general purpose processors…or other programmable logic devices” (Paragraph [0052-0053].
Claims 1 and 9: “judgment module” corresponds to corresponds to figure 1 element 102 “The judgment system 100 comprises a feature acquisition module 101 and a judgment module 102.” (Paragraph [0012].); “The processing units 1001-1 through 1001-R read corresponding computer programs…Such process forms the judgment system 100 and the electronic system 800…each module of the judgment system 100 and the electronic system 800 may also be implemented using hardware…processing units 1001-1 through 1001-R may be an integrated circuit chip with signal processing capability…The processing units 1001-1 through 1001-R may be general purpose processors…or other programmable logic devices” (Paragraph [0052-0053].
Claims 1 and 9: “output feature tensor generation module” corresponds to figure 4b element 401 the feature acquisition module 101 comprises a neural network module 400… the neural network module 400 comprises an output feature tensor generation module 401 and a prediction module” (Paragraph [0027]).
Claims 1 and 9: “prediction module” corresponds to figure 4b elements 402-1 through 402-M “the feature acquisition module 101 comprises a neural network module 400… the neural network module 400 comprises an output feature tensor generation module 401 and a prediction module 402-1 through a prediction module 402-M” (Paragraph [0027]).
Claim 8 and 16: “display module” corresponds to figure 10 element 1004 “The display element 1004 may be for example a liquid crystal display, a plasma display, a computer display (for example, a variable graphics array (VGA) display, a super VGA display or a cathode ray tube display), or a display device of another similar type, but the instant disclosure is not limited” (Paragraph [0051]).
Dependent claims 2, 4-5, 10, and 12-13 are similarly interpreted due to their dependency.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 8, 9, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Senechal et al. (US 10,614,289 B2) (hereinafter, “Senechal”) in view of Cheng et al. ("iRotate: automatic screen rotation based on face orientation." Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2012.) (hereinafter, “Cheng”), and further in view of Zeng et al ("Proposal pyramid networks for fast face detection." Information Sciences 495 (2019): 136-149.) (hereinafter, “Zeng”).
Regarding claim 1, Senechal discloses a judgment system comprising: a feature acquisition module configured to receive an image and obtain a first key point (facial landmarks in Column 5 [lines 31-39] equate to key point) coordinate, a second key point (facial landmarks in Column 5 [lines 31-39] equate to key point) coordinate (Column 5 [lines 31-39] “100 includes performing face detection to initialize locations 120 for a first set of facial landmarks (i.e. key points) within a first frame from the video. The face detection can be based on other facial points, identifying characteristics, etc. The landmarks can include corners of the mouth, corners of eyes, eyebrow corners, tip of nose, nostrils, chin, tips of ears, distinguishing marks and features, and so on.”; Column 15 [line 32-35] “The learning can include mapping of the x-y coordinates (locations) of the facial landmarks to the coordinates of the bounding box 1030.”), and a size of a face box (dimensions in Column 6 [lines 23-34] equate to size of face box) of a user based on the image (Column 6 [lines 23-34] “The providing of the output of the facial detector can include generating a bounding box 152 for the face. A first bounding box can be generated for a face that is detected in a first frame. The first bounding box can be a square, a rectangle, and/or any other appropriate geometric shape. The first bounding box can be substantially the same as the bounding box generated by a face detector. The first bounding box can be a minimum-dimension bounding box, where the dimension can include area, volume, hyper-volume (i.e. size of face box), and so on. The first bounding box can be generated based on analysis, estimation, simulation, prediction, and so on.”; Examiner interprets dimensions of the bounding box include the area, volume, and hyper-volume which are representative of the size of a face box); and
a judgment module configured to execute following steps: (a) obtaining a [judgment value] based on an ordinate of the first key point coordinate, an ordinate of the second key point coordinate (Column 5 [lines 31-39] “The face detection can be based on other facial points, identifying characteristics, etc. The landmarks can include corners of the mouth, corners of eyes, eyebrow corners, tip of nose, nostrils, chin, tips of ears, distinguishing marks and features, and so on.”; Column 15 [line 32-35] “The learning can include mapping of the x-y coordinates (locations) of the facial landmarks to the coordinates of the bounding box 1030.”; Examiner interprets the y-coordinate of the facial landmarks to be the ordinate of the key points), and the size of the face box (Column 6 [lines 23-34] “The providing of the output of the facial detector can include generating a bounding box 152 for the face. A first bounding box can be generated for a face that is detected in a first frame. The first bounding box can be a square, a rectangle, and/or any other appropriate geometric shape. The first bounding box can be substantially the same as the bounding box generated by a face detector. The first bounding box can be a minimum-dimension bounding box, where the dimension can include area, volume, hyper-volume (i.e. size of face box), and so on. The first bounding box can be generated based on analysis, estimation, simulation, prediction, and so on.”; Examiner interprets dimensions of the bounding box include the area, volume, and hyper-volume which are representative of the size of a face box);
(b) [sending a rotation signal in response to that the judgment value satisfies a rotation condition]; wherein the feature acquisition module comprises a neural network module (Column 10 [lines 25-33] “Classifiers can be binary, multiclass, linear and so on. Algorithms for classification can be implemented using a variety of techniques including neural networks, kernel estimation, support vector machines, use of quadratic surfaces, and so on. Classification can be used in many application areas such as computer vision, speech and handwriting recognition, and so on. Classification can be used for biometric identification of one or more people in one or more frames of one or more videos.”), and the neural network module is configured to receive the image and output the first key point coordinate and the second key point coordinate of the user (Column 7 [lines 34-39] “The flow 100 includes analyzing the face using a plurality of classifiers 175. The face that is analyzed can be the first face, the second face, the third face, and so on. The face can be analyzed to determine facial landmarks, facial features, facial points, and so on. The classifiers can be used to determine facial landmarks”) [and output the size of the face box of the user]; [wherein the neural network module comprises an output feature tensor generation module and a plurality of prediction modules, and the output feature tensor generation module is configured to generate a plurality of output feature tensors having different sizes based on the image; each of the prediction modules is configured to receive a corresponding one of the output feature tensors so as to correspondingly generate an information tensor which corresponds to the corresponding one of the output feature tensors; the information tensor is configured to indicate a location information of the face box, a confidence score information, and a category information as well as a location information of the first key point coordinate and a location information of the second key point coordinate; and the feature acquisition module outputs the first key point coordinate, the second key point coordinate, and the size of the face box of the user based on all of the information tensors generated by the prediction modules.]
However, Senechal fails to teach a judgment value, sending a rotation signal in response to that the judgment value satisfies a rotation condition, output the size of the face box of the user; wherein the neural network module comprises an output feature tensor generation module and a plurality of prediction modules, and the output feature tensor generation module is configured to generate a plurality of output feature tensors having different sizes based on the image; each of the prediction modules is configured to receive a corresponding one of the output feature tensors so as to correspondingly generate an information tensor which corresponds to the corresponding one of the output feature tensors; the information tensor is configured to indicate a location information of the face box, a confidence score information, and a category information as well as a location information of the first key point coordinate and a location information of the second key point coordinate; and the feature acquisition module outputs the first key point coordinate, the second key point coordinate, and the size of the face box of the user based on all of the information tensors generated by the prediction modules
Cheng teaches a judgment value (Page 2206 left column paragraph 3 “we define θ as the angle between device’s x-axis and earth’s horizontal plane, and φ as the angle between device’s y-axis. We experimentally measured the orientation threshold (i.e. judgment value) used by iPhone and iPad, by monitoring the accelerometer readings and rotating the devices as slowly as possible until the screen rotated. We found that the threshold is θ - φ =30, with 2 degrees of dead band, for both iPhone and iPad.”) and sending a rotation signal in response to that the judgment value satisfies a rotation condition (Page 2207 [left column paragraph 4] “Our functional prototype automatically rotates screens to the orientation detected by the face detection API. It counts the number of frames with detected face orientation within a 0.5-second window, and rotates to the most frequently detected orientation. The 0.5-second threshold is the average rotation delay for iPhone and iPad”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Senechal’s reference to include a judgment value and sending a rotation signal in response to that the judgment value satisfies a rotation condition taught by Cheng’s reference. The motivation for doing so would have been to auto rotate a screen of a device using the orientation threshold based on the face orientation as suggested by Cheng’s (see Cheng, Page 2206 left column paragraphs 2 and 3).
However, Senechal and Cheng fail to teach output the size of the face box of the user; wherein the neural network module comprises an output feature tensor generation module and a plurality of prediction modules, and the output feature tensor generation module is configured to generate a plurality of output feature tensors having different sizes based on the image; each of the prediction modules is configured to receive a corresponding one of the output feature tensors so as to correspondingly generate an information tensor which corresponds to the corresponding one of the output feature tensors; the information tensor is configured to indicate a location information of the face box, a confidence score information, and a category information as well as a location information of the first key point coordinate and a location information of the second key point coordinate; and the feature acquisition module outputs the first key point coordinate, the second key point coordinate, and the size of the face box of the user based on all of the information tensors generated by the prediction modules.
Zeng teaches output the size of the face box of the user (Page 141, Subsection 3.2 “We regress relative offsets of bounding boxes instead of absolute coordinates. Offsets are denoted by [ Δl , Δt , Δr , Δb ]:
PNG
media_image5.png
137
492
media_image5.png
Greyscale
(Page 142 first paragraph “[(
x
l
g
,
y
t
g
) , (
x
r
g
y
b
g
)] denote the top left coordinates and bottom right coordinates of the ground truth box respectively.” Examiner interprets equations wg and hg refer to the width and height of the face box);
wherein the neural network module comprises an output feature tensor generation module and a plurality of prediction modules, (Page 139, Subsection 3.1 Paragraph 1 “PPN is a fully-convolutional network (FCN) with 11 branches, consisting of convolutional layers, PReLU [7] activation layers and Softmax normalization layers…Each pixel on this output feature map represents the probability of containing a face within an 8 × 8 detection window on the input image.” and the output feature tensor generation module is configured to generate a plurality of output feature tensors having different sizes based on the image (Figure 1 Captions “Fig. 1. The network structure of PPN . It takes a single image as input and generates multi-scale face proposals simultaneously via multiple branches in a pyramid manner.”); each of the prediction modules is configured to receive a corresponding one of the output feature tensors so as to correspondingly generate an information tensor which corresponds to the corresponding one of the output feature tensors (Page 139 Section 3 paragraph 1 “The overall pipeline is shown in Fig. 2 . The first stage is the Proposal Pyramid Network (PPN) to generate multi-scale face proposals. The second stage named RNet-24 and the third stage named RNet-48 are both dual-task networks, which are used to refine proposals from PPN and predict offsets of corresponding bounding boxes.”); the information tensor is configured to indicate a location information of the face box (Page 139 subsection 3.1 “Each pixel on this output feature map represents the probability of containing a face within an 8 ×8 detection window on the input image. Actually, the process described above is equivalent to sliding a 8 ×8 window on the input image with a stride of 2.”),
Figure 1
PNG
media_image6.png
253
993
media_image6.png
Greyscale
a confidence score information (Page 137, paragraph 2 “Taking a single image with arbitrary size as input, each branch will generate a probability map, in which each element represents the probability that whether a specified size window on the input image contains a face.”), and a category information as well as a location information of the first key point (top left coordinates on Page 141, Subsection 3.2 equates to first key points) coordinate and a location information of the second key point (bottom right coordinates on Page 141, Subsection 3.2 equates to second key points) coordinate (Page 141, Subsection 3.2 “We regress relative offsets of bounding boxes instead of absolute coordinates. Offsets are denoted by [ Δl , Δt , Δr , Δb ]:
PNG
media_image7.png
92
330
media_image7.png
Greyscale
where [(x p l , y p t ) , (x p r , y p b )] denote the top left coordinates (i.e. first key point) and bottom right coordinates (i.e. second key point) of the proposal box respectively, [(x g l , y g t ) , (x g r , y g b )] denote the top left coordinates and bottom right coordinates of the ground truth box respectively.”); and the feature acquisition module outputs the first key point coordinate, the second key point coordinate (Page 141, Subsection 3.2 “We regress relative offsets of bounding boxes instead of absolute coordinates. Offsets are denoted by [ Δl , Δt , Δr , Δb ]:
PNG
media_image7.png
92
330
media_image7.png
Greyscale
where [(x p l , y p t ) , (x p r , y p b )] denote the top left coordinates (i.e. first key point) and bottom right coordinates (i.e. second key point) of the proposal box respectively, [(x g l , y g t ) , (x g r , y g b )] denote the top left coordinates and bottom right coordinates of the ground truth box respectively.), and the size of the face box of the user based on all of the information tensors generated by the prediction modules.
PNG
media_image5.png
137
492
media_image5.png
Greyscale
(Page 142 first paragraph “[(
x
l
g
,
y
t
g
) , (
x
r
g
y
b
g
)] denote the top left coordinates and bottom right coordinates of the ground truth box respectively.” Examiner interprets equations wg and hg refer to the width and height of the face box).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Senechal in view of Cheng to include output the size of the face box of the user; wherein the neural network module comprises an output feature tensor generation module and a plurality of prediction modules, and the output feature tensor generation module is configured to generate a plurality of output feature tensors having different sizes based on the image; each of the prediction modules is configured to receive a corresponding one of the output feature tensors so as to correspondingly generate an information tensor which corresponds to the corresponding one of the output feature tensors; the information tensor is configured to indicate a location information of the face box, a confidence score information, and a category information as well as a location information of the first key point coordinate and a location information of the second key point coordinate; and the feature acquisition module outputs the first key point coordinate, the second key point coordinate, and the size of the face box of the user based on all of the information tensors generated by the prediction modules taught by Zeng’s reference. The motivation for doing so would have been to use a network to generate face candidates extremely fast and reduces the major computational complexity as suggested by Zeng (see Zeng, Abstract).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Cheng and Zeng with Senechal to obtain the invention specified in claim 1.
Regarding claim 8, which claim 1 is incorporated, Senechal fails to teach a display module configured to change an orientation direction of a screen-displayed content in response to that the display module receives the rotation signal.
Cheng teaches a display module configured to change an orientation direction of a screen-displayed content in response to that the display module receives the rotation signal (Page 2207 [left column paragraph 4] “Our functional prototype automatically rotates screens to the orientation detected by the face detection API. It counts the number of frames with detected face orientation within a 0.5-second window, and rotates to the most frequently detected orientation. The 0.5-second threshold is the average rotation delay for iPhone and iPad”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Senechal’s reference to include a display module configured to change an orientation direction of a screen-displayed content in response to that the display module receives the rotation signal taught by Cheng’s reference. The motivation for doing so would have been to automatically rotate a screen of a device to match the face orientation as suggested by Cheng’s (see Cheng, Page 2206 left column paragraphs 2 and 3).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Cheng with Senechal and Zeng to obtain the invention specified in claim 8.
Regarding claim 9, Senechal discloses a judgment method, comprising:
(a) receiving an image by a feature acquisition module and obtaining a first key point (facial landmarks in Column 5 [lines 31-39] equates to key points) coordinate, a second key point (facial landmarks in Column 5 [lines 31-39] equates to key points) coordinate (Column 5 [lines 31-39] “100 includes performing face detection to initialize locations 120 for a first set of facial landmarks (i.e. key points) within a first frame from the video. The face detection can be based on other facial points, identifying characteristics, etc. The landmarks can include corners of the mouth, corners of eyes, eyebrow corners, tip of nose, nostrils, chin, tips of ears, distinguishing marks and features, and so on.”; Column 15 [line 32-35] “The learning can include mapping of the x-y coordinates (locations) of the facial landmarks to the coordinates of the bounding box 1030.”), and a size of a face box (dimensions in Column 6 [lines 23-34] equates to size of face box) of a user by the feature acquisition module based on the image (Column 6 [lines 23-34] “The providing of the output of the facial detector can include generating a bounding box 152 for the face. A first bounding box can be generated for a face that is detected in a first frame. The first bounding box can be a square, a rectangle, and/or any other appropriate geometric shape. The first bounding box can be substantially the same as the bounding box generated by a face detector. The first bounding box can be a minimum-dimension bounding box, where the dimension can include area, volume, hyper-volume (i.e. size of face box), and so on. The first bounding box can be generated based on analysis, estimation, simulation, prediction, and so on.”); and
(b) performing following steps by the judgment module: (b1) obtaining a [judgment value] based on an ordinate of the first key point coordinate, an ordinate of the second key point coordinate (Column 5 [lines 31-39] “The face detection can be based on other facial points, identifying characteristics, etc. The landmarks can include corners of the mouth, corners of eyes, eyebrow corners, tip of nose, nostrils, chin, tips of ears, distinguishing marks and features, and so on.”; Column 15 [line 32-35] “The learning can include mapping of the x-y coordinates (locations) of the facial landmarks to the coordinates of the bounding box 1030.”; Examiner interprets the y-coordinate of the facial landmarks to be the ordinate of the key points), and a size of the face box (Column 6 [lines 23-34] “The providing of the output of the facial detector can include generating a bounding box 152 for the face. A first bounding box can be generated for a face that is detected in a first frame. The first bounding box can be a square, a rectangle, and/or any other appropriate geometric shape. The first bounding box can be substantially the same as the bounding box generated by a face detector. The first bounding box can be a minimum-dimension bounding box, where the dimension can include area, volume, hyper-volume (i.e. size of face box), and so on. The first bounding box can be generated based on analysis, estimation, simulation, prediction, and so on.”; Examiner interprets dimensions of the bounding box include the area, volume, and hyper-volume which are representative of the size of a face box); and
(b2) [sending a rotation signal in response to that the judgment value satisfies a rotation condition]; wherein the feature acquisition module comprises a neural network module (Column 10 [lines 25-33] “Classifiers can be binary, multiclass, linear and so on. Algorithms for classification can be implemented using a variety of techniques including neural networks, kernel estimation, support vector machines, use of quadratic surfaces, and so on. Classification can be used in many application areas such as computer vision, speech and handwriting recognition, and so on. Classification can be used for biometric identification of one or more people in one or more frames of one or more videos.”), and
the step (a) comprises:
(a1) receiving the image and outputting the first key point coordinate and the second key point coordinate of the user [and outputting the size of the face box of the user] by the neural network module (Column 7 [lines 34-39] “The flow 100 includes analyzing the face using a plurality of classifiers 175. The face that is analyzed can be the first face, the second face, the third face, and so on. The face can be analyzed to determine facial landmarks, facial features, facial points, and so on. The classifiers can be used to determine facial landmarks”); [wherein the neural network module comprises an output feature tensor generation module and a plurality of prediction modules, and the step (al) comprises:
(a11) generating a plurality of output feature tensors having different sizes by the output feature tensor generation module based on the image;
(a12) receiving a corresponding one of the output feature tensors by each of the prediction modules so as to correspondingly generate an information tensor which corresponds to the corresponding one of the output feature tensors, wherein the information tensor is configured to indicate a location information of the face box, a confidence score information, and a category information as well as a location information of the first key point coordinate and a location information of the second key point coordinate; and
(a13) outputting the first key point coordinate, the second key point coordinate and the size of the face box of the user by the feature acquisition module based on all of the information tensors generated by the prediction modules].
However, Senechal fails to teach a judgment value, sending a rotation signal in response to that the judgment value satisfies a rotation condition, outputting the size of the face box of the user; wherein the neural network module comprises an output feature tensor generation module and a plurality of prediction modules, and the step (al) comprises:
(a11) generating a plurality of output feature tensors having different sizes by the output feature tensor generation module based on the image;
(a12) receiving a corresponding one of the output feature tensors by each of the prediction modules so as to correspondingly generate an information tensor which corresponds to the corresponding one of the output feature tensors, wherein the information tensor is configured to indicate a location information of the face box, a confidence score information, and a category information as well as a location information of the first key point coordinate and a location information of the second key point coordinate; and
(a13) outputting the first key point coordinate, the second key point coordinate and the size of the face box of the user by the feature acquisition module based on all of the information tensors generated by the prediction modules.
Cheng teaches a judgment value (orientation threshold on Page 2206 left column paragraph 3 equates to judgment value) (Page 2206 left column paragraph 3 “we define θ as the angle between device’s x-axis and earth’s horizontal plane, and φ as the angle between device’s y-axis. We experimentally measured the orientation threshold (i.e. judgment value) used by iPhone and iPad, by monitoring the accelerometer readings and rotating the devices as slowly as possible until the screen rotated. We found that the threshold is θ - φ =30, with 2 degrees of dead band, for both iPhone and iPad.”) and sending a rotation signal in response to that the judgment value satisfies a rotation condition (Page 2207 [left column paragraph 4] “Our functional prototype automatically rotates screens to the orientation detected by the face detection API. It counts the number of frames with detected face orientation within a 0.5-second window, and rotates to the most frequently detected orientation. The 0.5-second threshold is the average rotation delay for iPhone and iPad”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Senechal’s reference to include a judgment value and sending a rotation signal in response to that the judgment value satisfies a rotation condition taught by Cheng’s reference. The motivation for doing so would have been to auto rotate a screen of a device using the orientation threshold based on the face orientation as suggested by Cheng’s (see Cheng, Page 2206 left column paragraphs 2 and 3).
However, Senechal and Cheng both fail to teach outputting the size of the face box of the user; wherein the neural network module comprises an output feature tensor generation module and a plurality of prediction modules, and the step (al) comprises:
(a11) generating a plurality of output feature tensors having different sizes by the output feature tensor generation module based on the image;
(a12) receiving a corresponding one of the output feature tensors by each of the prediction modules so as to correspondingly generate an information tensor which corresponds to the corresponding one of the output feature tensors, wherein the information tensor is configured to indicate a location information of the face box, a confidence score information, and a category information as well as a location information of the first key point coordinate and a location information of the second key point coordinate; and
(a13) outputting the first key point coordinate, the second key point coordinate and the size of the face box of the user by the feature acquisition module based on all of the information tensors generated by the prediction modules.
Zeng teaches outputting the size of the face box of the user (Page 141, Subsection 3.2 “We regress relative offsets of bounding boxes instead of absolute coordinates. Offsets are denoted by [ Δl , Δt , Δr , Δb ]:
PNG
media_image5.png
137
492
media_image5.png
Greyscale
(Page 142 first paragraph “[(
x
l
g
,
y
t
g
) , (
x
r
g
y
b
g
)] denote the top left coordinates and bottom right coordinates of the ground truth box respectively.” Examiner interprets equations wg and hg refer to the width and height of the face box);
wherein the neural network module comprises an output feature tensor generation module and a plurality of prediction modules (Page 139, Subsection 3.1 Paragraph 1 “PPN is a fully-convolutional network (FCN) with 11 branches, consisting of convolutional layers, PReLU [7] activation layers and Softmax normalization layers…Each pixel on this output feature map represents the probability of containing a face within an 8 × 8 detection window on the input image.”), and the step (a1) comprises:
(a11) generating a plurality of output feature tensors having different sizes by the output feature tensor generation module based on the image (Figure 1 Captions “Fig. 1. The network structure of PPN . It takes a single image as input and generates multi-scale face proposals simultaneously via multiple branches in a pyramid manner.”);
(a12) receiving a corresponding one of the output feature tensors by each of the prediction modules so as to correspondingly generate an information tensor which corresponds to the corresponding one of the output feature tensors (Page 139 Section 3 paragraph 1 “The overall pipeline is shown in Fig. 2 . The first stage is the Proposal Pyramid Network (PPN) to generate multi-scale face proposals. The second stage named RNet-24 and the third stage named RNet-48 are both dual-task networks, which are used to refine proposals from PPN and predict offsets of corresponding bounding boxes.”), wherein the information tensor is configured to indicate a location information of the face box (Page 139 subsection 3.1 “Each pixel on this output feature map represents the probability of containing a face within an 8 ×8 detection window on the input image. Actually, the process described above is equivalent to sliding a 8 ×8 window on the input image with a stride of 2.”)
Figure 1
PNG
media_image6.png
253
993
media_image6.png
Greyscale
,
a confidence score information (Page 137, paragraph 2 “Taking a single image with arbitrary size as input, each branch will generate a probability map, in which each element represents the probability that whether a specified size window on the input image contains a face.”), and a category information as well as a location information of the first key point coordinate (top left coordinates on Page 141, Subsection 3.2 equates to first key points) and a location information of the second key point coordinate (bottom right coordinates on Page 141, Subsection 3.2 equates to second key points) (Page 141, Subsection 3.2 “We regress relative offsets of bounding boxes instead of absolute coordinates. Offsets are denoted by [ Δl , Δt , Δr , Δb ]:
PNG
media_image7.png
92
330
media_image7.png
Greyscale
where [(x p l , y p t ) , (x p r , y p b )] denote the top left coordinates (i.e. first key point) and bottom right coordinates (i.e. second key point) of the proposal box respectively, [(x g l , y g t ) , (x g r , y g b )] denote the top left coordinates and bottom right coordinates of the ground truth box respectively.”);
and
(a13) outputting the first key point coordinate, the second key point coordinate and the size of the face box of the user by the feature acquisition module based on all of the information tensors generated by the prediction modules ) (Page 141, Subsection 3.2 “We regress relative offsets of bounding boxes instead of absolute coordinates. Offsets are denoted by [ Δl , Δt , Δr , Δb ]:
PNG
media_image7.png
92
330
media_image7.png
Greyscale
where [(x p l , y p t ) , (x p r , y p b )] denote the top left coordinates (i.e. first key point) and bottom right coordinates (i.e. second key point) of the proposal box respectively, [(x g l , y g t ) , (x g r , y g b )] denote the top left coordinates and bottom right coordinates of the ground truth box respectively.”; Page 142 first paragraph “[(
x
l
g
,
y
t
g
) , (
x
r
g
y
b
g
)] denote the top left coordinates and bottom right coordinates of the ground truth box respectively.” Examiner interprets equations wg and hg refer to the width and height of the face box).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Senechal in view of Cheng to include outputting the size of the face box of the user; wherein the neural network module comprises an output feature tensor generation module and a plurality of prediction modules, and the step (al) comprises:
(a11) generating a plurality of output feature tensors having different sizes by the output feature tensor generation module based on the image;
(a12) receiving a corresponding one of the output feature tensors by each of the prediction modules so as to correspondingly generate an information tensor which corresponds to the corresponding one of the output feature tensors, wherein the information tensor is configured to indicate a location information of the face box, a confidence score information, and a category information as well as a location information of the first key point coordinate and a location information of the second key point coordinate; and
(a13) outputting the first key point coordinate, the second key point coordinate and the size of the face box of the user by the feature acquisition module based on all of the information tensors generated by the prediction modules taught by Zeng’s reference. The motivation for doing so would have been to use a network to generate face candidates extremely fast and reduces the major computational complexity as suggested by Zeng (see Zeng, Abstract).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Cheng and Zeng with Senechal to obtain the invention specified in claim 9.
Regarding claim 16 (drawn to a method), claim 16 is rejected the same as claim 8 and the arguments similar to that presented above for claim 8 are equally applicable to the claim 16, and all the other limitations similar to claim 8 are not repeated herein, but incorporated by reference.
Claims 2 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Senechal et al. (US 10614289 B2) (hereinafter, “Senechal”) in view of Cheng et al. ("iRotate: automatic screen rotation based on face orientation." Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2012.) (hereinafter, “Cheng”); Zeng et al ("Proposal pyramid networks for fast face detection." Information Sciences 495 (2019): 136-149.) (hereinafter, “Zeng”) as applied to claims 1 and 9 above; and further in view of Huang (US 2023/0237694 A1).
Regarding claim 2, which claim 1 is incorporated, Senechal, Cheng, and Zeng fail to teach wherein the first key point coordinate is a coordinate of a right shoulder point of the user, and the second key point coordinate is a coordinate of a left shoulder point of the user.
Huang teaches wherein the first key point coordinate is a coordinate of a right shoulder point of the user, and the second key point coordinate is a coordinate of a left shoulder point of the user (Paragraph [0011] “obtaining left-right shoulder relation information according to bone coordinates at left and right shoulders of the human body”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Senechal in view of Cheng; and further in view of Zeng to include wherein the first key point coordinate is a coordinate of a right shoulder point of the user, and the second key point coordinate is a coordinate of a left shoulder point of the user taught by Huang’s reference. The motivation for doing so would have been to determine the position of a person based on the shoulder information as suggested by Huang (see Huang, Paragraph [0011]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Huang with Senechal, Cheng, and Zeng to obtain the invention specified in claim 2.
Regarding claim 10 (drawn to a method), claim 10 is rejected the same as claim 2 and the arguments similar to that presented above for claim 2 are equally applicable to the claim 10, and all the other limitations similar to claim 2 are not repeated herein, but incorporated by reference.
Claims 3 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Senechal et al. (US 10,614,289 B2) (hereinafter, “Senechal”) in view of Cheng et al. ("iRotate: automatic screen rotation based on face orientation." Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2012.) (hereinafter, “Cheng”); Zeng et al ("Proposal pyramid networks for fast face detection." Information Sciences 495 (2019): 136-149.) (hereinafter, “Zeng”) as applied to claims 1 and 9 above; and further in view of Zutshi (US 10,525,599 B1).
Regarding claim 3, which claim 1 Is incorporated, Senechal discloses a face box (Column 6 [lines 23-34] “The providing of the output of the facial detector can include generating a bounding box 152 for the face…The first bounding box can be generated based on analysis, estimation, simulation, prediction, and so on.”).
However, Senechal, Cheng, and Zeng fail to teach wherein the feature acquisition module is configured to obtain the size of the [face] box based on following steps: subtracting an ordinate of a lower right point coordinate of the face box from an ordinate of an upper left point coordinate of the face box so as to obtain a difference; and setting the size of the [face] box as the difference.
Zutshi teaches wherein the feature acquisition module is configured to obtain the size of the [face] box based on following steps (Column 6 [lines 9-14] “the system may first determine the size thresholds (e.g., minimum width, minimum height, maximum width, and/or maximum height) based on the model number or other identifier associated with the mobile device 104, and apply the size thresholds on the bounding boxes.”):
subtracting an ordinate of a lower right point coordinate of the face box from an ordinate of an upper left point coordinate of the face box so as to obtain a difference; and setting the size of the [face] box as the difference (Column 8 [lines 2-9] “the pixel at the top-right corner of the bounding box may have coordinate values of (900, 1200), such that the bounding box has a width of 600 pixels (e.g., the difference between the x-coordinate values of the two pixels at the bottom-left and top-right corners) and a height of 900 pixels (e.g., the difference between the y-coordinate values of the two pixels).”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Senechal in view of Cheng; and further in view of Zeng to include wherein the feature acquisition module is configured to obtain the size of the [face] box based on following steps: subtracting an ordinate of a lower right point coordinate of the face box from an ordinate of an upper left point coordinate of the face box so as to obtain a difference; and setting the size of the [face] box as the difference taught by Zutshi’s reference. The motivation for doing so would have been to filter out contours that do not reach a predetermined size value as suggested by Zutshi (see Zutshi, Column 5 [lines 47-53]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Zutshi with Senechal, Cheng, and Zeng to obtain the invention specified in claim 3.
Regarding claim 11 (drawn to a method), claim 11 is rejected the same as claim 3 and the arguments similar to that presented above for claim 3 are equally applicable to the claim 11, and all the other limitations similar to claim 3 are not repeated herein, but incorporated by reference.
Allowable Subject Matter
Claims 4, 5, 12, and 13 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claims 4 and 5 contain subject matter that is not disclosed or made obvious in the cited art:
In regard to claim 4, when considering claim 4, the feature highlighted below are considered an improvement over the prior art and have not been found to be anticipated or rendered obvious by a combination of the prior art:
“wherein the step (a) comprises:
calculating an absolute value of a difference between the ordinate of the second key point coordinate and the ordinate of the first key point coordinate; and
setting the judgment value as a ratio of the absolute value of the difference over the size of the face box.”
In regard to claim 5, claim 5 depends on objected claim 4. Therefore, by virtue of its dependency, claim 5 also indicated as objected subject matter.
In regard to claim 12, when considering claim 12, the feature highlighted below are considered an improvement over the prior art and have not been found to be anticipated or rendered obvious by a combination of the prior art:
“wherein the step (b1) comprises:
calculating an absolute value of a difference between the ordinate of the second key point coordinate and the ordinate of the first key point coordinate; and
setting the judgment value as a ratio of the absolute value of the difference over the size of the face box.”
In regard to claim 13, claim 13 depends on objected claim 12. Therefore, by virtue of its dependency, claim 13 also indicated as objected subject matter.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Wang (US 2022/0270219 A1) discloses a method for extracting human face attributes of standard regions of respective faces in an image.
Xu et al. (US 2020/0050835 A1) discloses a method for determining face image quality by obtaining pose angle information and size information of a face in an image.
Senechal et al. (US 2016/0004904 A1) discloses detecting and tracking faces in a series of video frames by using classifiers to initialize and refine facial landmark locations.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to UROOJ FATIMA whose telephone number is (571)272-2096. The examiner can normally be reached M-F 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/UROOJ FATIMA/Examiner, Art Unit 2676
/Henok Shiferaw/Supervisory Patent Examiner, Art Unit 2676
Prosecution Insights