Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
Applicant’s election without traverse of Species I and Group I.B in the reply filed on 03/25/26 is acknowledged. Thus, application has withdrawn claims 2-3,13-14 and 18. Claims 1, 4-12, 15-17 and 19-20 are examined below.
Claim Objections
Claim 6 objected to because of the following informalities: The second row, third column comprises the rotational element
R
21
. Examiner believes that the correct element label should be
R
12
as similarly disclosed in Equation (12) of the Specification. Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4-6, 11, 17 and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over KANG CHEN CN-111768477-A, hereinafter CHEN.
As per claim 1, CHEN discloses a method of processing an image, comprising:constructing an initial three-dimensional face template by using a plurality of sample face images (see CHEN page 3/31, wherein a three-dimensional face expression base, or template, is generated corresponding to the target face. The target face is acquired from input data, which includes a plurality of face images);performing an iterative optimization on the initial three-dimensional face template by using a face image of a target object, so as to obtain a target three-dimensional face template (see CHEN page 7/31, wherein the target keypoints of a scanned face are iteratively optimized, which is then used to obtain the target three-dimensional face model); anddetermining a current face pose of the target object according to a corresponding relationship between a current face image of the target object and the target three-dimensional face template (see CHEN step S130 at the top of page 8/31, wherein a three-dimensional facial expression is obtained using the reference expression change according to a parameter relationship between the reference face model and the target face model).
While CHEN does disclose the input data including a plurality of face images, it does not explicitly disclose the three-dimensional face template is constructed based off the input data. However, it would have been obvious for one of ordinary skill in the art to use CHEN to disclose the three-dimensional face template using the input data. The reason is as disclosed in CHEN page 3/31, wherein the reference three-dimensional face model corresponds to the target key points. These key points are extracted from the target face image, which is the input data. Thus, it would have been obvious to one of ordinary skill in the art at the time of the invention by the applicant to use the input data of CHEN in order to generate, or construct, the three-dimensional face template.
As per claim 4, CHEN discloses the method of processing the image according to claim 1, wherein the performing an iterative optimization on the initial three-dimensional face template by using a face image of a target object so as to obtain a target three-dimensional face template (see CHEN page 7/31, wherein the target keypoints of a scanned face are iteratively optimized, which is then used to obtain the target three-dimensional face model) comprises:acquiring a plurality of two-dimensional target key points from the face image of the target object (see CHEN page 5/31, wherein the target keypoints from the input face image, i.e., two-dimensional face image, is extracted);determining a plurality of three-dimensional key points from the initial three-dimensional face template (see CHEN page 5/31, wherein the initial key points, which are in reference to the three-dimensional face model, are extracted and calibrated, as in determined);projecting the plurality of three-dimensional key points into a plurality of two-dimensional projection key points (see CHEN page 5/31, wherein the three-dimensional key points are projected onto a two-dimensional image);calculating an average error between the plurality of two-dimensional projection key points and the plurality of two-dimensional target key points (see CHEN page 5/31, wherein a camera projection matrix is estimated to minimize the reprojection error from 3D to 2D. See further CHEN page 7/31, wherein a plurality of reprojection errors of the plurality of frames are constrained, as in averaged, together as
m
i
n
∑
j
w
j
∑
j
M
j
P
S
i
-
P
i
'
, i.e., average error); and performing the iterative optimization on the initial three-dimensional face template according to the average error, so as to obtain the target three-dimensional face template (see CHEN page 7/31, wherein the target a matrix between the initial and target key points is estimated using the parameter relationship that minimizes the projection error. This is step is performed multiple times, i.e., iterative optimization).
While CHEN does disclose a constrained reprojection error, it does not explicitly disclose an average error. However, it would have been obvious for one of ordinary skill in the art to use the plurality of constrained reprojection errors as the average error. The reason is because the average error is the projection error calculated between the plurality of projection key points, as disclosed in the Specification ¶96-98, filed 9/28/2023. The reprojection error also uses the initial target key points as well as the projection key points. Thus, it would have been obvious to one of ordinary skill in the art at the time of the invention by the applicant to use the constrained reprojection errors as the average error.
As per claim 5, CHEN discloses the method of processing the image according to claim 4, wherein the projecting the plurality of three-dimensional key points into a plurality of two-dimensional projection key points (see CHEN page 5/31, wherein the three-dimensional key points are projected onto a two-dimensional image) comprises:constructing a weak perspective projection model according to coordinate values of the three-dimensional key points, a scaling ratio, a coordinate system rotation matrix, and an offset vector of a pixel coordinate system (see CHEN page 5/31, wherein a camera projection matrix, which serves as the weak perspective projection model, uses the three-dimensional points, a scale factor
s
, a rotation matrix
R
, and a translation vector
T
, i.e., the offset vector, of the pixel coordinate system
P
comprising
[
x
,
y
,
z
,
1
]
); andprojecting the plurality of three-dimensional key points into a plurality of two-dimensional projection key points by using the weak perspective projection model (see CHEN page 5/31, wherein the camera projection matrix is a matrix that projects a three-dimensional space into a two-dimensional image space using homogeneous coordinate points).
While CHEN does not explicitly disclose a center point offset vector, it would have been obvious for one of ordinary skill in the art to understand the translation vector to be the center point offset. As shown in CHEN FIG. 2 and page 5/31, the camera coordinate system
[
i
w
,
j
w
,
k
w
]
with the center point
O
w
is projected onto the 2D image using
R
,
T
. Therefore, it would have been obvious for one of ordinary skill in the art to use the camera’s coordinate system, in combination with the translation matrix of CHEN, as the center point offset when calculating the projection onto the 2D space.
As per claim 6, CHEN discloses The method of processing the image according to claim 5, wherein the weak perspective projection model is configured to project the plurality of three-dimensional key points into a plurality of two-dimensional projection key points according to:
x
y
=
s
c
a
l
e
∙
R
00
R
01
R
02
R
10
R
11
R
21
∙
X
Y
Z
+
t
x
t
y
(see CHEN page 5/31, wherein the formula
P
’
=
s
K
[
R
P
+
T
]
is disclosed)
where
x
and
y
respectively represent coordinate values of the two-dimensional projection key points on x-axis and y-axis of the pixel coordinate system (see CHEN page 5/31, wherein
P
’
=
[
u
,
v
]
, two-dimensional projection points),
X
,
Y
and
Z
respectively represent coordinate values of the three-dimensional key points on x-axis, y-axis and z-axis of a coordinate system where the target object is located (see CHEN page 5/31, wherein
P
=
x
,
y
,
z
in three-dimensional space, i.e., the X, Y, Z coordinates, of the face, i.e., target objection),
s
c
a
l
e
represents the scaling ratio (see CHEN page 5/31, wherein s is the scale factor),
R
00
R
01
R
02
R
10
R
11
R
21
represents a rotation matrix of the coordinate system where the target object is located with respect to a camera coordinate system (see CHEN page 5/31, wherein R is the rotation matrix), and
t
x
and
t
y
respectively represent offset vectors of an origin of the pixel coordinate system with respect to an origin of the camera coordinate system on x-axis and y-axis (see CHEN page 5/31, where T represents the translation vector, i.e., offset vector, which corresponds to the point in the three-dimensional space represented by a homogeneous coordinate).
While CHEN does disclose a different perspective projection equation, it would have been obvious for one to use a similar equation. The reason being is that the equation used by CHEN contains the same coordinate points, scaling ratio, rotation matrix, and offset vectors. Therefore, it would have been obvious to one of ordinary skill in the art to modify CHEN’s perspective matrix in order to achieve the two-dimensional projection key points because CHEN’s equation uses the scale, rotation matrix, and offset vectors to also acquire two-dimensional points.
As per claim 11, CHEN discloses the method of processing the image according to claim 4, wherein the calculating an average error between the plurality of two-dimensional projection key points and the plurality of two-dimensional target key points comprises (see CHEN page 5/31, wherein a camera projection matrix is estimated to minimize the reprojection error from 3D to 2D. It is further disclosed in page 7/31 that the reprojection error is constrained across a plurality of frames, i.e., average error):calculating a re-projection error according to the plurality of two-dimensional projection key points and the plurality of two-dimensional target key points (see CHEN page 5/31, wherein the minimized reprojection error,
m
i
n
∑
i
M
P
i
-
P
i
, is calculated according to the target key points, i.e., projection key points, and initial key points); andcalculating the average error according to the re-projection error (see CHEN page 7/31, wherein a plurality of reprojection errors of the plurality of frames are constrained, as in averaged, together as
m
i
n
∑
j
w
j
∑
j
M
j
P
S
i
-
P
i
'
, i.e., average error).
As per claim 17, the rationale provided in claim 1 is incorporated herein. In addition, the apparatus of claim 17 corresponds to the method of claim 1.
As per claim 19, CHEN discloses an electronic device, comprising:one or more processors (see CHEN page 3/31, wherein a processor is disclosed); anda memory configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, are configured to cause the one or more processors to implement the method of claim 1 (see CHEN page 11/31, wherein a memory that stores programs is disclosed).
As per claim 20, CHEN discloses a computer-readable storage medium having executable instructions therein, wherein the instructions, when executed by a processor, are configured to cause the processor to implement the method of claim 1 (see CHEN page 11/31, wherein a computer-readable storage medium configured to execute program code is disclosed).
Claims 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over CHEN, in further view of Yan-zhi WEI CN-115049738-A, hereinafter WEI.
As per claim 15, CHEN fails to explicitly disclose where WEI teaches:The method of processing the image according to claim 4, wherein the acquiring a plurality of two-dimensional target key points from the face image of the target object comprises:performing a distortion correction on the face image to obtain a corrected face image (see WEI bottom of page 13/41 and more specifically page 20/41, wherein a de-distortion process is formed on the camera image in order to obtain the face of the person); anddetermining the plurality of two-dimensional target key points from the corrected face image by using a key point detection algorithm (see WEI page 20/41, wherein the two-dimensional points of the pupils, which serves as the key points, are obtained by the camera projection relationship model formula, i.e., key point detection algorithm).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify CHEN’s method by using WEI’s teaching by including distortion correction to the face image in order to more clearly acquire the target key points from a clearer image.
As per claim 16, CHEN, in combination with WEI, discloses The method of processing the image according to claim 15, wherein the performing a distortion correction on the face image to obtain a corrected face image (see WEI bottom of page 13/41 and more specifically page 20/41, wherein a de-distortion process is formed on the camera image in order to obtain the face of the person) comprises:performing the distortion correction on the face image according to:
x
0
=
x
(
1
+
k
1
r
2
+
k
2
r
4
+
k
3
r
6
)
y
0
=
y
(
1
+
k
1
r
2
+
k
2
r
4
+
k
3
r
6
)
x
0
=
x
+
[
2
p
1
x
y
+
p
2
(
r
2
+
2
x
2
)
]
x
0
=
x
+
[
2
p
1
x
y
+
p
2
(
r
2
+
2
x
2
)
]
(see WEI top of page 19/41, for the following equation)where
x
0
and
y
0
respectively represent coordinate values of a coordinate point on the face image on x-axis and y-axis (see WEI page 15/41, wherein the point
[
x
,
y
]
is a two-dimensional point on the camera coordinate system),
x
and
y
respectively represent coordinate values of a coordinate point on the corrected face image on x-axis and y-axis (see WEI page 19/41, wherein
[
u
d
i
s
t
o
r
t
e
d
,
v
d
i
s
t
o
r
t
e
d
]
is the corrected facial pixel point),
r
represents a distance between a center point of the face image and the coordinate point
(
x
,
y
)
(see WEI page 19/41, wherein
r
represents a distance between a center point of the face image and the coordinate point
(
x
,
y
)
),
k
i
,
k
2
and
k
3
are radial distortion coefficients (see WEI page 15/41, wherein
k
i
,
k
2
and
k
3
are radial distortion coefficients), and
p
1
and
p
2
are tangential distortion coefficients (see WEI page 15/41, wherein
p
1
and
p
2
are tangential distortion coefficients).
Allowable Subject Matter
Claims 7-9 and 12 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Bradley Obas Felix whose telephone number is (703)756-1314. The examiner can normally be reached M-F 8-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent Rudolph can be reached at 5712728243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BRADLEY O FELIX/Examiner, Art Unit 2671
/VINCENT RUDOLPH/Supervisory Patent Examiner, Art Unit 2671
Prosecution Insights