Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
Applicant’s election without traverse of claims 6-8, 10-12, 25-27, and 29-31 in the reply filed on 04/10/2026 is acknowledged. Claims 1, 23 and 39-40 are generic.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 02/05/2024 and 01/03/2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Claim Objections
Claims 8, 10-12 are objected to because of the following informalities.
Claim 8 recites, recalling the claim language of claim 1, “a face feature set acquired from the images comprising the face“ in line 3. Claim 1 does not recite “a face feature set.” Instead, it recites “a human eye feature set.”
For the prior art purposes, the examiner has interpreted claim 8 to recite “a human eye feature set.”
Claim 10, which depends on claim 6, recites “wherein obtaining the geometric position relationships based on the internal and external parameters of the plurality of cameras and by utilizing a plane mirror and an auxiliary camera to transmit calibration information for calibration comprises:“ in lines 1-4. Instead of claim 6, the quoted language is recited in claim 7. Appropriate correction is required.
For the prior art purposes, the examiner has interpreted claim 10 to depend on claim 7.
Claim 11, which depends on claim 6, recites “wherein obtaining the geometric position relationships based on the internal and external parameters of the plurality of cameras and by utilizing an auxiliary camera to transmit calibration information for calibration comprises:“ in lines 1-4. Instead of claim 6, the quoted language is recited in claim 7. Appropriate correction is required
For the prior art purposes, the examiner has interpreted claim 11 to depend on claim 7.
Claim 12, which depends on claim 6, recites “obtaining the geometric position relationships based on the internal and external parameters of the plurality of cameras and by utilizing a plane mirror to transmit calibration information for calibration comprises:“ in lines 1-4. Instead of claim 6, the quoted language is recited in claim 7. Appropriate correction is required.
For the prior art purposes, the examiner has interpreted claim 12 to depend on claim 7.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
Acquisition module in claim 23;
Key point determination module in claim 23;
Line-of-sight reconstruction module in claim 23;
Point-of-sight determination module in claim 23;
Calibration unit in claim 26;
First determination unit in claim 27;
Second determination unit in claim 27;
Third determination unit in claim 27;
First calibration unit in claim 29;
First calibration subunit in claim 29;
Second calibration subunit in claim 29;
Third calibration subunit in claim 29;
Fourth calibration subunit in claim 29;
Second calibration unit in claim 29;
Fifth calibration subunit in claim 29;
Sixth calibration subunit in claim 29;
Seventh calibration subunit in claim 29;
Eighth calibration subunit in claim 29;
Third calibration unit in claim 29;
Ninth calibration subunit in claim 29;
Tenth calibration subunit in claim 29;
Eleventh calibration subunit in claim 29;
Twelfth calibration subunit in claim 29;
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 39 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because it is directed to products that do not have a physical or tangible form, such as information (often referred to as "data per se") or a computer program per se (often referred to as "software per se") when claimed as a product without any structural recitations.
The examiner suggests amending the claim to be directed to “A non-transitory computer readable medium storing a program.”
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 6, 23, 25, 39, and 40 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Us patent application publication no. 2019/0172222 to Ebisawa.
For claim 1, Ebisawa as applied discloses a line-of-sight direction tracking method (see, e.g., FIG. 10), comprising:
providing corneal reflection for an eye of a user by using a plurality of light sources, and capturing images comprising a face of the user by using a plurality of cameras (see, e.g., pars. 80-84 and 107-112 and FIG. 10, which teach obtaining corneal reflection from images acquired using light sources and cameras);
determining coordinates of a light source reflection point and coordinates of a pupil center in a world coordinate system by means of a human eye feature set acquired from the images comprising the face, based on hardware calibration parameters (see, e.g., pars. 110-118 and FIGS. 1-2 and 9-10, which teach determining coordinates of a corneal reflection point and coordinates of a pupil center in world coordinate system based on human eye features in the images and the hardware calibration data);
determining a line-of-sight optical axis according to the coordinates of the light source reflection point and the coordinates of the pupil center (see, e.g., pars. 120-130 and FIGS. 1-2 and 9-10 , which teach obtaining an optical axis using the cornea reflection-pupil center vector, which is based on the coordinates of the cornea reflection and pupil center) and reconstructing, based on the line-of-sight optical axis, a line-of-sight visual axis by means of a compensation angle (see, e.g., pars. 131-138 and FIGS. 1-2 and 9-10, which teach acquiring the visual axis by calibrating the optical axis by means of a deviation angle); and
determining a point of sight on a target object according to the line-of-sight visual axis and a position of the target object in the world coordinate system (see, e.g., pars. 52-61 and 135-138 and FIGS. 1-2, and 9-10, which teach determining a gazing point according to the visual axis and a position of the visual target with known coordinates).
For claims 23, 39 and 40, Ebisawa as applied discloses a line-of-sight direction tracking apparatus (see, e.g., FIGS. 8 and 9), comprising:
an acquisition module configured to provide corneal reflection for an eye of a user by using a plurality of light sources, and capture images comprising a face of the user by using a plurality of cameras(see, e.g., pars. 80-84 and 107-112 and FIG. 10, which teach obtaining corneal reflection from images acquired using light sources and cameras);
a key point determination module configured to determine coordinates of a light source reflection point and coordinates of a pupil center in a world coordinate system by means of a human eye feature set acquired from the images comprising the face, based on hardware calibration parameters (see, e.g., pars. 110-118 and FIGS. 1-2 and 9-10, which teach determining coordinates of a corneal reflection point and coordinates of a pupil center in world coordinate system based on human eye features in the images and the hardware calibration data);
a line-of-sight reconstruction module configured to determine a line-of-sight optical axis according to the coordinates of the light source reflection point and the coordinates of the pupil center (see, e.g., pars. 120-130 and FIGS. 1-2 and 9-10 , which teach obtaining an optical axis using the cornea reflection-pupil center vector, which is based on the coordinates of the cornea reflection and pupil center), and reconstruct, based on the line-of-sight optical axis, a line- of-sight visual axis by means of a compensation angle (see, e.g., pars. 131-138 and FIGS. 1-2, and 9-10, which teach acquiring the visual axis by calibrating the optical axis by means of a deviation angle); and
a point-of-sight determination module configured to determine a point of sight on a target object according to the line-of-sight visual axis and a position of the target object in the world coordinate system (see, e.g., pars. 52-61 and 135-138 and FIGS. 1-2 and 9-10, which teach determining a gazing point according to the visual axis and a position of the visual target with known coordinates).
For claims 6 and 25, Ebisawa as applied discloses that the hardware calibration parameters comprise:
internal and external parameters of the plurality of camera coordinate systems and geometric position relationships (see, e.g., pars. 91, 112 and 116, which teach obtaining positions in the space using internal and external camera parameters and relational equations between the coordinate systems), wherein the geometric position relationships comprise a first position relationship between the plurality of light source coordinate systems and the plurality of camera coordinate systems, a second position relationship between the plurality of light source coordinate systems and the target object coordinate system, and a third position relationship between the plurality of camera coordinate systems and the target object coordinate system (see, e.g., pars. 91, 112, 116, and 183-190 and FIGS. 18 and 20, which teach determining the geometric position relationships among the world coordinate system, the camera coordinate system and the image coordinate system).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 7 and 26 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ebisawa in view of cn patent application publication no. 110827360 to Li et al. (hereinafter Li).
For claims 7 and 26, Ebisawa as applied teaches that the geometric position relationships are obtained based on the internal and external parameters of the plurality of cameras (see, e.g., pars. 91 and 116, which teach obtaining positions in the space using internal and external camera parameters and relational equations between the coordinate systems).
While Ebisawa does not explicitly teach, Li in the analogous art teaches by utilizing at least one of a plane mirror and an auxiliary camera to transmit calibration information for calibration (see, e.g., pars. 7-12 of Li, which teach calibrating a light source using a plane mirror and a camera).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Ebisawa to calibrate as taught by Li because doing so would improve the calibration accuracy of the light source direction, providing a high-precision calibration method (see pars. 4 and 33 of Li).
Allowable Subject Matter
Claims 8, 10-12, 27, and 29-31 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
In regard to claims 8 and 27, when considered as a whole, prior art of record fails to disclose or render obvious, alone or in combination:
“The line-of-sight direction tracking method according to claim 6, …
when the world coordinate system is the light source coordinate system, determining, based on the face feature set, the coordinates of the light source reflection point and the coordinates of the pupil center in the light source coordinate system through the internal and external parameters of the plurality of cameras and based on the first position relationship, or based on the second position relationship and the third position relationship;
when the world coordinate system is the camera coordinate system, determining, based on the face feature set, the coordinates of the light source reflection point and the coordinates of the pupil center in the plurality of camera coordinate systems through the internal and external parameters of the plurality of cameras; and
when the world coordinate system is the target coordinate system, determining, based on the face feature set, the coordinates of the light source reflection point and the coordinates of the pupil center in the target object coordinate system through the internal and external parameters of the plurality of cameras and based on the third position relationship, or based on the first position relationship with the second position relationship.”
The examiner notes that although claim 8 recites a condition with multiple possible outcomes, additional limitations such as a subsequent step of determining a point of sight are present that tie the claimed condition and the multiple possible outcomes together into a single process, making it essential that the multiple possible outcomes all be performed, even if at different times (HYTERA COMMUNICATIONS CO. LTD. v. MOTOROLA SOLUTIONS, INC. , No. 19-2124 (Fed. Cir. 2021). Thus, the claim is integrated into one method and does not diverge into two distinct methods as in Ex parte Schulhauser, Appeal 2013-007847, 2016 WL 6277792 (PTAB Apr. 28, 2016) (precedential). The examiner concludes that the broadest reasonable interpretation of the claim requires the performance of all of the claimed steps, including all of the multiple possible outcomes of the condition.
In regard to claims 10 and 29, when considered as a whole, prior art of record fails to disclose or render obvious, alone or in combination:
“The line-of-sight direction tracking method according to claim 6, …
acquiring, by a first auxiliary camera, a plurality of first marker images of the target object containing first markers reflected by the plane mirror in a plurality of different poses;
calculating the third position relationship based on an orthogonal constraint according to the plurality of first marker images and based on the internal and external parameters of the plurality of cameras;
acquiring, by a second auxiliary camera, second marker images containing the plurality of light sources, and acquiring the first position relationship based on the internal and external parameters of the plurality of cameras and based on the second marker images, wherein the second auxiliary camera is a stereo vision system; and
determining the second position relationship according to the first position relationship and the third position relationship.”
In regard to claims 11 and 30, when considered as a whole, prior art of record fails to disclose or render obvious, alone or in combination:
“The line-of-sight direction tracking method according to claim 6, …
acquiring, by a third auxiliary camera, third marker images containing the plurality of light sources, and acquiring the first position relationship based on the internal and external parameters of the plurality of cameras and based on the third marker images, wherein the third auxiliary camera is a stereo vision system;
configuring a fourth auxiliary camera to have a field of view comprising the plurality of cameras and the third auxiliary camera, disposing a calibration plate beside the third auxiliary camera, acquiring, by the plurality of cameras, fourth marker images containing a region of the calibration plate, and acquiring, by the third auxiliary camera, fifth marker images of the target object containing fifth markers;
by taking a position relationship between the fourth auxiliary camera and the plurality of cameras as a pose conversion bridge, determining the third position relationship according to the fourth marker images and the fifth marker images and based on internal and external parameters of the third auxiliary camera and the internal and external parameters of the plurality of cameras; and
determining the second position relationship according to the first position relationship and the third position relationship.”
In regard to claims 12 and 31, when considered as a whole, prior art of record fails to disclose or render obvious, alone or in combination:
“The line-of-sight direction tracking method according to claim 6, …
with the plane mirror having no less than 4 markers pasted thereon as an aid, acquiring, by the plurality of cameras, reflected images containing the plurality of light sources, the target object, and the markers;
calculating marker coordinates of the respective markers, mirrored light source coordinates of the plurality of light sources, and mirrored target object coordinates of the target object in the plurality of camera coordinate systems according to the reflected images;
reconstructing a mirror plane according to all the marker coordinates, and determining the first position relationship and the third position relationship based on the mirrored light source coordinates and the mirrored target object coordinates according to a principle of specular reflection; and
determining the second position relationship according to the first position relationship and the third position relationship.”
Additional Citations
The following table lists several references that are relevant to the subject matter claimed and disclosed in this Application. The references are not relied on by the Examiner, but are provided to assist the Applicant in responding to this Office action.
Citation
Relevance
Ebisawa et al. (us pat. pub. 2007/0279590)
Describes a method and device for detecting a line-of-sight of a subject and a three-dimensional view-point measurement device. In one embodiment, the method for detecting a line-of-sight uses a first camera 10, 11 that measure the position of a pupil relative to a coordinate system, a second camera 12 having a light source arranged at a known position in the coordinate system and forming a corneal reflection center to obtain data of a size of vector r from the corneal reflection center to the pupil center and an angle .phi. of the vector r relative to a coordinate axis of the coordinate system, and a calculation means for calculating a line-of-sight direction based on information from each of the cameras. In a stage of determining a relational formula, a subject is made to gaze at a known point to perform measurement and a relational formula is determined. In a stage of determining a line-of-sight, the subject is measured again to determine the line-of-sight using the relational formula. Further, a three-dimensional view-point measurement device can be configured so as to simultaneously measure the line-of-sights of both eyes with two cameras and two light sources.
Kojima et al. (us pat. pub. 2020/0311870)
Describes a gaze-tracking device, program, and method, and particularly relates to a gaze-tracking device, computer readable medium, and method for measuring a gaze vector in an image captured of a face. Ine one embodiment, a corneal-reflection-based gaze detection section calculates a time series of a three-dimensional gaze vector in a camera coordinate system from a time series of facial images. A face position-and-orientation estimation section estimates a time series of a three-dimensional position and orientation of a face. An eyeball-center-coordinates transformation section calculates a time series of a three-dimensional position of the eyeball center in a coordinate system of a three-dimensional facial model. A fixed parameter calculation section calculates for use as a fixed parameter a three-dimensional position of the eyeball center in the three-dimensional facial-model coordinate system. An eyeball-center-based gaze detection section uses the three-dimensional position of the eyeball center calculated by the fixed parameter calculation section to calculate a three-dimensional gaze vector from a three-dimensional position of the eyeball center to a three-dimensional position of a pupil center in the camera coordinate system. This enables accurate gaze tracking to be performed using a simple configuration and without performing calibration.
Lin et al. (us pat. 11017557)
Describes an apparatus and a method for compressing an image using lower resolution imaging and restoring a compressed image using super resolution imaging. For example, the method includes receiving monochrome image data and low resolution color image data generated from an original color image, decoding the monochrome image data and generating a low resolution monochrome image. The method further includes decoding the low resolution color image data to generate a low resolution color image, processing the low resolution monochrome image and generating a high resolution monochrome image in accordance with a super resolution imaging neural network. The method lastly includes generating a high resolution color image based on the low resolution color image and the high resolution monochrome image in accordance with a colorization imaging neural network.
Table 1
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. See form 892 and Table 1.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WOO RHIM whose telephone number is (571)272-6560. The examiner can normally be reached Mon - Fri 9:30 am - 6:00 pm et.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at 571-272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WOO C RHIM/Examiner, Art Unit 2676
Prosecution Insights