DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-11 are pending.
Priority
This application is a continuation of International Application No. PCT/KR2022/015073 filed on October 7, 2022, which claims priority to Korean Patent Application No. 10-2021-0133021 filed on October 7, 2021.
Information Disclosure Statement
The IDS filed 04/05/24 is considered.
Claim Interpretation
Claim 1 recites the language “on the basis of an unspecified number of times”. This language is non-descriptive, but upon further looking into the specification, applicant defines the term (Page 16, lines 7-13: “To measure the information regarding the number of eye blinks of the measurement target, the parameter calculator 105 may measure the number of eye blinks by counting frequencies at which the pupil size is 0 in frames in which the pupil size is not 0. Here, the pupil size being 0 indicates that the measurement target temporarily closes eyes and thus the pupil is not confirmed by an infrared camera that captures an eye image, and the number of times the pupil size is 0 may be referred to as an unspecified number of times of the pupil size”). Due to this description the examiner will interpret the language as “based on when the eye closes/when the pupil size is zero”.
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are:
"an image receiver configured to" in claim 7 (and present by dependency in claims 8-11)
"a pupil size determiner configured to" in claim 7 (and present by dependency in claims 8-11)
"a parameter calculator configured to" in claim 7 (and present by dependency in claims 8-11)
"a fatigue occurrence duration determiner configured to" in claim 7 (and present by dependency in claims 8-11)
Because these claim limitations are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have these limitations interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitations to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitations recite sufficient structure to perform the claimed function so as to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
35 U.S.C. 101 requires that a claimed invention must fall within one of the four eligible categories of invention (i.e. process, machine, manufacture, or composition of matter) and must not be directed to subject matter encompassing a judicially recognized exception as interpreted by the courts. MPEP 2106. Three categories of subject matter are found to be judicially recognized exceptions to 35 U.S.C. § 101 (i.e. patent ineligible) (1) laws of nature, (2) physical phenomena, and (3) abstract ideas. MPEP 2106(II). To be patent-eligible, a claim directed to a judicial exception must as whole be integrated into a practical application or directed to significantly more than the exception itself (MPEP 2106). Hence, the claim must describe a process or product that applies the exception in a meaningful way, such that it is more than a drafting effort designed to monopolize the exception.
Claims 1-6 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., an abstract idea) without integration into a practical application or recitation of significantly more. In the analysis below, the method of claim 1 will be analyzed. Furthermore, claim 1 is directed to one of the four statutory categories of eligible subject matter; thus, the claims pass Step 1 of the Subject Matter Eligibility Test (See flowchart in MPEP 2106).
Step 2A, Prong 1 Analysis
The independent claims are directed determining a size of a pupil by analyzing the eye image; calculating each of a number of eye blinks, an eye closed time, and a pupil size change speed on the basis of an unspecified number of times and a change in the size of the pupil, in some frames where the size of the pupil in the eye image is determined; and determining a duration in which visual fatigue of the measurement target occurs, from among the plurality of frames, as a result of combining information regarding the calculated number of eye blinks, eye closed time, and pupil size change speed.
Each of the above steps can be performed mentally. An individual can mentally determining a size of a pupil by analyzing the eye image with the assistance of measurement tools. An individual can mentally calculate each of a number of eye blinks, an eye closed time, and a pupil size change speed on the basis of an unspecified number of times and a change in the size of the pupil, in some frames where the size of the pupil in the eye image is determined by keeping track of the blinks, timing the eye closed time, and tracking the size changes of the eyes between frames using measurement tools. An individual can mentally determine a duration in which visual fatigue of the measurement target occurs, from among the plurality of frames, as a result of combining information regarding the calculated number of eye blinks, eye closed time, and pupil size change speed. As such, the description in independent claim 1 is an abstract idea – namely, a mental process. Accordingly, the analysis under prong one of step 2A of the Subject Matter Eligibility Test does not result in a conclusion of eligibility (See flowchart in MPEP 2106).
Additional elements
The independent claims further recites receiving an eye image that is an image comprising a plurality of frames and obtained by photographing an eye of a measurement target.
Step 2A, prong 2 analysis
The above-identified additional elements do not integrate the judicial exception into a practical application.
The additional elements of receiving an eye image that is an image comprising a plurality of frames and obtained by photographing an eye of a measurement target can be defined as data gathering. Such data gathering steps amount to insignificant pre-solution activity which does not integrate the abstract idea into a practical application (MPEP 2106.05(g)).
Moreover, the additional elements of the claims do not recite an improvement in the functioning of a computer or other technology or technical field, the claimed steps are not performed using a particular machine, the claimed steps do not effect a transformation, and the claims do not apply the judicial exception in any meaningful way beyond generically linking the use of the judicial exception to a particular technological environment (See MPEP 2106.04(d)). Therefore, the analysis under prong two of step 2A of the Subject Matter Eligibility Test does not result in a conclusion of eligibility (See flowchart in MPEP 2106).
Step 2B
The additional elements of receiving an eye image that is an image comprising a plurality of frames and obtained by photographing an eye of a measurement target does not amount to more than data gathering. Thus, the additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea).
Accordingly, the analysis under step 2B of the Subject Matter Eligibility Test does not result in a conclusion of eligibility (See flowchart in MPEP 2106).
For all of the foregoing reasons, independent claim 1 does not recite eligible subject matter under 35 USC 101.
Dependent claims 2-6 are dependent on independent claim 1 and therefore include all of the limitations of claim 1. Therefore, claims 2-6 recite the same abstract idea of a mental process which can be performed in the mind.
Claim 2 recites determining of the duration in which the visual fatigue occurs comprises combining a tendency to gradually increase the calculated number of eye blinks and a tendency to gradually increase the eye closed time, comparing a result of the combination with a preset condition, and determining, from among the plurality of frames, the duration in which the visual fatigue occurs. An individual can mentally determine the duration of fatigue by mentally analyzing a gradual increase of the number of eye blinks and a gradual increase of the eye closed time. Thus, the feature of claim 2 is directed to the mental process. Accordingly, the claim does not recite any additional limitations that integrate the judicial exception into a practical application or amount to significantly more than the judicial exception.
Claim 3 recites determining of the duration in which the visual fatigue occurs comprises determining a reduction duration in which the calculated pupil size change speed is decreased and determining, from among the plurality of frames, the duration in which the visual fatigue of the measurement target occurs, on the basis of the determined reduction duration. An individual can mentally determine the duration of fatigue by mentally analyzing the pupil size change speed decreasing. Thus, the feature of claim 3 is directed to the mental process. Accordingly, the claim does not recite any additional limitations that integrate the judicial exception into a practical application or amount to significantly more than the judicial exception.
Claim 4 recites determining of the duration in which the visual fatigue occurs comprises determining the duration in which the visual fatigue of the measurement target occurs, from among the plurality of frames, as a result of combining a tendency to gradually increase the calculated number of eye blinks, a tendency to gradually increase the eye closed time, and a reduction duration in which the calculated pupil size change speed is decreased. An individual can mentally determine the duration of fatigue by mentally analyzing a gradual increase of the number of eye blinks, a gradual increase of the eye closed time, and a pupil size change speed decreasing. Thus, the feature of claim 4 is directed to the mental process and mathematical concept. Accordingly, the claim does not recite any additional limitations that integrate the judicial exception into a practical application or amount to significantly more than the judicial exception.
Claim 5 recites determining of the size of the pupil comprises: extracting, from among the plurality of frames, a frame in which the eye of the measurement target is not closed; detecting at least one candidate group by applying binarization and contour detection techniques to an extracted frame; removing, from among the at least one candidate group detected, reflected light caused by infrared light; and measuring a diameter of the pupil by considering a distribution of black pixels from among the candidate group from which the reflected light is removed. While the processing of data cannot be described as simple data gathering, the processing does not integrate the mental process into a practical application or constitute as significantly more. Additionally, measuring the diameter of the pupil is a mental process with the support of measurement tools. Thus, the feature of claim 5 is directed to the mental process. Accordingly, the claim does not recite any additional limitations that integrate the judicial exception into a practical application or amount to significantly more than the judicial exception.
Claim 6 recites a non-transitory computer-readable recording medium for performing the method of claim 1. An individual can mentally perform the method of claim 1 (as seen from previous analysis). The non-transitory computer-readable recording medium amount to generic computer components. Thus, the feature of claim 6 is directed to the mental process. Accordingly, the claim does not recite any additional limitations that integrate the judicial exception into a practical application or amount to significantly more than the judicial exception
Claim 7 (and by dependency 8-10) are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter as follows.
Claim 7 recites a program. Specifically, the 112 invoked for “a imager receiver, pupil size determiner, parameter calculator, and fatigue occurrence determiner” requires the specification to be pointed to for structure. However, the language describing the structure is broad in nature and encompasses a program (Page 25, lines 7-14: “Embodiments according to the present disclosure described above may be implemented in the form of a computer program that may be executed through various components on a computer, and the computer program may be recorded on a computer-readable medium”. The language of “may” implies the computer program may not be executed through components on a computer, and hence is a program). Computer programs, per se, are not in one of the statutory categories of invention because a computer program is merely a set of instructions capable of being executed by a computer - the computer program itself is not a process. MPEP § 2106.
A computer program, at best, is a functional descriptive material per se. Descriptive material can be characterized as either "functional descriptive material" or "nonfunctional descriptive material." Both types of "descriptive material" are nonstatutory when claimed as descriptive material per se, 33 F.3d at 1360, 31 USPQ2d at 1759. When functional descriptive material is recorded on some computer-readable medium, it becomes structurally and functionally interrelated to the medium and will be statutory in most cases since use of technology permits the function of the descriptive material to be realized. Compare In re Lowry, 32 F.3d 1579, 1583-84, 32 USPQ2d 1031, 1035 (Fed. Cir. 1994) )(discussing patentable weight of data structure limitations in the context of a statutory claim to a data structure stored on a computer readable medium that increases computer efficiency) and >In re Warmerdam, 33 F.3d *>1354, 1360- 61,31 USPQ2d *>1754, 1759 (claim to computer having a specific data structure stored in memory held statutory product-by-process claim) with Warmerdam, 33 F.3d at 1361,31 USPQ2d at 1760 (claim to a data structure per se held nonstatutory). See MPEP 2106.01.
The rejection of claim 7 above may be overcome by amending the claim to incorporate hardware that the software is stored on.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
The following is a quotation of pre-AIA 35 U.S.C. 112, fourth paragraph:
Subject to the following paragraph [i.e., the fifth paragraph of pre-AIA 35 U.S.C. 112], a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claim 6 is rejected under 35 U.S.C. 112(d) or pre-AIA 35 U.S.C. 112, 4th paragraph, as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends. Claim 6 recites “A non-transitory computer-readable recording medium storing a program for executing the method of claim 1”. The claim depends on claim 1, however one does not need to perform the steps of claim 1 to anticipate/infringe this claim. The claim simply describes a non-transitory computer-readable recording medium storing a program, which does not need the steps of claim 1. Applicant may cancel the claim(s), amend the claim(s) to place the claim(s) in proper dependent form, rewrite the claim(s) in independent form, or present a sufficient showing that the dependent claim(s) complies with the statutory requirements. See MPEP 608.01(n)(III) for more information.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4 and 6-10 rejected under 35 U.S.C. 103 as being unpatentable over SUN et al. (US 20210042497 A1 Hereinafter “SUN”) in view of Roe et al. (US 20180139434 A1 Hereinafter “Roe”).
Regarding claim 1, SUN teaches a method of determining a visual fatigue occurrence duration, the method comprising:
receiving an eye image that is an image comprising a plurality of frames and obtained by photographing an eye of a measurement target (Fig. 1: S10 describes receiving an eye image of a measurement target (user). This eye image can be multiple frames “For example, in a case where the eye image of the user is acquired by an image capture device (e.g., an infrared camera), the image capture device may capture a plurality of frames of eye images” [0048]);
determining a size of a pupil by analyzing the eye image (Fig. 1, [0046]: “The extracting the visual features from the eye image as described in the above step S20 may correspondingly include: step S201, acquiring pupil positions, pupil areas, and/or blink counts from successive frames of the eye images”. The area of a pupil is the size of the pupil);
calculating each of a number of eye blinks, an eye closed time, ([0046]: “In some examples, the visual features may include a velocity of pupil motion, a angular velocity of pupil motion, a time duration of closing eyes and/or a blink frequency”. This is on the basis of when the eye closes because blinks and time duration of closing eyes require the eye to be closed and is based on the eye being closed. This also happens in the frames captured); and
determining a duration in which visual fatigue of the measurement target occurs, from among the plurality of frames, as a result of combining information regarding the calculated number of eye blinks, eye closed time, ([0056]: “In some examples, corresponding with extracting the visual features, the calculating the visual fatigue value according to the visual features includes: acquiring a first visual fatigue value by comparing the mean velocity of pupil motion with mean velocity grade threshold values of pupil motion, acquiring a second visual fatigue value by comparing the mean angular velocity of pupil motion with mean angular velocity grade threshold values of pupil motion, acquiring a third visual fatigue value by comparing the mean time duration of closing eyes with mean time duration grade threshold values of closing eyes, and/or acquiring a fourth visual fatigue value by comparing the mean blink frequency with grade threshold values of a mean blink frequency”. All four of these values are used to determine fatigue “For example, calculating visual fatigue value according to the visual features further includes determining the visual fatigue value according to the first visual fatigue value, the second visual fatigue value, the third visual fatigue value, and/or the fourth visual fatigue value” [0070]. The parameter values are also captured during specific time periods which allow determination of fatigue for certain durations (time periods) “calculating the mean velocity of pupil motion according to each of the pupil positions within a first preset time period; step S203, calculating the mean angular velocity of pupil motion according to each of the pupil positions within a second preset time period; step S204 calculating the mean time duration of closing eyes according to each of the pupil areas within a third preset time period; and/or step S205, calculating the mean blink frequency according to the blink counts within a fourth preset time period” [0046]. These time periods can also be different or the same “For example, the first preset time period, the second preset time period, the third preset time period, and the fourth preset time period described above may be the same time period, or may be different time periods, and embodiments of the present disclosure are not limited thereto” [0047]).
While SUN does disclose using multiple parameter values to determine fatigue, SUN does not explicitly disclose pupil size change speed as one of the parameters for determining fatigue.
However, Roe teaches monitoring pupil size change speed to determine eye fatigue ([0039]: “Pupil diameter, eye movement velocity, and speed of pupil accommodation are found to change as a result of fatigue or visual fatigue. In a particular environmental condition, a person whose eyes are fatigued will have a smaller pupil diameter than if that person were not fatigued. Eye movement velocity, such as the speed of saccades (which are quick, simultaneous movement of the eyes while scanning), can decrease as a result of visual fatigue. Pupil accommodation is a reflex action of the pupil in which it contracts or dilates based on lighting conditions or change of focus; a slower accommodation speed can indicate visual fatigue”).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify SUN’s visual fatigue detection to include Roe’s pupil diameter measurements and use of pupil size change speed (speed of pupil accommodation) for fatigue determination because such a modification is the result of applying a known technique to a known device ready for improvement to yield predictable results. More specifically, include Roe’s pupil diameter measurements and use of pupil size change speed (speed of pupil accommodation) for fatigue determination permits method for accurately determining fatigue. This known benefit in Roe is applicable to SUN’s visual fatigue detection as they both share characteristics and capabilities, namely, they are directed to detection of visual fatigue by extracting features associated with the eye. Additionally, SUN already uses multiple parameters (a velocity of pupil motion, a angular velocity of pupil motion, a time duration of closing eyes and blink frequency) to determine visual fatigue, SUN would find it beneficial to have an additional way to detect visual fatigue that differs from what SUN is already using. Therefore, it would have been recognized that SUN’s visual fatigue detection to include Roe’s pupil diameter measurements and use of pupil size change speed (speed of pupil accommodation) for fatigue determination would have yielded predictable results because (i) the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate Roe’s pupil diameter measurements and use of pupil size change speed (speed of pupil accommodation) for fatigue determination in detection of visual fatigue by extracting features associated with the eye and (ii) the benefits of such a combination would have been recognized by those of ordinary skill in the art.
Regarding claim 2, the combination of SUN and Roe teaches the method of claim 1, in addition, SUN further teaches wherein the determining of the duration in which the visual fatigue occurs comprises combining a tendency to gradually increase the calculated number of eye blinks ([0089]: “The mild fatigue threshold value of the mean blink frequency is less than the moderate fatigue threshold value of the mean blink frequency, and the moderate fatigue threshold value of the mean blink frequency is less than the severe fatigue threshold value of the mean blink frequency”. Given the higher the mean blink frequency the more severe the fatigue, the visual fatigue is defined by a tendency to gradually increase the calculated number of eye blinks) and a tendency to gradually increase the eye closed time ([0088]: “The mild fatigue threshold value of the mean time duration of closing eyes is less than the moderate fatigue threshold value of the mean time duration of closing eyes, and the moderate fatigue threshold value of the mean time duration of closing eyes is less than the severe fatigue threshold value of the mean time duration of closing eyes”. Given the longer the eye closed time, the more severe the fatigue, the visual fatigue is defined by a tendency to gradually increase the eye closed time), comparing a result of the combination with a preset condition ([0071]: “For example, in a case where the first sequence, the second sequence, the third sequence, and the fourth sequence are the same, the visual fatigue value may be calculated by step S210: calculating a sum of the first visual fatigue value, the second visual fatigue value, and the third visual fatigue value and/or the fourth visual fatigue value described above as the visual fatigue value, i.e., the visual fatigue value m is a sum of the first visual fatigue value m1, the second visual fatigue value m2, the third visual fatigue value m3, and/or the fourth visual fatigue value m4, i.e., m=m1+m2+m3+m4, which may also be referred to as a visual fatigue overall value”. This fatigue value is compared to a preset condition (threshold) to determine fatigue “For example, comparing the visual fatigue value with fatigue grade threshold values and determining the visual fatigue grade according to the comparison result includes: determining the visual fatigue grade to be a mild fatigue grade in a case where the visual fatigue value is greater than or equal to the mild fatigue threshold value and less than the moderate fatigue threshold value” [0073]), and determining, from among the plurality of frames, the duration in which the visual fatigue occurs (The parameter values are captured during specific time periods which allow determination of fatigue for certain durations (time periods) “calculating the mean velocity of pupil motion according to each of the pupil positions within a first preset time period; step S203, calculating the mean angular velocity of pupil motion according to each of the pupil positions within a second preset time period; step S204 calculating the mean time duration of closing eyes according to each of the pupil areas within a third preset time period; and/or step S205, calculating the mean blink frequency according to the blink counts within a fourth preset time period” [0046]. These time periods can also be different or the same “For example, the first preset time period, the second preset time period, the third preset time period, and the fourth preset time period described above may be the same time period, or may be different time periods, and embodiments of the present disclosure are not limited thereto” [0047]. If a fatigue value is above a threshold for any of the time periods that time period would be a duration in which fatigue occurs).
Regarding claim 3, the combination of SUN and Roe teaches the method of claim 1, in addition, Roe further teaches wherein the determining of the duration in which the visual fatigue occurs comprises determining a reduction duration in which the calculated pupil size change speed is decreased ([0039]: “Pupil diameter, eye movement velocity, and speed of pupil accommodation are found to change as a result of fatigue or visual fatigue. In a particular environmental condition, a person whose eyes are fatigued will have a smaller pupil diameter than if that person were not fatigued. Eye movement velocity, such as the speed of saccades (which are quick, simultaneous movement of the eyes while scanning), can decrease as a result of visual fatigue. Pupil accommodation is a reflex action of the pupil in which it contracts or dilates based on lighting conditions or change of focus; a slower accommodation speed can indicate visual fatigue”. Slower accommodation speed is the calculated pupil size change speed decreasing) and determining, from among the plurality of frames, the duration in which the visual fatigue of the measurement target occurs, on the basis of the determined reduction duration ([0041]: “In some embodiments, the media guidance application calculates the first metric by measuring a first eye strain factor of the first viewer during a first time period”. The eye strain is being measured in the first time period. If fatigue is determined, then the duration is during the first time period. Furthermore, Roe is relied on for tracking pupil size change speed, SUN already had time periods for measuring SUN’s parameters, using Roe’s pupil size change speed tracking would result in using one of the time periods in SUN for the pupil size change speed and therefore determining the duration of fatigue. Hence the combination of SUN and Roe teaches determining a duration of fatigue based on the pupil size change speed).
The rationale for this combination is similar to the claim 1 combination due to similar methods of combination (Roe’s pupil size change speed is used for fatigue detection in claim 1), and similar benefits (accurate method of fatigue determination).
Regarding claim 4, the combination of SUN and Roe teaches the method of claim 1, in addition, SUN further teaches wherein the determining of the duration in which the visual fatigue occurs comprises determining the duration in which the visual fatigue of the measurement target occurs, from among the plurality of frames, as a result of combining a tendency to gradually increase the calculated number of eye ([0089]: “The mild fatigue threshold value of the mean blink frequency is less than the moderate fatigue threshold value of the mean blink frequency, and the moderate fatigue threshold value of the mean blink frequency is less than the severe fatigue threshold value of the mean blink frequency”. Given the higher the mean blink frequency the more severe the fatigue, the visual fatigue is defined by a tendency to gradually increase the calculated number of eye blinks), a tendency to gradually increase the eye closed time ([0088]: “The mild fatigue threshold value of the mean time duration of closing eyes is less than the moderate fatigue threshold value of the mean time duration of closing eyes, and the moderate fatigue threshold value of the mean time duration of closing eyes is less than the severe fatigue threshold value of the mean time duration of closing eyes”. Given the longer the eye closed time, the more severe the fatigue, the visual fatigue is defined by a tendency to gradually increase the eye closed time), and ([0071]: “For example, in a case where the first sequence, the second sequence, the third sequence, and the fourth sequence are the same, the visual fatigue value may be calculated by step S210: calculating a sum of the first visual fatigue value, the second visual fatigue value, and the third visual fatigue value and/or the fourth visual fatigue value described above as the visual fatigue value, i.e., the visual fatigue value m is a sum of the first visual fatigue value m1, the second visual fatigue value m2, the third visual fatigue value m3, and/or the fourth visual fatigue value m4, i.e., m=m1+m2+m3+m4, which may also be referred to as a visual fatigue overall value”. This fatigue value is compared to a preset condition (threshold) to determine fatigue “For example, comparing the visual fatigue value with fatigue grade threshold values and determining the visual fatigue grade according to the comparison result includes: determining the visual fatigue grade to be a mild fatigue grade in a case where the visual fatigue value is greater than or equal to the mild fatigue threshold value and less than the moderate fatigue threshold value” [0073]. The parameter values are captured during specific time periods which allow determination of fatigue for certain durations (time periods) “calculating the mean velocity of pupil motion according to each of the pupil positions within a first preset time period; step S203, calculating the mean angular velocity of pupil motion according to each of the pupil positions within a second preset time period; step S204 calculating the mean time duration of closing eyes according to each of the pupil areas within a third preset time period; and/or step S205, calculating the mean blink frequency according to the blink counts within a fourth preset time period” [0046]. These time periods can also be different or the same “For example, the first preset time period, the second preset time period, the third preset time period, and the fourth preset time period described above may be the same time period, or may be different time periods, and embodiments of the present disclosure are not limited thereto” [0047]. If a fatigue value is above a threshold for any of the time periods that time period would be a duration in which fatigue occurs).
Roe further teaches the determining of the duration in which the visual fatigue occurs comprises determining a reduction duration in which the calculated pupil size change speed is decreased ([0039]: “Pupil diameter, eye movement velocity, and speed of pupil accommodation are found to change as a result of fatigue or visual fatigue. In a particular environmental condition, a person whose eyes are fatigued will have a smaller pupil diameter than if that person were not fatigued. Eye movement velocity, such as the speed of saccades (which are quick, simultaneous movement of the eyes while scanning), can decrease as a result of visual fatigue. Pupil accommodation is a reflex action of the pupil in which it contracts or dilates based on lighting conditions or change of focus; a slower accommodation speed can indicate visual fatigue”. Slower accommodation speed is the calculated pupil size change speed decreasing).
SUN already uses multiple parameters (a velocity of pupil motion, a angular velocity of pupil motion, a time duration of closing eyes and blink frequency) to determine visual fatigue, so using a combination of measured parameters to determine fatigue is already contemplated. Roe provides another method for determining fatigue. If used in SUN’s fatigue detection system, Roe’s eye size changing speed would be used in conjunction with the other parameters to determine fatigue (in a combination). Therefore, the combination of SUN and Roe teaches determining of the duration in which the visual fatigue occurs comprising determining the duration in which the visual fatigue of the measurement target occurs, from among the plurality of frames, as a result of combining (SUN and Roe combination) a tendency to gradually increase the calculated number of eye (SUN), a tendency to gradually increase the eye closed time (SUN), and a reduction duration in which the calculated pupil size change speed is decreased (Roe).
The rationale for this combination is similar to the claim 1 combination due to similar methods of combination (Roe’s pupil size change speed is used for fatigue detection in claim 1), and similar benefits (accurate method of fatigue determination).
Regarding claim 6, SUN teaches a non-transitory computer-readable recording medium storing a program ([0024]: “Some embodiments of the present disclosure further provide a storage medium, non-transitorily storing computer readable instructions. The above visual fatigue recognition method is performed in a case where the non-transitory computer readable instructions are executed by a computer”) for executing the method of claim 1 as taught by the combination of SUN and Roe.
Regarding claim 7, the content of claim 7 is similar to the content of claim 1, with the additional teachings of a imager receiver, pupil size determiner, parameter calculator, and fatigue occurrence determiner. SUN also discloses this information (Fig. 5A shows the different hardware units used to perform their method.
The eye image acquisition unit: The imager receiver.
The visual fatigue value acquisition unit: is pupil size determiner ([0078]: “a visual fatigue value acquisition unit 502 configured to acquire visual features from the eye image and to calculate a visual fatigue value according to the visual features”. These features can include pupil size determination “correspondingly, the visual fatigue value acquisition unit comprises: a visual feature acquisition sub-unit configured to respectively acquire pupil positions, pupil areas” [0018]).
The visual fatigue value acquisition unit: also the parameter calculator ([0078]: “a visual fatigue value acquisition unit 502 configured to acquire visual features from the eye image and to calculate a visual fatigue value according to the visual features”. These features can include calculations of number of blinks and eye close time “In some examples, the visual features may include a velocity of pupil motion, a angular velocity of pupil motion, a time duration of closing eyes and/or a blink frequency” [0046])
The visual fatigue grade determination unit: fatigue occurrence determiner ([0078]: “a visual fatigue grade determination unit 503 configured to compare the visual fatigue value with a fatigue grade threshold values and to determine a visual fatigue grade according to the comparison result”).
All of these units can be implemented as hardware “The individual units in the device of the embodiments of the present disclosure may be implemented by virtue of software, or by mean of both software and hardware, and by hardware as well” [0099]). Therefore, claim 7 is rejected for the same reasons of obviousness as claim 1, along with the additional teachings above.
Regarding claim 8, the content of claim 8 is similar to the content of claim 2, therefore it is rejected for the same reasons of obviousness as claim 2.
Regarding claim 9, the content of claim 9 is similar to the content of claim 3, therefore it is rejected for the same reasons of obviousness as claim 3.
Regarding claim 10, the content of claim 10 is similar to the content of claim 4, therefore it is rejected for the same reasons of obviousness as claim 4.
Claims 5 and 11 rejected under 35 U.S.C. 103 as being unpatentable over SUN et al. (US 20210042497 A1 Hereinafter “SUN”) in view of Roe et al. (US 20180139434 A1 Hereinafter “Roe”) in further view of Ohta (US 20160286092 A1 Hereinafter “Ohta”) in further view of MacDougall et al. (US 20200029808 A1 Hereinafter “MacDougall”).
Regarding claim 5, the combination of SUN and Roe teaches the method of claim 1, in addition, SUN further teaches wherein the determining of the size of the pupil comprises:
extracting, from among the plurality of frames, a frame in which the eye of the measurement target is not closed ([0039]: “In some examples, a deep learning-based image recognition algorithm can be employed to extract visual features from the eye image, the visual features for example including a velocity of pupil motion, an angular velocity of pupil motion”. If pupil motion is detected images must be captured where the eye of the measurement target (user) is not closed);
Roe further teaches measuring a diameter of the pupil by considering a distribution of black pixels from among the candidate group ([0128]: “As another example, control circuitry 604 may measure a pupil diameter of the first viewer by assessing the distance across of the black circle in the center of the viewer's eye(s)”. Measuring the diameter allows Roe to perform pupil size change speed tracking).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify SUN’s visual fatigue detection to include Roe’s pupil diameter measurements and use of pupil size change speed (speed of pupil accommodation) for fatigue determination because such a modification is the result of applying a known technique to a known device ready for improvement to yield predictable results. More specifically, include Roe’s pupil diameter measurements and use of pupil size change speed (speed of pupil accommodation) for fatigue determination permits method for accurately determining fatigue. This known benefit in Roe is applicable to SUN’s visual fatigue detection as they both share characteristics and capabilities, namely, they are directed to detection of visual fatigue by extracting features associated with the eye. Additionally, SUN already uses multiple parameters (a velocity of pupil motion, a angular velocity of pupil motion, a time duration of closing eyes and blink frequency) to determine visual fatigue, SUN would find it beneficial to have an additional way to detect visual fatigue that differs from what SUN is already using. Therefore, it would have been recognized that SUN’s visual fatigue detection to include Roe’s pupil diameter measurements and use of pupil size change speed (speed of pupil accommodation) for fatigue determination would have yielded predictable results because (i) the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate Roe’s pupil diameter measurements and use of pupil size change speed (speed of pupil accommodation) for fatigue determination in detection of visual fatigue by extracting features associated with the eye and (ii) the benefits of such a combination would have been recognized by those of ordinary skill in the art.
While SUN does describe preprocessing the eye image using grey level processing and contrast enhancement ([0045]: “In some examples, the brightness and contrast of the eye image may be improved by an image processing algorithm, such as a grey-level transformation method or a histogram adjustment method. Moreover, because the acquired original eye image generally has certain noises, in order to further improve the quality of the eye image, the noises in the eye image can be removed by a further filtering process on the eye image using a filtering algorithm, which helps to extract the visual features from the eye image of a high quality”), the combination of SUN and Roe does not expressly disclose using binarization and contour detection for preprocessing the eye image.
However, Ohta teaches using binarization and contour detection for preprocessing the eye image to detect a pupil ([0052]: “The processed-image generating unit 13 is a unit that detects an edge with respect to the image clipped by the pupil detecting unit 12 and binarizes an obtained image”).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify the combination of SUN and Roe’s visual fatigue detection system to include Ohta’s edge detection and binarization because such a modification is the result of applying a known technique to a known device ready for improvement to yield predictable results. More specifically, include Ohta’s edge detection and binarization permits a method for accurately extracting the pupil region. This known benefit in Ohta is applicable to the combination of SUN and Roe’s visual fatigue detection system as they both share characteristics and capabilities, namely, they are directed to processing images of eyes for further operations. Therefore, it would have been recognized that the combination of SUN and Roe’s visual fatigue detection system to include Ohta’s edge detection and binarization would have yielded predictable results because (i) the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate Ohta’s edge detection and binarization in processing images of eyes for further operations and (ii) the benefits of such a combination would have been recognized by those of ordinary skill in the art.
The combination of SUN, Roe, and Ohta does not expressly disclose removing reflected light caused by infrared light from the images.
However, MacDougall teaches removing reflected light caused by infrared light from the images ([0029]: “The system 100 is utilized for the removal of reflection 118 or hotspots from an image of an eye”. This reflected light can come from infrared lights “The first and second illuminators can be infrared light emitting diodes (IR LEDs) or any other light source” [0008]).
At the time the invention was effectively filed, it would have been obvious to one of ordinary skill in the art to modify the combination of SUN, Roe, and Ohta’s eye fatigue detection system’s image pre-processing to include MacDougall’s removal of reflections in the image caused by infrared light because such a modification is taught, suggested, or motivated by the art. More specifically, the motivation to modify the combination of SUN, Roe, and Ohta to include MacDougall is implicitly provided by MacDougall, stating that removal of reflected light is done to obtain a clear image for further processes using the eye data ([0006]: “Such a system and method would focus on removal of reflected light from every portion of the eye to obtain a clear image that may be utilized for diagnosis or biometric access”). Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify the combination of SUN, Roe, and Ohta’s eye fatigue detection system’s image pre-processing to include MacDougall’s removal of reflections in the image caused by infrared light with the motivation of obtaining a clear image for extracting features of the eye. The person of ordinary skill in the art would have recognized the benefit of improved image data for feature extraction.
Regarding claim 11, the content of claim 11 is similar to the content of claim 5, therefore it is rejected for the same reasons of obviousness as claim 5.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Sakamoto et al. (US 20080117323 A1) teaches fatigue detection and measuring rate of change of pupil diameter, eye close time, and number of blinks.
AI et al. (WO 2021068256 A1) teaches fatigue detection using blinking frequency and blink time.
Kuperman (US 20220180993 A1) teaches fatigue determination and measure attributes such as blink amount, blink duration, and a plethora of other attributes.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to STEFANO A DARDANO whose telephone number is (703)756-4543. The examiner can normally be reached Monday - Friday 11:00 - 7:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Greg Morse can be reached at (571) 272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/STEFANO ANTHONY DARDANO/ Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698