DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
The reply filed on 4/10/2026 has been entered. Applicant’s arguments with respect to claims 1-20 have been considered but are moot in view of new ground(s) of rejection caused by the amendments. Claims 1-20 are pending in this application and have been considered below.
Priority
Receipt is acknowledged that application claims priority to foreign application with application number CN202310182114.1 dated 2/20/2023. Copies of certified papers required by 37 CFR 1.55 have been received. Priority is acknowledged under 35 USC 119(e) and 37 CFR 1.78.
Information Disclosure Statement
The IDS dated 5/14/2024 that have been previously considered remain placed in the application file.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 4, 6, 7, 8, 11, 13, 14, 15, 18, 19, and 20 are rejected under 35 U.S.C. 103 as obvious over US Patent Publication 2021 0068652 A1, (Nistico) in view of US Patent Publication 2021 0165993 A1, (Wang et al.) and Chinese Patent Application 2022 114708232 A, (Hou).
Claim 1
Regarding Claim 1, Nistico teach a method for detecting a direction of sight, comprising:
after emitting detection light to a cornea of a user, obtaining a first image by acquiring an image of the cornea, ("The one or more light sources 422 emit light onto the eye of the user 10 that reflects light (e.g., a directional beam) that can be detected by the sensor 424," par. 57) wherein at least one reflection light spot is displayed in the first image, ("In some implementations, the one or more image sensor systems 314 are configured to obtain image data that corresponds to at least a portion of the face of the user that includes the eyes of the user," par. 46) and the reflection light spot is formed by the cornea reflecting the detection light; ("The pixels associated with a reflection (e.g., glint) and/or the sensor's known position or orientation relative to the light source can be used to determine the direction (e.g., angle) the reflection," par. 90 wherein a "glint" is the reflection light spot) determining a third position of the reflection light spot according to the first prediction diagram; ("The pixels associated with a reflection (e.g., glint) and/or the sensor's known position or orientation relative to the light source can be used to determine the direction (e.g., angle) the reflection," par. 90) and determining a gazing direction of the user based on the third position of the reflection light spot ("Based on the reflected glint(s), the controller 480 can determine a gaze direction of the user 10," par. 57).
Nistico do not explicitly teach generating a first prediction diagram by processing the first image based on a pre-trained image processing model, wherein the first prediction diagram characterizes a probability of a pixel on the first image being a second pixel, and the second pixel is a pixel constituting the reflection light spot, wherein the first prediction diagram is a pixel-level probability diagram, and a probability value of each pixel corresponds to a probability that the pixel belongs to the reflection light spot.
However, Wang et al. teach generating a first prediction diagram by processing the first image based on a pre-trained image processing model ("the method further includes a following operation: after the eye-area image is inputted into the neural network trained in advance and the line-of-sight direction of the eye-area image is outputted," par. 53).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the gaze direction determination method as taught by Nistico to use the neural network training methods as taught by Wang et al. The suggestion/motivation for doing so would have been that, “the device for training the neural network not only may automatically obtain first line-of-sight directions but also may accurately obtain a large number of first line-of-sight directions. Therefore, a large amount of accurate and reliable data are provided to train the neural network, which increases an efficiency of the training and accuracy in predicting the line-of-sight directions” as noted by the Wang et al. disclosure in paragraph [0048].
Additionally, Hou teach wherein the first prediction diagram characterizes a probability of a pixel on the first image being a second pixel, and the second pixel is a pixel constituting the reflection light spot, wherein the first prediction diagram is a pixel-level probability diagram, and a probability value of each pixel corresponds to a probability that the pixel belongs to the reflection light spot ("generating reflection intensity map. the reflection intensity pattern can be used for feeding back at least one pixel point corresponding to the pixel point intensity, at least one pixel point corresponding to the received light energy is directly proportional to the pixel intensity of the pixel point in the reflection intensity image, namely the pixel point corresponding to the received light energy is stronger, the pixel intensity of the pixel point is higher," translated page 4).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the gaze direction determination method as taught by Nistico and the neural network training methods as taught by Wang et al. to use the reflection intensity pattern as taught by Hou. The suggestion/motivation for doing so would have been that, “the light intensity values received by each Pixel (pixel) of the light emitting end (RX) are typically characterized by the confidence of the pixel (point) to generate a depth value, i.e., Confidence. and the reflection intensity map can be understood as feedback for each Pixel (pixel) received light intensity value, in some embodiments, the reflection intensity map also can be referred to as a confidence graph (Confidence graph), the parameter value of each pixel point in the confidence graph is characterized by the intensity of the received signal light, so as to be the confidence of the point target distance measuring value” as noted by the Hou et al. disclosure on translated page 4, paragraph 6, which also motivates combination because the combination would predictably have a higher accuracy as there is a reasonable expectation that incorporating this confidence data into the model would improve its depth perception, thereby providing more precise spatial coordinates for estimating gaze direction; and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
The rejection system of claim 1 above applies mutatis mutandis to the corresponding limitations of apparatus claim 8 and electronic device claim 15 while noting that the rejection above cites to both device and method disclosures. Claims 8 and 15 are mapped below for clarity of the record and to specify any new limitations not included in claim 1.
Claim 4
Regarding Claim 4, Nistico teach obtaining a fourth image according to the first image and the second image, wherein the fourth image is a collection of pixels in the contour range of the first image, ("Each image includes a matrix of pixel values corresponding to pixels of the image which correspond to locations of a matrix of light sensors of the camera," par. 72) wherein generating the first prediction diagram by processing the first image based on the pre-trained image processing model, comprises: generating the first prediction diagram ("Each image includes a matrix of pixel values corresponding to pixels of the image which correspond to locations of a matrix of light sensors of the camera," par. 72).
Nistico do not explicitly teach all of obtaining a second image corresponding to the first image, wherein the second image is configured to characterize a contour range corresponding to the cornea in the first image; and generating the first prediction diagram by processing the fourth image according to the pre-trained image processing model.
However, Wang et al. teach obtaining a second image corresponding to the first image, wherein the second image is configured to characterize a contour range corresponding to the cornea in the first image; ("positions where images of light sources are formed on the cornea reference point, namely coordinates of reflection points in the second camera coordinate system, may be determined," par. 87) and generating the first prediction diagram by processing the fourth image according to the pre-trained image processing model ("the method further includes a following operation: after the eye-area image is inputted into the neural network trained in advance and the line-of-sight direction of the eye-area image is outputted," par. 53).
Nistico, Wang et al., and Hou are combined as per claim 1.
Claim 6
Regarding Claim 6, Nistico teach wherein determining the gazing direction of the user based on the third position of the reflection light spot comprises: determining a cornea center based on the third position of the reflection light spot; ("the method 600 determines a cornea center of the eye based on the first direction of the light beam and the second direction of the reflection. For example, when a glint is detected, the associated angle of the light source associated with the light that produced the glint may be identified and used to determine the glint location on the cornea surface," par. 91) detecting a pupil position of the user, and determining a pupil center based on a preset refraction angle; ("A gaze tracking system including light source 716 and camera 728 is used to track the gaze direction. The light source produces omnidirectional light that produces glint 722 by reflecting off the surface of the cornea 710. The camera 728 captures an image that includes a pupil image 730 and a glint image 732 … The pupil center 712 is also determined, for example, using image processing on the image data obtained by the camera 728 … FIG. 11 illustrates a functional block diagram illustrating gaze tracking using two variable angle light sources (e.g., a scanners 916a and 916b) at a known angles," par. 100, 103, and 104) and determining the gazing direction according to the cornea center and the pupil center ("the method 600 determines the gaze direction by determining a direction from the pupil center through the cornea center or a direction from the eyeball center through the cornea center," par. 93).
Nistico, Wang et al., and Hou are combined as per claim 1.
Claim 7
Regarding Claim 7, Nistico teach wherein before obtaining the first image by acquiring the image of the cornea, the method further comprises: performing image acquisition for an eye of the user to obtain a third image; ("In some implementations, the one or more image sensor systems 314 are configured to obtain image data that corresponds to at least a portion of the face of the user that includes the eyes of the
PNG
media_image1.png
536
486
media_image1.png
Greyscale
user," par. 46) determining, based on the third image, a first positional relationship between the cornea of the user and a camera unit for obtaining the first image; (" FIG. 11 illustrates a functional block diagram illustrating gaze tracking using two variable angle light sources (e.g., a scanners 916a and 916b) at a known angles. In this example, a first glint produced by [AltContent: textbox (Figure 11 shows gaze-tracking using variable light sources at known angles.)]a light beam from a light source directed by scanner 916a forms a glint image 932a at the camera 728. The glint and angle of the scanner 916a are used to determine cornea point a 1210a. The second glint produced by a light beam from a light source directed by scanner 916b forms a glint image 932b at the camera," par. 104) determining a first light source according to light source matrix information and the first positional relationship, ("For example, the rows or columns of a Walsh matrix can be used as the orthogonal codes. Accordingly, in various implementations, a first light source of the plurality of light sources is modulated according to a first orthogonal code and a second light source of the plurality of light sources is modulated according to a second orthogonal code," par. 67) wherein the light source matrix information is configured to characterize a second positional relationship between the camera unit and at least two alternative light sources for emitting detection light; ("the one or more light sources 422 include or are coupled to a scanner that is configured to scan the light from the light source over multiple angles … When a glint, reflected by the eye and detected by the sensor 424, is analyzed, the identity of the glint and the corresponding light source angle (e.g., direction) can be determined, par. 62 and 64) and emitting detection light to the cornea of the user based on the first light source ("In various implementations, the one or more light sources 422 modulate the intensity of emitted light with different modulation frequencies," par. 66).
Nistico, Wang et al., and Hou are combined as per claim 1.
Claim 8
Regarding Claim 8, Nistico et al. teach an apparatus for detecting a direction of sight, comprising:
an acquisition module, configured to, after emitting detection light to a cornea of a user, ("The one or more light sources 422 emit light onto the eye of the user 10 that reflects light (e.g., a directional beam) that can be detected by the sensor 424," par. 57) obtain a first image by acquiring an image of the cornea, ("In some implementations, the one or more image sensor systems 314 are configured to obtain image data that corresponds to at least a portion of the face of the user that includes the eyes of the user," par. 46) wherein at least one reflection light spot is displayed in the first image, and the reflection light spot is formed by the cornea reflecting the detection light; ("The pixels associated with a reflection (e.g., glint) and/or the sensor's known position or orientation relative to the light source can be used to determine the direction (e.g., angle) the reflection," par. 90 wherein a "glint" is the reflection light spot) a first determination module, configured to determine a third position of the reflection light spot according to the first prediction diagram; ("The pixels associated with a reflection (e.g., glint) and/or the sensor's known position or orientation relative to the light source can be used to determine the direction (e.g., angle) the reflection," par. 90) and a second determination module, configured to determine a gazing direction of the user based on the third position of the reflection light spot ("Based on the reflected glint(s), the controller 480 can determine a gaze direction of the user 10," par. 57).
Nistico do not explicitly teach all of a processing module, configured to generate a first prediction diagram by processing the first image based on a pre-trained image processing model, wherein the first prediction diagram characterizes a probability of a pixel on the first image being a second pixel, and the second pixel is a pixel constituting the reflection light spot, wherein the first prediction diagram is a pixel-level probability diagram, and a probability value of each pixel corresponds to a probability that the pixel belongs to the reflection light spot.
However, Wang et al. teach a processing module, configured to generate a first prediction diagram by processing the first image based on a pre-trained image processing model ("the method further includes a following operation: after the eye-area image is inputted into the neural network trained in advance and the line-of-sight direction of the eye-area image is outputted," par. 53).
Additionally, Hou teach wherein the first prediction diagram characterizes a probability of a pixel on the first image being a second pixel, and the second pixel is a pixel constituting the reflection light spot, wherein the first prediction diagram is a pixel-level probability diagram, and a probability value of each pixel corresponds to a probability that the pixel belongs to the reflection light spot ("generating reflection intensity map. the reflection intensity pattern can be used for feeding back at least one pixel point corresponding to the pixel point intensity, at least one pixel point corresponding to the received light energy is directly proportional to the pixel intensity of the pixel point in the reflection intensity image, namely the pixel point corresponding to the received light energy is stronger, the pixel intensity of the pixel point is higher," translated page 4).
Nistico, Wang et al., and Hou are combined as per claim 1.
Claim 11
Regarding Claim 11, Nistico et al. teach wherein the acquisition module is further configured to: obtain a fourth image according to the first image and the second image, wherein the fourth image is a collection of pixels in the contour range of the first image; and the processing module is configured to: generate the first prediction diagram.
Nistico do not explicitly teach all of obtain a second image corresponding to the first image, wherein the second image is configured to characterize a contour range corresponding to the cornea in the first image, ("Each image includes a matrix of pixel values corresponding to pixels of the image which correspond to locations of a matrix of light sensors of the camera," par. 72) and the processing module is configured to: generate the first prediction diagram by processing the fourth image according to the pre-trained image processing model ("Each image includes a matrix of pixel values corresponding to pixels of the image which correspond to locations of a matrix of light sensors of the camera," par. 72).
However, Wang et al. teach obtain a second image corresponding to the first image, wherein the second image is configured to characterize a contour range corresponding to the cornea in the first image, ("positions where images of light sources are formed on the cornea reference point, namely coordinates of reflection points in the second camera coordinate system, may be determined," par. 87) and the processing module is configured to: generate the first prediction diagram by processing the fourth image according to the pre-trained image processing model ("the method further includes a following operation: after the eye-area image is inputted into the neural network trained in advance and the line-of-sight direction of the eye-area image is outputted," par. 53).
Nistico, Wang et al., and Hou are combined as per claim 1.
Claim 13
Regarding Claim 13, Nistico et al. teach wherein the second determining module is configured to: determine a cornea center based on the third position of the reflection light spot; ("At block 630, the method 600 determines a cornea center of the eye based on the first direction of the light beam and the second direction of the reflection. For example, when a glint is detected, the associated angle of the light source associated with the light that produced the glint may be identified and used to determine the glint location on the cornea surface," par. 91) detect a pupil position of the user, and determining a pupil center based on a preset refraction angle; ("A gaze tracking system including light source 716 and camera 728 is used to track the gaze direction. The light source produces omnidirectional light that produces glint 722 by reflecting off the surface of the cornea 710. The camera 728 captures an image that includes a pupil image 730 and a glint image 732 … The pupil center 712 is also determined, for example, using image processing on the image data obtained by the camera 728 … FIG. 11 illustrates a functional block diagram illustrating gaze tracking using two variable angle light sources (e.g., a scanners 916a and 916b) at a known angles," par. 100, 103, and 104) and determine the gazing direction according to the cornea center and the pupil center ("At block 650, the method 600 determines the gaze direction by determining a direction from the pupil center through the cornea center or a direction from the eyeball center through the cornea center," par. 93).
Nistico, Wang et al., and Hou are combined as per claim 1.
Claim 14
Regarding Claim 14, Nistico et al. teach wherein the acquisition module is further configured to: perform image acquisition for an eye of the user to obtain a third image; ("In some implementations, the one or more image sensor systems 314 are configured to obtain image data that corresponds to at least a portion of the face of the user that includes the eyes of the user," par. 46) determine, based on the third image, a first positional relationship between the cornea of the user and a camera unit for obtaining the first image; (" FIG. 11 illustrates a functional block diagram illustrating gaze tracking using two variable angle light sources (e.g., a scanners 916a and 916b) at a known angles. In this example, a first glint produced by a light beam from a light source directed by scanner 916a forms a glint image 932a at the camera 728. The glint and angle of the scanner 916a are used to determine cornea point a 1210a. The second glint produced by a light beam from a light source directed by scanner 916b forms a glint image 932b at the camera," par. 104) determine a first light source according to light source matrix information and the first positional relationship, ("For example, the rows or columns of a Walsh matrix can be used as the orthogonal codes. Accordingly, in various implementations, a first light source of the plurality of light sources is modulated according to a first orthogonal code and a second light source of the plurality of light sources is modulated according to a second orthogonal code," par. 67) wherein the light source matrix information is configured to characterize a second positional relationship between the camera unit and at least two alternative light sources for emitting detection light; ("the one or more light sources 422 include or are coupled to a scanner that is configured to scan the light from the light source over multiple angles … When a glint, reflected by the eye and detected by the sensor 424, is analyzed, the identity of the glint and the corresponding light source angle (e.g., direction) can be determined, par. 62 and 64) and emit detection light to the cornea of the user based on the first light source ("In various implementations, the one or more light sources 422 modulate the intensity of emitted light with different modulation frequencies," par. 66).
Nistico, Wang et al., and Hou are combined as per claim 1.
Claim 15
Regarding Claim 15, Nistico et al. teach an electronic device, comprising: a processor; ("the processor may execute instructions," par. 5) and a memory, being in communication connection to the processor, ("a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors," par. 11) wherein one or more computer-executable instructions are stored on the memory, and the processor is configured to execute the one or more computer-executable instructions stored on the memory to implement a method for detecting a direction of sight, which comprises: ("the processor may execute instructions stored in a non-transitory computer-readable medium to determine or track a gaze direction," par. 5) after emitting detection light to a cornea of a user, ("The one or more light sources 422 emit light onto the eye of the user 10 that reflects light (e.g., a directional beam) that can be detected by the sensor 424," par. 57) obtaining a first image by acquiring an image of the cornea, ("In some implementations, the one or more image sensor systems 314 are configured to obtain image data that corresponds to at least a portion of the face of the user that includes the eyes of the user," par. 46) wherein at least one reflection light spot is displayed in the first image, and the reflection light spot is formed by the cornea reflecting the detection light; ("The pixels associated with a reflection (e.g., glint) and/or the sensor's known position or orientation relative to the light source can be used to determine the direction (e.g., angle) the reflection," par. 90 wherein a "glint" is the reflection light spot)
determining a third position of the reflection light spot according to the first prediction diagram; ("The pixels associated with a reflection (e.g., glint) and/or the sensor's known position or orientation relative to the light source can be used to determine the direction (e.g., angle) the reflection," par. 90) and determining a gazing direction of the user based on the third position of the reflection light spot ("Based on the reflected glint(s), the controller 480 can determine a gaze direction of the user 10," par. 57).
Nistico do not explicitly teach all of generating a first prediction diagram by processing the first image based on a pre-trained image processing model, wherein the first prediction diagram characterizes a probability of a pixel on the first image being a second pixel, and the second pixel is a pixel constituting the reflection light spot, wherein the first prediction diagram is a pixel-level probability diagram, and a probability value of each pixel corresponds to a probability that the pixel belongs to the reflection light spot.
However, Wang et al. teach generating a first prediction diagram by processing the first image based on a pre-trained image processing model ("the method further includes a following operation: after the eye-area image is inputted into the neural network trained in advance and the line-of-sight direction of the eye-area image is outputted," par. 53).
Additionally, Hou teach wherein the first prediction diagram characterizes a probability of a pixel on the first image being a second pixel, and the second pixel is a pixel constituting the reflection light spot, wherein the first prediction diagram is a pixel-level probability diagram, and a probability value of each pixel corresponds to a probability that the pixel belongs to the reflection light spot ("generating reflection intensity map. the reflection intensity pattern can be used for feeding back at least one pixel point corresponding to the pixel point intensity, at least one pixel point corresponding to the received light energy is directly proportional to the pixel intensity of the pixel point in the reflection intensity image, namely the pixel point corresponding to the received light energy is stronger, the pixel intensity of the pixel point is higher," translated page 4).
Nistico, Wang et al., and Hou are combined as per claim 1.
Claim 18
Regarding Claim 8, Nistico et al. teach wherein determining the gazing direction of the user based on the third position of the reflection light spot comprises: determining a cornea center based on the third position of the reflection light spot; ("At block 630, the method 600 determines a cornea center of the eye based on the first direction of the light beam and the second direction of the reflection. For example, when a glint is detected, the associated angle of the light source associated with the light that produced the glint may be identified and used to determine the glint location on the cornea surface," par. 91) detecting a pupil position of the user, and determining a pupil center based on a preset refraction angle; ("A gaze tracking system including light source 716 and camera 728 is used to track the gaze direction. The light source produces omnidirectional light that produces glint 722 by reflecting off the surface of the cornea 710. The camera 728 captures an image that includes a pupil image 730 and a glint image 732 … The pupil center 712 is also determined, for example, using image processing on the image data obtained by the camera 728 … FIG. 11 illustrates a functional block diagram illustrating gaze tracking using two variable angle light sources (e.g., a scanners 916a and 916b) at a known angles," par. 100, 103, and 104) and determining the gazing direction according to the cornea center and the pupil center ("At block 650, the method 600 determines the gaze direction by determining a direction from the pupil center through the cornea center or a direction from the eyeball center through the cornea center," par. 93).
Nistico, Wang et al., and Hou are combined as per claim 1.
Claim 19
Regarding Claim 19, Nistico and Wang et al. teach claim 1.
Nistico teach a computer-readable storage medium, wherein the computer-readable storage medium is configured to store computer-executable instructions, and the computer- executable instructions, when executed by a processor, cause the processor to implement the method for detecting the direction of sight ("For example, the processor may execute instructions stored in a non-transitory computer-readable medium to determine or track a gaze direction," par. 5).
Nistico, Wang et al., and Hou are combined as per claim 1.
Claim 20
Regarding Claim 20, Nistico and Wang et al. teach claim 1.
Nistico teach a computer program product, comprising a computer program, wherein the computer program, when executed by a processor, is configured to implement the method for detecting the direction of sight ("Suitable computing devices include multipurpose microprocessor-based computer systems accessing stored software that programs or configures the computing system from a general purpose computing apparatus to a specialized computing apparatus implementing one or more implementations of the present subject matter, par. 109).
Nistico, Wang et al., and Hou are combined as per claim 1.
2nd Claim Rejections - 35 USC § 103
Claims 2, 9, and 16 are rejected under 35 U.S.C. 103 as obvious over US Patent Publication 2021 0068652 A1, (Nistico), US Patent Publication 2021 0165993 A1, (Wang et al.), and Chinese Patent Application 2022 114708232 A, (Hou) in view of US Patent Publication 2023 0329549 A1, (Tal et al.) and US Patent Publication 2020 0394408 A1, (Sydorenko et al.).
Claim 2
Regarding Claim 2, Nistico, Wang et al., and Hou teach the method according to claim 1, wherein determining the third position of the reflection light spot according to the first prediction diagram comprises: performing detection on the first prediction diagram according to the first pixel threshold to obtain a corresponding first binary diagram characterizing a position distribution of second pixels in the first image, wherein a pixel value of the second pixel is greater than the first pixel threshold; ("The voltage across the photodiode is proportional to the intensity of light impinging on the light sensor … The voltage across the first capacitor and the voltage across the second capacitor are fed to a comparator. When the difference between the voltage across the first capacitor and the voltage across the second capacitor is less than a threshold amount, the comparator outputs a ‘0’ voltage. When the voltage across the first capacitor is higher than the voltage across the second capacitor by at least the threshold amount, the comparator outputs a ‘1’ voltage," par. 75 and 77) according to the first binary diagram, ("The pixels associated with a reflection (e.g., glint) and/or the sensor's known position or orientation relative to the light source can be used to determine the direction (e.g., angle) the reflection," par. 90) and determining the third position of the reflection light spot according to a center point of the light spot region ("The processor may calculate a center of the remaining pixels except for the region removed from the thresholded binary image, and may determine that a corresponding central region is a center of the pupil 691," par. 137).
Nistico, Wang et al., and Hou do not explicitly teach all of obtaining a preset first pixel threshold; and performing connected component merging on the second pixels to obtain a light spot region.
However, Tal et al. teach obtaining a preset first pixel threshold ("In one example, thresholds (e.g., thresholds of pixel brightness of pixels, relationship to nearby pixels, etc.) are used to identify a spot and/or its size, position, and/or other detectable attributes," par. 29).
Sydorenko et al. teach performing connected component merging on the second pixels to obtain a light spot region ("The processor may configure, as a threshold, a maximum value (or a gray level of the darkest pixel in an image) among gray levels of pixels in the pupil region 691. The processor may convert and process a binary image generated with reference to the threshold (thresholded binary image), and thus may remove, from the thresholded binary image, a region smaller than the pupil region 691, par. 137 wherein the processor is converting all pixels so the images will be the same size).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the gaze direction determination method as taught by Nistico, the neural network training methods as taught by Wang et al., and the reflection intensity pattern as taught by Hou to use the imaging-based eye detection system as taught by Tal et al. and corneal region image extraction as taught by Sydorenko et al. The suggestion/motivation for doing so would have been that, “In some implementations, a spot is identified (e.g., its size, shaped, and/or position) and/or distinguished from other less-illuminated portions of the retina. A spot may be identified via an algorithm (e.g. based on a threshold) and/or using a machine learning (ML) model. A ML model may be used to assess/compare the size and/or position of the spot and/or a relationship between multiple spots” as noted by the Tal et al. disclosure in paragraph [0029].
Claim 9
Regarding Claim 9, Nistico, Wang et al., and Hou teach the apparatus according to claim 8, wherein the first determination module is configured to: perform detection on the first prediction diagram according to the first pixel threshold to obtain a corresponding first binary diagram characterizing a position distribution of second pixels in the first image, wherein a pixel value of the second pixel is greater than the first pixel threshold; ("The voltage across the photodiode is proportional to the intensity of light impinging on the light sensor … The voltage across the first capacitor and the voltage across the second capacitor are fed to a comparator. When the difference between the voltage across the first capacitor and the voltage across the second capacitor is less than a threshold amount, the comparator outputs a ‘0’ voltage. When the voltage across the first capacitor is higher than the voltage across the second capacitor by at least the threshold amount, the comparator outputs a ‘1’ voltage," par. 75 and 77), according to the first binary diagram, and determine the third position of the reflection light spot according to a center point of the light spot region ("The processor may calculate a center of the remaining pixels except for the region removed from the thresholded binary image, and may determine that a corresponding central region is a center of the pupil 691," par. 137).
Nistico, Wang et al., and Hou do not explicitly teach all of to obtain a preset first pixel threshold; and perform connected component merging on the second pixels to obtain a light spot region.
However, Tal et al. teach to obtain a preset first pixel threshold ("In one example, thresholds (e.g., thresholds of pixel brightness of pixels, relationship to nearby pixels, etc.) are used to identify a spot and/or its size, position, and/or other detectable attributes," par. 29).
Sydorenko et al. teach to perform connected component merging on the second pixels to obtain a light spot region ("The processor may configure, as a threshold, a maximum value (or a gray level of the darkest pixel in an image) among gray levels of pixels in the pupil region 691. The processor may convert and process a binary image generated with reference to the threshold (thresholded binary image), and thus may remove, from the thresholded binary image, a region smaller than the pupil region 691, par. 137 wherein the processor is converting all pixels so the images will be the same size).
Nistico, Wang et al., Hou, Tal et al., and Sydorenko et al. are combined as per claim 2.
Claim 16
Regarding Claim 16, Nistico, Wang et al., and Hou teach the electronic device according to claim 15, wherein determining the third position of the reflection light spot according to the first prediction diagram comprises: performing detection on the first prediction diagram according to the first pixel threshold to obtain a corresponding first binary diagram characterizing a position distribution of second pixels in the first image, wherein a pixel value of the second pixel is greater than the first pixel threshold; ("The voltage across the photodiode is proportional to the intensity of light impinging on the light sensor … The voltage across the first capacitor and the voltage across the second capacitor are fed to a comparator. When the difference between the voltage across the first capacitor and the voltage across the second capacitor is less than a threshold amount, the comparator outputs a ‘0’ voltage. When the voltage across the first capacitor is higher than the voltage across the second capacitor by at least the threshold amount, the comparator outputs a ‘1’ voltage," par. 75 and 77) according to the first binary diagram, ("The pixels associated with a reflection (e.g., glint) and/or the sensor's known position or orientation relative to the light source can be used to determine the direction (e.g., angle) the reflection," par. 90) and determining the third position of the reflection light spot according to a center point of the light spot region ("The processor may calculate a center of the remaining pixels except for the region removed from the thresholded binary image, and may determine that a corresponding central region is a center of the pupil 691," par. 137).
Nistico, Wang et al., and Hou do not explicitly teach all of obtaining a preset first pixel threshold; and performing connected component merging on the second pixels to obtain a light spot region.
However, Tal et al. teach obtaining a preset first pixel threshold ("In one example, thresholds (e.g., thresholds of pixel brightness of pixels, relationship to nearby pixels, etc.) are used to identify a spot and/or its size, position, and/or other detectable attributes," par. 29).
Sydorenko et al. teach performing connected component merging on the second pixels to obtain a light spot region ("The processor may configure, as a threshold, a maximum value (or a gray level of the darkest pixel in an image) among gray levels of pixels in the pupil region 691. The processor may convert and process a binary image generated with reference to the threshold (thresholded binary image), and thus may remove, from the thresholded binary image, a region smaller than the pupil region 691, par. 137 wherein the processor is converting all pixels so the images will be the same size).
Nistico, Wang et al., Hou, Tal et al., and Sydorenko et al. are combined as per claim 2.
3rd Claim Rejections - 35 USC § 103
Claims 3, 10, and 17 are rejected under 35 U.S.C. 103 as obvious over US Patent Publication 2021 0068652 A1, (Nistico), US Patent Publication 2021 0165993 A1, (Wang et al.), and Chinese Patent Application 2022 114708232 A, (Hou) in view of US Patent Publication 2022 0301217 A1, (Stuart et al.).
Claim 3
Regarding Claim 3, Nistico, Wang et al., and Hou teach the method according to claim 1, wherein determining the third position of the reflection light spot according to the first prediction diagram comprises: the first prediction diagram ("Each image includes a matrix of pixel values corresponding to pixels of the image which correspond to locations of a matrix of light sensors of the camera," par. 72).
Nistico, Wang et al., and Hou do not explicitly teach all of performing a Gaussian fitting to obtain at least one light spot region conforming to a Gaussian distribution; and determining the third position of the reflection light spot according to a Gaussian expectation corresponding to the light spot region.
However, Stuart et al. teach performing a Gaussian fitting to obtain at least one light spot region conforming to a Gaussian distribution; (" In these embodiments, and as an example, Gaussian distributions may be determined for a glint via determining a maxima intensity value within a bounding box surrounding an estimated glint," par. 163) and determining the third position of the reflection light spot according to a Gaussian expectation corresponding to the light spot region ("The image location corresponding to the maxima intensity value may be assigned as the glint location. The estimated glint may be determined based on an increase in image intensity (e.g., brightness) as compared to another portion of the second image," par. 163).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the gaze direction determination method as taught by Nistico, the neural network training methods as taught by Wang et al., and the reflection intensity pattern as taught by Hou to use the glint location identification methods as taught by Stuart et al. The suggestion/motivation for doing so would have been that, “shape regression has become the state-of-the-art approach for accurate and efficient shape alignment. It has been successfully used in face, hand and ear shape estimation. Regression techniques are advantageous because, for example, they are capable of capturing large variances in appearance; they enforce shape constraint between landmarks (e.g., iris between eyelids, pupil inside iris); and they are computationally efficient. While regression techniques are described, it may be appreciated that neural networks may be employed as an alternative to and/or in combination with regression techniques” as noted by the Stuart et al. disclosure in paragraph [0029].
Claim 10
Regarding Claim 10, Nistico, Wang et al., and Hou teach the apparatus according to claim 8, wherein the first determination module is configured to: the first prediction diagram ("Each image includes a matrix of pixel values corresponding to pixels of the image which correspond to locations of a matrix of light sensors of the camera," par. 72).
Nistico, Wang et al., and Hou do not explicitly teach all of to perform a Gaussian fitting to obtain at least one light spot region conforming to a Gaussian distribution; and to determine the third position of the reflection light spot according to a Gaussian expectation corresponding to the light spot region.
However, Stuart et al. teach to perform a Gaussian fitting to obtain at least one light spot region conforming to a Gaussian distribution; (" In these embodiments, and as an example, Gaussian distributions may be determined for a glint via determining a maxima intensity value within a bounding box surrounding an estimated glint," par. 163) and to determine the third position of the reflection light spot according to a Gaussian expectation corresponding to the light spot region ("The image location corresponding to the maxima intensity value may be assigned as the glint location. The estimated glint may be determined based on an increase in image intensity (e.g., brightness) as compared to another portion of the second image," par. 163).
Nistico, Wang et al., Hou, and Stuart et al. are combined as per claim 3.
Claim 17
Regarding Claim 17, Nistico, Wang et al., and Hou teach the electronic device according to claim 15, wherein determining the third position of the reflection light spot according to the first prediction diagram comprises: the first prediction diagram ("Each image includes a matrix of pixel values corresponding to pixels of the image which correspond to locations of a matrix of light sensors of the camera," par. 72).
Nistico, Wang et al., and Hou do not explicitly teach all of performing a Gaussian fitting to obtain at least one light spot region conforming to a Gaussian distribution; and determining the third position of the reflection light spot according to a Gaussian expectation corresponding to the light spot region.
However, Stuart et al. teach performing a Gaussian fitting to obtain at least one light spot region conforming to a Gaussian distribution; (" In these embodiments, and as an example, Gaussian distributions may be determined for a glint via determining a maxima intensity value within a bounding box surrounding an estimated glint," par. 163) and determining the third position of the reflection light spot according to a Gaussian expectation corresponding to the light spot region ("The image location corresponding to the maxima intensity value may be assigned as the glint location. The estimated glint may be determined based on an increase in image intensity (e.g., brightness) as compared to another portion of the second image," par. 163).
Nistico, Wang et al., Hou, and Stuart et al. are combined as per claim 3.
4th Claim Rejections - 35 USC § 103
Claims 5 and 12 are rejected under 35 U.S.C. 103 as obvious over US Patent Publication 2021 0068652 A1, (Nistico), US Patent Publication 2021 0165993 A1, (Wang et al.), and Chinese Patent Application 2022 114708232 A, (Hou) in view of US Patent Publication 2020 0394408 A1, (Sydorenko et al.).
Claim 5
Regarding Claim 5, Nistico, Wang et al., and Hou teach the method according to claim 4.
Nistico, Wang et al., and Hou do not explicitly teach all of wherein the second image is a second binary diagram with an identical image size as the first image; and obtaining the fourth image according to the first image and the second image comprises: obtaining an image size corresponding to the first image and the second image, and performing channel splicing based on the first image and the second image to obtain the fourth image.
However, Sydorenko et al. teach wherein the second image is a second binary diagram with an identical image size as the first image; ("The processor may configure, as a threshold, a maximum value (or a gray level of the darkest pixel in an image) among gray levels of pixels in the pupil region 691. The processor may convert and process a binary image generated with reference to the threshold (thresholded binary image), and thus may remove, from the thresholded binary image, a region smaller than the pupil region 691, par. 137 wherein the processor is converting all pixels therefore the images are the same size) and obtaining the fourth image according to the first image and the second image comprises: obtaining an image size corresponding to the first image and the second image, ("The processor may configure, as a threshold, a maximum value (or a gray level of the darkest pixel in an image) among gray levels of pixels in the pupil region 691, par. 137) and performing channel splicing based on the first image and the second image to obtain the fourth image ("The processor may configure, as a threshold, a maximum value (or a gray level of the darkest pixel in an image) among gray levels of pixels in the pupil region 691. The processor may convert and process a binary image generated with reference to the threshold (thresholded binary image), and thus may remove, from the thresholded binary image, a region smaller than the pupil region 691, par. 137).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the gaze direction determination method as taught by Nistico, the neural network training methods as taught by Wang et al., and the reflection intensity pattern as taught by Hou to use corneal region image extraction as taught by Sydorenko et al. The suggestion/motivation for doing so would have been that, “a method for detecting the pupil region 691 is not limited to the above description. The processor may extract the corneal region 695 by analyzing an ellipse including the center of the pupil 691. The corneal region 695 may be detected by an ellipse detector for finding ellipse parameters (c.sub.u, c.sub.v, r.sub.max, r.sub.min, φ) which maximize a response to an operator applied to an image I(u, v)” as noted by the Sydorenko et al. disclosure in paragraphs [0138 through 0140].
Claim 12
Regarding Claim 12, Nistico, Wang et al., and Hou teach the apparatus according to claim 11.
Nistico, Wang et al., and Hou do not explicitly teach all of wherein the second image is a second binary diagram with an identical image size as the first image; and the acquisition module is configured to: obtain an image size corresponding to the first image and the second image, and perform channel splicing based on the first image and the second image to obtain the fourth image.
However, Sydorenko et al. teach wherein the second image is a second binary diagram with an identical image size as the first image; ("The processor may configure, as a threshold, a maximum value (or a gray level of the darkest pixel in an image) among gray levels of pixels in the pupil region 691. The processor may convert and process a binary image generated with reference to the threshold (thresholded binary image), and thus may remove, from the thresholded binary image, a region smaller than the pupil region 691, par. 137 wherein the processor is converting all pixels therefore the images are the same size) and the acquisition module is configured to: obtain an image size corresponding to the first image and the second image, ("The processor may convert and process a binary image generated with reference to the threshold (thresholded binary image), and thus may remove, from the thresholded binary image, a region smaller than the pupil region 691, par. 137) and perform channel splicing based on the first image and the second image to obtain the fourth image ("The processor may configure, as a threshold, a maximum value (or a gray level of the darkest pixel in an image) among gray levels of pixels in the pupil region 691. The processor may convert and process a binary image generated with reference to the threshold (thresholded binary image), and thus may remove, from the thresholded binary image, a region smaller than the pupil region 691, par. 137).
Nistico, Wang et al., Hou, and Sydorenko et al. are combined as per claim 5.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Karsten F. Lantz whose telephone number is (571)272-4564. The examiner can normally be reached Monday-Friday 8:00-4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ms. Jennifer Mehmood can be reached on 571-272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Karsten F. Lantz/Examiner, Art Unit 2664
Date: 5/27/2026
/JENNIFER MEHMOOD/Supervisory Patent Examiner, Art Unit 2664
Prosecution Insights