DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after February 16, 2024, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged that application claims priority to foreign application with application number CN202310182114.1 dated 2/20/2023. Copies of certified papers required by 37 CFR 1.55 have been received. Priority is acknowledged under 35 USC 119(e) and 37 CFR 1.78.
Information Disclosure Statement
The IDS dated 5/14/2024 has been considered and placed in the application file.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 4, 6, 7, 8, 11, 13, 14, 15, 18, 19, and 20 are rejected under 35 U.S.C. 103 as obvious over US Patent Publication 2021 0068652 A1, (Nistico) in view of US Patent Publication 2021 0165993 A1, (Wang et al.).
Claim 1
Regarding Claim 1, Nistico teach a method for detecting a direction of sight, comprising: after emitting detection light to a cornea of a user, ("The one or more light sources 422 emit light onto the eye of the user 10 that reflects light (e.g., a directional beam) that can be detected by the sensor 424," par. 57) obtaining a first image by acquiring an image of the cornea, ("In some implementations, the one or more image sensor systems 314 are configured to obtain image data that corresponds to at least a portion of the face of the user that includes the eyes of the user," par. 46) wherein at least one reflection light spot is displayed in the first image, and the reflection light spot is formed by the cornea reflecting the detection light; ("The pixels associated with a reflection (e.g., glint) and/or the sensor's known position or orientation relative to the light source can be used to determine the direction (e.g., angle) the reflection," par. 90 wherein a "glint" is the reflection light spot) wherein the first prediction diagram characterizes a probability of a pixel on the first image being a target pixel, ("The pixels associated with a reflection (e.g., glint) and/or the sensor's known position or orientation relative to the light source can be used to determine the direction (e.g., angle) the reflection," par. 90) and the target pixel is a pixel constituting the reflection light spot; ("Each image includes a matrix of pixel values corresponding to pixels of the image which correspond to locations of a matrix of light sensors of the camera," par. 72) determining a target position of the reflection light spot according to the first prediction diagram; ("Each image includes a matrix of pixel values corresponding to pixels of the image which correspond to locations of a matrix of light sensors of the camera," par. 72) and determining a gazing direction of the user based on the target position of the reflection light spot ("Based on the reflected glint(s), the controller 480 can determine a gaze direction of the user 10," par. 57).
Nistico do not explicitly teach all of generating a first prediction diagram by processing the first image based on a pre-trained image processing model.
However, Wang et al. teach generating a first prediction diagram by processing the first image based on a pre-trained image processing model ("the method further includes a following operation: after the eye-area image is inputted into the neural network trained in advance and the line-of-sight direction of the eye-area image is outputted," par. 53).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the gaze direction determination method as taught by Nistico to use the neural network training methods as taught by Wang et al.
The suggestion/motivation for doing so would have been that, “the device for training the neural network not only may automatically obtain first line-of-sight directions but also may accurately obtain a large number of first line-of-sight directions. Therefore, a large amount of accurate and reliable data are provided to train the neural network, which increases an efficiency of the training and accuracy in predicting the line-of-sight directions” as noted by the Wang et al. disclosure in paragraph [0048].
The rejection system of claim 1 above applies mutatis mutandis to the corresponding limitations of apparatus claim 8 and electronic device claim 15 while noting that the rejection above cites to both device and method disclosures. Claims 8 and 15 are mapped below for clarity of the record and to specify any new limitations not included in claim 1.
Claim 4
Regarding Claim 4, Nistico teach obtaining a target image according to the first image and the second image, wherein the target image is a collection of pixels in the contour range of the first image, ("Each image includes a matrix of pixel values corresponding to pixels of the image which correspond to locations of a matrix of light sensors of the camera," par. 72) wherein generating the first prediction diagram by processing the first image based on the pre-trained image processing model, comprises: generating the first prediction diagram ("Each image includes a matrix of pixel values corresponding to pixels of the image which correspond to locations of a matrix of light sensors of the camera," par. 72).
Nistico do not explicitly teach all of obtaining a second image corresponding to the first image, wherein the second image is configured to characterize a contour range corresponding to the cornea in the first image; and generating the first prediction diagram by processing the target image according to the pre-trained image processing model.
However, Wang et al. teach obtaining a second image corresponding to the first image, wherein the second image is configured to characterize a contour range corresponding to the cornea in the first image; ("positions where images of light sources are formed on the cornea reference point, namely coordinates of reflection points in the second camera coordinate system, may be determined," par. 87) and generating the first prediction diagram by processing the target image according to the pre-trained image processing model ("the method further includes a following operation: after the eye-area image is inputted into the neural network trained in advance and the line-of-sight direction of the eye-area image is outputted," par. 53).
Nistico and Wang et al. are combined as per claim 1.
The rejection system of claim 4 above applies mutatis mutandis to the corresponding limitations of apparatus claim 11 while noting that the rejection above cites to both device and method disclosures. Claim 11 is mapped below for clarity of the record and to specify any new limitations not included in claim 4.
Claim 6
Regarding Claim 6, Nistico teach wherein determining the gazing direction of the user based on the target position of the reflection light spot comprises: determining a cornea center based on the target position of the reflection light spot; ("the method 600 determines a cornea center of the eye based on the first direction of the light beam and the second direction of the reflection. For example, when a glint is detected, the associated angle of the light source associated with the light that produced the glint may be identified and used to determine the glint location on the cornea surface," par. 91) detecting a pupil position of the user, and determining a pupil center based on a preset refraction angle; ("A gaze tracking system including light source 716 and camera 728 is used to track the gaze direction. The light source produces omnidirectional light that produces glint 722 by reflecting off the surface of the cornea 710. The camera 728 captures an image that includes a pupil image 730 and a glint image 732 … The pupil center 712 is also determined, for example, using image processing on the image data obtained by the camera 728 … FIG. 11 illustrates a functional block diagram illustrating gaze tracking using two variable angle light sources (e.g., a scanners 916a and 916b) at a known angles," par. 100, 103, and 104) and determining the gazing direction according to the cornea center and the pupil center ("the method 600 determines the gaze direction by determining a direction from the pupil center through the cornea center or a direction from the eyeball center through the cornea center," par. 93).
Nistico and Wang et al. are combined as per claim 1.
The rejection system of claim 6 above applies mutatis mutandis to the corresponding limitations of apparatus claim 13 and electronic device claim 18 while noting that the rejection above cites to both device and method disclosures. Claims 13 and 18 are mapped below for clarity of the record and to specify any new limitations not included in claim 6.
Claim 7
PNG
media_image1.png
536
486
media_image1.png
Greyscale
Regarding Claim 7, Nistico teach wherein before obtaining the first image by acquiring the image of the cornea, the method further comprises: performing image acquisition for an eye of the user to obtain a third image; ("In some implementations, the one or more image sensor systems 314 are configured to obtain image data that corresponds to at least a portion of the face of the user that includes the eyes of the user," par. 46) determining, based on the third image, a first positional relationship between the cornea of the user and a camera unit for obtaining the first image; (" FIG. 11 illustrates a functional block diagram illustrating gaze [AltContent: textbox (Figure 11 shows gaze-tracking using variable light sources at known angles.)]tracking using two variable angle light sources (e.g., a scanners 916a and 916b) at a known angles. In this example, a first glint produced by a light beam from a light source directed by scanner 916a forms a glint image 932a at the camera 728. The glint and angle of the scanner 916a are used to determine cornea point a 1210a. The second glint produced by a light beam from a light source directed by scanner 916b forms a glint image 932b at the camera," par. 104) determining a target light source according to light source matrix information and the first positional relationship, ("For example, the rows or columns of a Walsh matrix can be used as the orthogonal codes. Accordingly, in various implementations, a first light source of the plurality of light sources is modulated according to a first orthogonal code and a second light source of the plurality of light sources is modulated according to a second orthogonal code," par. 67) wherein the light source matrix information is configured to characterize a second positional relationship between the camera unit and at least two alternative light sources for emitting detection light; ("the one or more light sources 422 include or are coupled to a scanner that is configured to scan the light from the light source over multiple angles … When a glint, reflected by the eye and detected by the sensor 424, is analyzed, the identity of the glint and the corresponding light source angle (e.g., direction) can be determined, par. 62 and 64) and emitting detection light to the cornea of the user based on the target light source ("In various implementations, the one or more light sources 422 modulate the intensity of emitted light with different modulation frequencies," par. 66).
Nistico and Wang et al. are combined as per claim 1.
The rejection system of claim 7 above applies mutatis mutandis to the corresponding limitations of apparatus claim 14 while noting that the rejection above cites to both device and method disclosures. Claim 14 is mapped below for clarity of the record and to specify any new limitations not included in claim 7.
Claim 8
Regarding Claim 8, Nistico et al. teach an apparatus for detecting a direction of sight, comprising: an acquisition module, configured to, after emitting detection light to a cornea of a user, ("The one or more light sources 422 emit light onto the eye of the user 10 that reflects light (e.g., a directional beam) that can be detected by the sensor 424," par. 57) obtain a first image by acquiring an image of the cornea, ("In some implementations, the one or more image sensor systems 314 are configured to obtain image data that corresponds to at least a portion of the face of the user that includes the eyes of the user," par. 46) wherein at least one reflection light spot is displayed in the first image, and the reflection light spot is formed by the cornea reflecting the detection light; ("The pixels associated with a reflection (e.g., glint) and/or the sensor's known position or orientation relative to the light source can be used to determine the direction (e.g., angle) the reflection," par. 90 wherein a "glint" is the reflection light spot) a processing module, configured to generate a first prediction diagram, ("Each image includes a matrix of pixel values corresponding to pixels of the image which correspond to locations of a matrix of light sensors of the camera," par. 72) wherein the first prediction diagram characterizes a probability of a pixel on the first image being a target pixel, and the target pixel is a pixel constituting the reflection light spot; ("The pixels associated with a reflection (e.g., glint) and/or the sensor's known position or orientation relative to the light source can be used to determine the direction (e.g., angle) the reflection," par. 90) a first determination module, configured to determine a target position of the reflection light spot according to the first prediction diagram; ("Each image includes a matrix of pixel values corresponding to pixels of the image which correspond to locations of a matrix of light sensors of the camera," par. 72) and a second determination module, configured to determine a gazing direction of the user based on the target position of the reflection light spot ("Based on the reflected glint(s), the controller 480 can determine a gaze direction of the user 10," par. 57).
Nistico do not explicitly teach all of a processing module, configured to generate a first prediction diagram by processing the first image based on a pre-trained image processing model.
However, Wang et al. teach a processing module, configured to generate a first prediction diagram by processing the first image based on a pre-trained image processing model ("the method further includes a following operation: after the eye-area image is inputted into the neural network trained in advance and the line-of-sight direction of the eye-area image is outputted," par. 53).
Nistico and Wang et al. are combined as per claim 1.
Claim 11
Regarding Claim 11,. Nistico et al. teach wherein the acquisition module is further configured to: obtain a target image according to the first image and the second image, wherein the target image is a collection of pixels in the contour range of the first image; and the processing module is configured to: generate the first prediction diagram.
Nistico do not explicitly teach all of obtain a second image corresponding to the first image, wherein the second image is configured to characterize a contour range corresponding to the cornea in the first image, ("Each image includes a matrix of pixel values corresponding to pixels of the image which correspond to locations of a matrix of light sensors of the camera," par. 72) and the processing module is configured to: generate the first prediction diagram by processing the target image according to the pre-trained image processing model ("Each image includes a matrix of pixel values corresponding to pixels of the image which correspond to locations of a matrix of light sensors of the camera," par. 72).
However, Wang et al. teach obtain a second image corresponding to the first image, wherein the second image is configured to characterize a contour range corresponding to the cornea in the first image, ("positions where images of light sources are formed on the cornea reference point, namely coordinates of reflection points in the second camera coordinate system, may be determined," par. 87) and the processing module is configured to: generate the first prediction diagram by processing the target image according to the pre-trained image processing model ("the method further includes a following operation: after the eye-area image is inputted into the neural network trained in advance and the line-of-sight direction of the eye-area image is outputted," par. 53).
Nistico and Wang et al. are combined as per claim 4.
Claim 13
Regarding Claim 13,. Nistico et al. teach wherein the second determining module is configured to: determine a cornea center based on the target position of the reflection light spot; ("At block 630, the method 600 determines a cornea center of the eye based on the first direction of the light beam and the second direction of the reflection. For example, when a glint is detected, the associated angle of the light source associated with the light that produced the glint may be identified and used to determine the glint location on the cornea surface," par. 91) detect a pupil position of the user, and determining a pupil center based on a preset refraction angle; ("A gaze tracking system including light source 716 and camera 728 is used to track the gaze direction. The light source produces omnidirectional light that produces glint 722 by reflecting off the surface of the cornea 710. The camera 728 captures an image that includes a pupil image 730 and a glint image 732 … The pupil center 712 is also determined, for example, using image processing on the image data obtained by the camera 728 … FIG. 11 illustrates a functional block diagram illustrating gaze tracking using two variable angle light sources (e.g., a scanners 916a and 916b) at a known angles," par. 100, 103, and 104) and determine the gazing direction according to the cornea center and the pupil center ("At block 650, the method 600 determines the gaze direction by determining a direction from the pupil center through the cornea center or a direction from the eyeball center through the cornea center," par. 93).
Nistico and Wang et al. are combined as per claim 6.
Claim 14
Regarding Claim 14,. Nistico et al. teach wherein the acquisition module is further configured to: perform image acquisition for an eye of the user to obtain a third image; ("In some implementations, the one or more image sensor systems 314 are configured to obtain image data that corresponds to at least a portion of the face of the user that includes the eyes of the user," par. 46) determine, based on the third image, a first positional relationship between the cornea of the user and a camera unit for obtaining the first image; (" FIG. 11 illustrates a functional block diagram illustrating gaze tracking using two variable angle light sources (e.g., a scanners 916a and 916b) at a known angles. In this example, a first glint produced by a light beam from a light source directed by scanner 916a forms a glint image 932a at the camera 728. The glint and angle of the scanner 916a are used to determine cornea point a 1210a. The second glint produced by a light beam from a light source directed by scanner 916b forms a glint image 932b at the camera," par. 104) determine a target light source according to light source matrix information and the first positional relationship, ("For example, the rows or columns of a Walsh matrix can be used as the orthogonal codes. Accordingly, in various implementations, a first light source of the plurality of light sources is modulated according to a first orthogonal code and a second light source of the plurality of light sources is modulated according to a second orthogonal code," par. 67) wherein the light source matrix information is configured to characterize a second positional relationship between the camera unit and at least two alternative light sources for emitting detection light; ("the one or more light sources 422 include or are coupled to a scanner that is configured to scan the light from the light source over multiple angles … When a glint, reflected by the eye and detected by the sensor 424, is analyzed, the identity of the glint and the corresponding light source angle (e.g., direction) can be determined, par. 62 and 64) and emit detection light to the cornea of the user based on the target light source ("In various implementations, the one or more light sources 422 modulate the intensity of emitted light with different modulation frequencies," par. 66).
Nistico and Wang et al. are combined as per claim 7.
Claim 15
Regarding Claim 15, Nistico et al. teach an electronic device, comprising: a processor; ("the processor may execute instructions," par. 5) and a memory, ("a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors," par. 11) being in communication connection to the processor, wherein one or more computer-executable instructions are stored on the memory, and the processor is configured to execute the one or more computer-executable instructions stored on the memory to implement a method for detecting a direction of sight, ("the processor may execute instructions stored in a non-transitory computer-readable medium to determine or track a gaze direction," par. 5) which comprises: after emitting detection light to a cornea of a user, ("The one or more light sources 422 emit light onto the eye of the user 10 that reflects light (e.g., a directional beam) that can be detected by the sensor 424," par. 57) obtaining a first image by acquiring an image of the cornea, ("In some implementations, the one or more image sensor systems 314 are configured to obtain image data that corresponds to at least a portion of the face of the user that includes the eyes of the user," par. 46) wherein at least one reflection light spot is displayed in the first image, and the reflection light spot is formed by the cornea reflecting the detection light; ("The pixels associated with a reflection (e.g., glint) and/or the sensor's known position or orientation relative to the light source can be used to determine the direction (e.g., angle) the reflection," par. 90 wherein a "glint" is the reflection light spot) generating a first prediction diagram ("Each image includes a matrix of pixel values corresponding to pixels of the image which correspond to locations of a matrix of light sensors of the camera," par. 72) wherein the first prediction diagram characterizes a probability of a pixel on the first image being a target pixel, and the target pixel is a pixel constituting the reflection light spot; ("The pixels associated with a reflection (e.g., glint) and/or the sensor's known position or orientation relative to the light source can be used to determine the direction (e.g., angle) the reflection," par. 90) determining a target position of the reflection light spot according to the first prediction diagram; ("Each image includes a matrix of pixel values corresponding to pixels of the image which correspond to locations of a matrix of light sensors of the camera," par. 72) and determining a gazing direction of the user based on the target position of the reflection light spot ("Based on the reflected glint(s), the controller 480 can determine a gaze direction of the user 10," par. 57).
Nistico do not explicitly teach all of generating a first prediction diagram by processing the first image based on a pre-trained image processing model.
However, Wang et al. teach generating a first prediction diagram by processing the first image based on a pre-trained image processing model ("the method further includes a following operation: after the eye-area image is inputted into the neural network trained in advance and the line-of-sight direction of the eye-area image is outputted," par. 53).
Nistico and Wang et al. are combined as per claim 1.
Claim 18
Regarding Claim 8, Nistico et al. teach wherein determining the gazing direction of the user based on the target position of the reflection light spot comprises: determining a cornea center based on the target position of the reflection light spot; ("At block 630, the method 600 determines a cornea center of the eye based on the first direction of the light beam and the second direction of the reflection. For example, when a glint is detected, the associated angle of the light source associated with the light that produced the glint may be identified and used to determine the glint location on the cornea surface," par. 91) detecting a pupil position of the user, and determining a pupil center based on a preset refraction angle; ("A gaze tracking system including light source 716 and camera 728 is used to track the gaze direction. The light source produces omnidirectional light that produces glint 722 by reflecting off the surface of the cornea 710. The camera 728 captures an image that includes a pupil image 730 and a glint image 732 … The pupil center 712 is also determined, for example, using image processing on the image data obtained by the camera 728 … FIG. 11 illustrates a functional block diagram illustrating gaze tracking using two variable angle light sources (e.g., a scanners 916a and 916b) at a known angles," par. 100, 103, and 104) and determining the gazing direction according to the cornea center and the pupil center ("At block 650, the method 600 determines the gaze direction by determining a direction from the pupil center through the cornea center or a direction from the eyeball center through the cornea center," par. 93).
Nistico and Wang et al. are combined as per claim 6.
Claim 19
Regarding Claim 19, Nistico and Wang et al. teach claim 1.
Nistico teach a computer-readable storage medium, wherein the computer-readable storage medium is configured to store computer-executable instructions, and the computer- executable instructions, when executed by a processor, cause the processor to implement the method for detecting the direction of sight ("For example, the processor may execute instructions stored in a non-transitory computer-readable medium to determine or track a gaze direction," par. 5).
Nistico and Wang et al. are combined as per claim 1.
Claim 20
Regarding Claim 20, Nistico and Wang et al. teach claim 1.
Nistico teach a computer program product, comprising a computer program, wherein the computer program, when executed by a processor, is configured to implement the method for detecting the direction of sight ("Suitable computing devices include multipurpose microprocessor-based computer systems accessing stored software that programs or configures the computing system from a general purpose computing apparatus to a specialized computing apparatus implementing one or more implementations of the present subject matter, par. 109).
Nistico and Wang et al. are combined as per claim 1.
2nd Claim Rejections - 35 USC § 103
Claims 2, 9, and 16 are rejected under 35 U.S.C. 103 as obvious over US Patent Publication 2021 0068652 A1, (Nistico) and US Patent Publication 2021 0165993 A1, (Wang et al.) in view of US Patent Publication 2023 0329549 A1, (Tal et al.) and US Patent Publication 2020 0394408 A1, (Sydorenko et al.).
Claim 2
Regarding Claim 2, Nistico and Wang et al. teach the method according to claim 1, wherein determining the target position of the reflection light spot according to the first prediction diagram comprises: performing detection on the first prediction diagram according to the first pixel threshold to obtain a corresponding first binary diagram characterizing a position distribution of target pixels in the first image, wherein a pixel value of the target pixel is greater than the first pixel threshold; ("The voltage across the photodiode is proportional to the intensity of light impinging on the light sensor … The voltage across the first capacitor and the voltage across the second capacitor are fed to a comparator. When the difference between the voltage across the first capacitor and the voltage across the second capacitor is less than a threshold amount, the comparator outputs a ‘0’ voltage. When the voltage across the first capacitor is higher than the voltage across the second capacitor by at least the threshold amount, the comparator outputs a ‘1’ voltage," par. 75 and 77) according to the first binary diagram, ("The pixels associated with a reflection (e.g., glint) and/or the sensor's known position or orientation relative to the light source can be used to determine the direction (e.g., angle) the reflection," par. 90) and determining the target position of the reflection light spot according to a center point of the light spot region ("The processor may calculate a center of the remaining pixels except for the region removed from the thresholded binary image, and may determine that a corresponding central region is a center of the pupil 691," par. 137).
Nistico and Wang et al. do not explicitly teach all of obtaining a preset first pixel threshold; and performing connected component merging on the target pixels to obtain a light spot region.
However, Tal et al. teach obtaining a preset first pixel threshold ("In one example, thresholds (e.g., thresholds of pixel brightness of pixels, relationship to nearby pixels, etc.) are used to identify a spot and/or its size, position, and/or other detectable attributes," par. 29).
Sydorenko et al. teach performing connected component merging on the target pixels to obtain a light spot region ("The processor may configure, as a threshold, a maximum value (or a gray level of the darkest pixel in an image) among gray levels of pixels in the pupil region 691. The processor may convert and process a binary image generated with reference to the threshold (thresholded binary image), and thus may remove, from the thresholded binary image, a region smaller than the pupil region 691, par. 137 wherein the processor is converting all pixels so the images will be the same size).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the gaze direction determination method as taught by Nistico and use the neural network training methods as taught by Wang et al. to use the imaging-based eye detection system as taught by Tal et al. and corneal region image extraction as taught by Sydorenko et al.
The suggestion/motivation for doing so would have been that, “In some implementations, a spot is identified (e.g., its size, shaped, and/or position) and/or distinguished from other less-illuminated portions of the retina. A spot may be identified via an algorithm (e.g. based on a threshold) and/or using a machine learning (ML) model. A ML model may be used to assess/compare the size and/or position of the spot and/or a relationship between multiple spots” as noted by the Tal et al. disclosure in paragraph [0029].
The rejection system of claim 2 above applies mutatis mutandis to the corresponding limitations of apparatus claim 9 and electronic device claim 16 while noting that the rejection above cites to both device and method disclosures. Claims 9 and 16 are mapped below for clarity of the record and to specify any new limitations not included in claim 2.
Claim 9
Regarding Claim 9, Nistico and Wang et al. teach the apparatus according to claim 8, wherein the first determination module is configured to: perform detection on the first prediction diagram according to the first pixel threshold to obtain a corresponding first binary diagram characterizing a position distribution of target pixels in the first image, wherein a pixel value of the target pixel is greater than the first pixel threshold; ("The voltage across the photodiode is proportional to the intensity of light impinging on the light sensor … The voltage across the first capacitor and the voltage across the second capacitor are fed to a comparator. When the difference between the voltage across the first capacitor and the voltage across the second capacitor is less than a threshold amount, the comparator outputs a ‘0’ voltage. When the voltage across the first capacitor is higher than the voltage across the second capacitor by at least the threshold amount, the comparator outputs a ‘1’ voltage," par. 75 and 77), according to the first binary diagram, and determine the target position of the reflection light spot according to a center point of the light spot region ("The processor may calculate a center of the remaining pixels except for the region removed from the thresholded binary image, and may determine that a corresponding central region is a center of the pupil 691," par. 137).
Nistico and Wang et al. do not explicitly teach all of to obtain a preset first pixel threshold; and perform connected component merging on the target pixels to obtain a light spot region.
However, Tal et al. teach to obtain a preset first pixel threshold ("In one example, thresholds (e.g., thresholds of pixel brightness of pixels, relationship to nearby pixels, etc.) are used to identify a spot and/or its size, position, and/or other detectable attributes," par. 29).
Sydorenko et al. teach to perform connected component merging on the target pixels to obtain a light spot region ("The processor may configure, as a threshold, a maximum value (or a gray level of the darkest pixel in an image) among gray levels of pixels in the pupil region 691. The processor may convert and process a binary image generated with reference to the threshold (thresholded binary image), and thus may remove, from the thresholded binary image, a region smaller than the pupil region 691, par. 137 wherein the processor is converting all pixels so the images will be the same size).
Nistico, Wang et al., Tal et al., and Sydorenko et al. are combined as per claim 2.
Claim 16
Regarding Claim 16, Nistico and Wang et al. teach the electronic device according to claim 15, wherein determining the target position of the reflection light spot according to the first prediction diagram comprises: performing detection on the first prediction diagram according to the first pixel threshold to obtain a corresponding first binary diagram characterizing a position distribution of target pixels in the first image, wherein a pixel value of the target pixel is greater than the first pixel threshold; ("The voltage across the photodiode is proportional to the intensity of light impinging on the light sensor … The voltage across the first capacitor and the voltage across the second capacitor are fed to a comparator. When the difference between the voltage across the first capacitor and the voltage across the second capacitor is less than a threshold amount, the comparator outputs a ‘0’ voltage. When the voltage across the first capacitor is higher than the voltage across the second capacitor by at least the threshold amount, the comparator outputs a ‘1’ voltage," par. 75 and 77) according to the first binary diagram, ("The pixels associated with a reflection (e.g., glint) and/or the sensor's known position or orientation relative to the light source can be used to determine the direction (e.g., angle) the reflection," par. 90) and determining the target position of the reflection light spot according to a center point of the light spot region ("The processor may calculate a center of the remaining pixels except for the region removed from the thresholded binary image, and may determine that a corresponding central region is a center of the pupil 691," par. 137).
Nistico and Wang et al. do not explicitly teach all of obtaining a preset first pixel threshold; and performing connected component merging on the target pixels to obtain a light spot region.
However, Tal et al. teach obtaining a preset first pixel threshold ("In one example, thresholds (e.g., thresholds of pixel brightness of pixels, relationship to nearby pixels, etc.) are used to identify a spot and/or its size, position, and/or other detectable attributes," par. 29).
Sydorenko et al. teach performing connected component merging on the target pixels to obtain a light spot region ("The processor may configure, as a threshold, a maximum value (or a gray level of the darkest pixel in an image) among gray levels of pixels in the pupil region 691. The processor may convert and process a binary image generated with reference to the threshold (thresholded binary image), and thus may remove, from the thresholded binary image, a region smaller than the pupil region 691, par. 137 wherein the processor is converting all pixels so the images will be the same size).
Nistico, Wang et al., Tal et al., and Sydorenko et al. are combined as per claim 2.
3rd Claim Rejections - 35 USC § 103
Claims 3, 10, and 17 are rejected under 35 U.S.C. 103 as obvious over US Patent Publication 2021 0068652 A1, (Nistico) and US Patent Publication 2021 0165993 A1, (Wang et al.) in view of US Patent Publication 2022 0301217 A1, (Stuart et al.).
Claim 3
Regarding Claim 3, Nistico and Wang et al. teach the method according to claim 1, wherein determining the target position of the reflection light spot according to the first prediction diagram comprises: the first prediction diagram ("Each image includes a matrix of pixel values corresponding to pixels of the image which correspond to locations of a matrix of light sensors of the camera," par. 72).
Nistico and Wang et al. do not explicitly teach all of performing a Gaussian fitting to obtain at least one light spot region conforming to a Gaussian distribution; and determining the target position of the reflection light spot according to a Gaussian expectation corresponding to the light spot region.
However, Stuart et al. teach performing a Gaussian fitting to obtain at least one light spot region conforming to a Gaussian distribution; (" In these embodiments, and as an example, Gaussian distributions may be determined for a glint via determining a maxima intensity value within a bounding box surrounding an estimated glint," par. 163) and determining the target position of the reflection light spot according to a Gaussian expectation corresponding to the light spot region ("The image location corresponding to the maxima intensity value may be assigned as the glint location. The estimated glint may be determined based on an increase in image intensity (e.g., brightness) as compared to another portion of the second image," par. 163).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the gaze direction determination method as taught by Nistico and use the neural network training methods as taught by Wang et al. to use the glint location identification methods as taught by Stuart et al.
The suggestion/motivation for doing so would have been that, “shape regression has become the state-of-the-art approach for accurate and efficient shape alignment. It has been successfully used in face, hand and ear shape estimation. Regression techniques are advantageous because, for example, they are capable of capturing large variances in appearance; they enforce shape constraint between landmarks (e.g., iris between eyelids, pupil inside iris); and they are computationally efficient. While regression techniques are described, it may be appreciated that neural networks may be employed as an alternative to and/or in combination with regression techniques” as noted by the Stuart et al. disclosure in paragraph [0029].
The rejection system of claim 3 above applies mutatis mutandis to the corresponding limitations of apparatus claim 10 and electronic device claim 17 while noting that the rejection above cites to both device and method disclosures. Claims 10 and 17 are mapped below for clarity of the record and to specify any new limitations not included in claim 3.
Claim 10
Regarding Claim 10, Nistico and Wang et al. teach the apparatus according to claim 8, wherein the first determination module is configured to: the first prediction diagram ("Each image includes a matrix of pixel values corresponding to pixels of the image which correspond to locations of a matrix of light sensors of the camera," par. 72).
Nistico and Wang et al. do not explicitly teach all of to perform a Gaussian fitting to obtain at least one light spot region conforming to a Gaussian distribution; and to determine the target position of the reflection light spot according to a Gaussian expectation corresponding to the light spot region.
However, Stuart et al. teach to perform a Gaussian fitting to obtain at least one light spot region conforming to a Gaussian distribution; (" In these embodiments, and as an example, Gaussian distributions may be determined for a glint via determining a maxima intensity value within a bounding box surrounding an estimated glint," par. 163) and to determine the target position of the reflection light spot according to a Gaussian expectation corresponding to the light spot region ("The image location corresponding to the maxima intensity value may be assigned as the glint location. The estimated glint may be determined based on an increase in image intensity (e.g., brightness) as compared to another portion of the second image," par. 163).
Nistico, Wang et al., and Stuart et al. are combined as per claim 3.
Claim 17
Regarding Claim 17, Nistico and Wang et al. teach the electronic device according to claim 15, wherein determining the target position of the reflection light spot according to the first prediction diagram comprises: the first prediction diagram ("Each image includes a matrix of pixel values corresponding to pixels of the image which correspond to locations of a matrix of light sensors of the camera," par. 72).
Nistico and Wang et al. do not explicitly teach all of performing a Gaussian fitting to obtain at least one light spot region conforming to a Gaussian distribution; and determining the target position of the reflection light spot according to a Gaussian expectation corresponding to the light spot region.
However, Stuart et al. teach performing a Gaussian fitting to obtain at least one light spot region conforming to a Gaussian distribution; (" In these embodiments, and as an example, Gaussian distributions may be determined for a glint via determining a maxima intensity value within a bounding box surrounding an estimated glint," par. 163) and determining the target position of the reflection light spot according to a Gaussian expectation corresponding to the light spot region ("The image location corresponding to the maxima intensity value may be assigned as the glint location. The estimated glint may be determined based on an increase in image intensity (e.g., brightness) as compared to another portion of the second image," par. 163).
Nistico, Wang et al., and Stuart et al. are combined as per claim 3.
4th Claim Rejections - 35 USC § 103
Claims 5 and 12 are rejected under 35 U.S.C. 103 as obvious over US Patent Publication 2021 0068652 A1, (Nistico) and US Patent Publication 2021 0165993 A1, (Wang et al.) in view of US Patent Publication 2020 0394408 A1, (Sydorenko et al.).
Claim 5
Regarding Claim 5, Nistico and Wang et al. teach the method according to claim 4.
Nistico and Wang et al. do not explicitly teach all of wherein the second image is a second binary diagram with an identical image size as the first image; and obtaining the target image according to the first image and the second image comprises: obtaining an image size corresponding to the first image and the second image, and performing channel splicing based on the first image and the second image to obtain the target image.
However, Sydorenko et al. teach wherein the second image is a second binary diagram with an identical image size as the first image; ("The processor may configure, as a threshold, a maximum value (or a gray level of the darkest pixel in an image) among gray levels of pixels in the pupil region 691. The processor may convert and process a binary image generated with reference to the threshold (thresholded binary image), and thus may remove, from the thresholded binary image, a region smaller than the pupil region 691, par. 137 wherein the processor is converting all pixels therefore the images are the same size) and obtaining the target image according to the first image and the second image comprises: obtaining an image size corresponding to the first image and the second image, ("The processor may configure, as a threshold, a maximum value (or a gray level of the darkest pixel in an image) among gray levels of pixels in the pupil region 691, par. 137) and performing channel splicing based on the first image and the second image to obtain the target image ("The processor may configure, as a threshold, a maximum value (or a gray level of the darkest pixel in an image) among gray levels of pixels in the pupil region 691. The processor may convert and process a binary image generated with reference to the threshold (thresholded binary image), and thus may remove, from the thresholded binary image, a region smaller than the pupil region 691, par. 137).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the gaze direction determination method as taught by Nistico and use the neural network training methods as taught by Wang et al. to use corneal region image extraction as taught by Sydorenko et al.
The suggestion/motivation for doing so would have been that, “a method for detecting the pupil region 691 is not limited to the above description. The processor may extract the corneal region 695 by analyzing an ellipse including the center of the pupil 691. The corneal region 695 may be detected by an ellipse detector for finding ellipse parameters (c.sub.u, c.sub.v, r.sub.max, r.sub.min, φ) which maximize a response to an operator applied to an image I(u, v)” as noted by the Sydorenko et al. disclosure in paragraphs [0138 through 0140].
The rejection system of claim 5 above applies mutatis mutandis to the corresponding limitations of apparatus claim 12 while noting that the rejection above cites to both device and method disclosures. Claim 12 is mapped below for clarity of the record and to specify any new limitations not included in claim 5.
Claim 12
Regarding Claim 12, Nistico and Wang et al. teach the apparatus according to claim 11.
Nistico and Wang et al. do not explicitly teach all of wherein the second image is a second binary diagram with an identical image size as the first image; and the acquisition module is configured to: obtain an image size corresponding to the first image and the second image, and perform channel splicing based on the first image and the second image to obtain the target image.
However, Sydorenko et al. teach wherein the second image is a second binary diagram with an identical image size as the first image; ("The processor may configure, as a threshold, a maximum value (or a gray level of the darkest pixel in an image) among gray levels of pixels in the pupil region 691. The processor may convert and process a binary image generated with reference to the threshold (thresholded binary image), and thus may remove, from the thresholded binary image, a region smaller than the pupil region 691, par. 137 wherein the processor is converting all pixels therefore the images are the same size) and the acquisition module is configured to: obtain an image size corresponding to the first image and the second image, ("The processor may convert and process a binary image generated with reference to the threshold (thresholded binary image), and thus may remove, from the thresholded binary image, a region smaller than the pupil region 691, par. 137) and perform channel splicing based on the first image and the second image to obtain the target image ("The processor may configure, as a threshold, a maximum value (or a gray level of the darkest pixel in an image) among gray levels of pixels in the pupil region 691. The processor may convert and process a binary image generated with reference to the threshold (thresholded binary image), and thus may remove, from the thresholded binary image, a region smaller than the pupil region 691, par. 137).
Nistico, Wang et al., and Sydorenko et al. are combined as per claim 5.
Reference Cited
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
US Patent Publication 2018 0164880 A1 to Kim et al. discloses using converging reflected light rays to generate pattern data and compute an estimated gaze direction.
US Patent Publication 2022 0350153 A1 to Takashima discloses digital illumination assisted gaze tracking.
.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KARSTEN LANTZ whose telephone number is 571-272-4564. The examiner can normally be reached Monday-Friday 8:00-5:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ms. Jennifer Mehmood can be reached on 571-272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/K.F.L./Examiner, Art Unit 2664
Date: 1/5/2026
/JENNIFER MEHMOOD/Supervisory Patent Examiner, Art Unit 2664