DETAILED ACTION
Election to Restriction Requirement
In response to the Restriction Requirement through phone conversation between Applicant’s Representative: Sanjesh Sharma (Reg. No. 65054), on 7/22/2026, and Applicant elects without traverse Group II (claims 8-13, 21-26) for examination, and other non-elected claims are in Group I (1-7, and 14-20).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 8, and 10-13 are rejected under 35 U.S.C. 103 as being unpatentable over RUBENS (US 20240144722 A1), and in view of LEE (KR 20120049605 A), and further in view of DU (US 8768014 B2).
Re Claim 8, RUBENS discloses a method for detecting a pupil of an eye (see RUBENS: e.g.,-- Methods and systems are described for estimating an optimal pupil binarization threshold using adaptive binarization thresholding. The disclosed methods and systems are designed to determine a pupil binarization threshold from a histogram of an eye image. An eye image is obtained and an eye image histogram is computed from the eye image. A pupil region and an iris region are identified in the eye image histogram and the second derivative of the eye image histogram is computed. The pupil binarization threshold is determined based on the second derivative of the eye image histogram, the identified pupil region and the identified iris region and then used to generate a binarized eye image. A pupil contour may be determined from the binarized eye image. The disclosed system may help to overcome challenges associated with robust and precise detection of a pupil contour, for example, in images of varying quality.--, in abstract), comprising:
obtaining a color image of the eye, the color image being represented in an RGB color space (see RUBENS: e.g., --a camera is used to capture eye images. Cameras may be infrared (IR) cameras (which capture IR spectrum data) or RGB cameras (which capture visible spectrum data). Automatic pupil detection or segmentation in captured eye images--, in [0003], [0066], and [0073]);
although RUBENS discloses processing the eye image in gray values (see RUBENS: e.g., -- the eye image 210 may be a grayscale eye image in which each pixel of the grayscale eye image may have a corresponding gray value.--, in [0005], [0049], and,
-- the eye image 210 may be a grayscale eye image in which each pixel of the grayscale eye image may have a corresponding gray value.--, in [0059], [0061], and [0074]; and,
-- 0076] FIG. 3 is an example schematic diagram of an eye region 300, including a dark pupil 301, suitable for implementation of examples described herein. In examples, a pupil 301 may exhibit the characteristic of absorbing light very efficiently, therefore a pupil 301 may present as a dark prominent feature in an eye image 210. Similarly, a pupil 301 may also present as a dark prominent feature in an eye image histogram 230 in one or more color spaces.--, in [0076]);
RUBENS however does not explicitly disclose converting the color image into an HSV color space representation having a hue channel, a saturation channel, and a value channel, to obtain a hue channel image;
LEE discloses converting the color image into an HSV color space representation having a hue channel, a saturation channel, and a value channel, to obtain a hue channel image (see LEE: e.g., -- Converts the color model of the specified search region to the Hue value of the HSV. The hue value is less affected by lighting, but the specified search area gray histogram is obtained except when it is very dark or bright enough to make the object difficult to identify. Therefore, the object is tracked based on this information.--, in the last para. of page 3/11 through page 4/11 of English version of KR-20120049605-A, as provided in the Office Action);
RUBENS and LEE are combinable as they are in the same field of endeavor: image processing and detecting pupil, the boundary and the center of the pupil from eye image. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify RUBENS’s method using LEE’s teachings by including converting the color image into an HSV color space representation having a hue channel, a saturation channel, and a value channel, to obtain a hue channel image to RUBENS’s segmentation in order to improve segmentation and identification of regions (see LEE: e.g. in the last para. of page 3/11 through page 4/11 of English version of KR-20120049605-A, as provided in the Office Action);
And although RUBENS as modified by LEE disclose filtering the eye image {to detect pupil, the boundary of the pupil, and the center of the pupil based on filtered eye image (see RUBENS: e.g., -- applying an incremental high-pass filter to refine the upper boundary of the threshold search area improves the robustness to anomalous features between pupil and iris histographic features.--, in [0010]; and, -- the method further comprises: prior to identifying a pupil region in the eye image histogram: processing the eye image histogram using an incremental high-pass filter… the pupil region in the eye image histogram includes an upper pupil region boundary and a lower pupil region boundary and the iris region in the eye image histogram includes an upper iris region boundary and a lower iris region boundary.
… wherein the eye image is a grayscale eye image and the histogram represents a frequency distribution of pixel gray values in the grayscale eye image. --, in [0019]-[0021]; and, -- [0080] FIG. 5A is a block diagram illustrating an example architecture of an eye feature extractor 240, in accordance with examples of the present disclosure. In examples, a filter block 510 of the eye feature extractor 240 may receive the eye image histogram 230 and may output an interpolated eye image histogram 520. In examples, the filter block 510 may serve to remove noise or other undesirable values from the eye image histogram, or to more effectively expose eye features in the histogram, for example, the pupil region 405 or the iris region 410, among other benefits. In examples, the filter block 510 may also include a linear interpolation operation to help smooth the eye image histogram 230. An example interpolated eye image histogram 520 is presented in FIG. 5B, in accordance with examples of the present disclosure. In examples, a histogram features block 530 may determine upper and lower boundaries for both the pupil region 405 and the iris region 410, where the boundaries correspond to a gray value in the interpolated eye image histogram 520. In examples, the boundaries of the pupil region 405 and the iris region 410 may be used be used by the histogram features block 530 to define a pupil-iris window 540 that spans a gray value corresponding to the pupil region 405 and the iris region 410 in the interpolated eye image histogram 520.--, in [0080]-[0081]);
RUBENS as modified by LEE however still do not explicitly disclose filtering the hue channel image to obtain a filtered hue channel image;
DU discloses filtering the hue channel image to obtain a filtered hue channel image (see DU: e.g., -- eye image are converted from an initial red, green, blue (RGB) color space, to an intermediate luma, red-difference, blue-difference (YCrCb) color space, and then into a hue, saturation, brightness value (HSV) color space (block 312) using transformations listed below.
(17) .function..times..times..times..times..times. ##EQU00001##
(18) In Equation 1, the red (R), green (G), and blue (B) numeric values corresponding to each pixel undergo a cross product matrix transform with the weighting matrix of equation 1. The numeric RGB values in equation 1 are in a range of 0 to 255, but larger or smaller ranges using modified matrix coefficients are possible for alternative binary image formats. The resulting matrix is then adjusted by adding 128 to each of the Cr and Cb values, resulting in a YCrCb matrix. The YCrCb matrix is then transformed into an HSV matrix using the transformations listed in equation 2. The transformations listed above are repeated for each pixel in the eye image, producing an image in the HSV color space. Both the original RGB and the transformed HSV color space values are used in different steps of the segmentation process of FIG. 3.--, in col. 3 through col. 4; and, -- (31) Returning to the process of FIG. 2, the segmented sclera image contains features such as vein patterns, vein lines, and extrema points that are enhanced to improve feature detection (block 212). The surface of an eye is often highly reflective, which makes focusing of an imaging device photographing the eye difficult. The result is that features within the sclera region may be blurred, and have a low contrast ratio in the image. In order to enhance feature patterns, a Gabor filtering technique may be used to differentiate the features from the surrounding sclera. The Gabor filtering process applies the Gabor filter of equation 11 using the transformation of equation 12 on the segmented sclera image.--, in lines 18-67, col. 6);
RUBENS (as modified by LEE) and DU are combinable as they are in the same field of endeavor: image processing and segmentation eye image. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify RUBENS’s method using DU’s teachings by including filtering the hue channel image to obtain a filtered hue channel image to RUBENS (as modified by LEE)’s segmentation in order to improve segmentation and identification of regions (see DU: e.g. in col. 3-4, and in lines 18-67, col. 6);
RUBENS as modified by LEE and DU further disclose detecting a pupil boundary and a pupil center of the eye based on the filtered hue channel image (see RUBENS: e.g., -- applying an incremental high-pass filter to refine the upper boundary of the threshold search area improves the robustness to anomalous features between pupil and iris histographic features.--, in [0010]; and, -- the method further comprises: prior to identifying a pupil region in the eye image histogram: processing the eye image histogram using an incremental high-pass filter… the pupil region in the eye image histogram includes an upper pupil region boundary and a lower pupil region boundary and the iris region in the eye image histogram includes an upper iris region boundary and a lower iris region boundary.
… wherein the eye image is a grayscale eye image and the histogram represents a frequency distribution of pixel gray values in the grayscale eye image. --, in [0019]-[0021]; and, -- [0080] FIG. 5A is a block diagram illustrating an example architecture of an eye feature extractor 240, in accordance with examples of the present disclosure. In examples, a filter block 510 of the eye feature extractor 240 may receive the eye image histogram 230 and may output an interpolated eye image histogram 520. In examples, the filter block 510 may serve to remove noise or other undesirable values from the eye image histogram, or to more effectively expose eye features in the histogram, for example, the pupil region 405 or the iris region 410, among other benefits. In examples, the filter block 510 may also include a linear interpolation operation to help smooth the eye image histogram 230. An example interpolated eye image histogram 520 is presented in FIG. 5B, in accordance with examples of the present disclosure. In examples, a histogram features block 530 may determine upper and lower boundaries for both the pupil region 405 and the iris region 410, where the boundaries correspond to a gray value in the interpolated eye image histogram 520. In examples, the boundaries of the pupil region 405 and the iris region 410 may be used be used by the histogram features block 530 to define a pupil-iris window 540 that spans a gray value corresponding to the pupil region 405 and the iris region 410 in the interpolated eye image histogram 520.--, in [0080]-[0081]; and, --the binarized eye image 270 may be input to an optional pupil geometry estimator 280 to generate a pupil contour 290 and a pupil center 295. In examples, a variety of methods may be used to estimate a pupil contour 290 and a pupil center 295 from the binarized eye image 270. FIG. 2 illustrates the pupil geometry estimator 280 as being external to the pupil segmentation system 200. In other examples, the pupil geometry estimator 280 may be part of the pupil segmentation system 200.
[0087] FIG. 6A illustrates an example embodiment of the pupil segmentation system 200, in accordance with examples of the present disclosure. In the example embodiment, an artifact binarization threshold 255 may also be generated by the eye feature extractor 240 and input to the binarization block 260. In some examples, the eye image 210 may include areas of the image that appear to be darker than the pupil 301, for example, when the eye image 210 includes pigments associated with eye makeup 305 worn by an individual, or due to insufficient lighting conditions, among others. In examples, dark areas of an image may be visible as a dark artifact region 605 feature adjacent to a pupil region 405 in a corresponding interpolated eye image histogram 520. In this regard, dark artifacts that remain visible in a binarized eye image 270 may interfere with pupil contour detection or other processing of the binarized eye image 270.--, in [0086]-[0087]; and,
also see DU: e.g., --FIG. 6. A fixed central reference point 604, usually the center of the pupil 606, is used as a central point from which radial lines such as line 620 extend. The use of a known central point allows for correct relative positions of vein features to be recorded even if the eye moves to a different position relative to the imaging device. Each radial line 620 intersects the center point of a linear element 628. The linear element 628 approximates a non-linear section of a vein feature 624. The linear element 628 is described by the polar coordinate 618 of its center point, its length, and the angle 616 of the linear element 628 from a horizontal line 622 (o). The polar coordinate of the center of linear element 628 is the radius of radial line 620 and the angle 624 of the radial line (.theta.) from a horizontal line 622. By storing the identifying elements for each linear element 628, the vein features present in the sclera may be extracted and stored in a computer readable manner. The coordinates (x.sub.p, y.sub.p) locating the pupil's center 604 are also stored, providing an absolute position from which each linear element 628 may be referenced by polar coordinates.--, in lines 36-63, col. 7).
Re Claim 10, RUBENS as modified by LEE and DU further disclose wherein the filtering step includes median filtering the hue channel image (see RUBENS: e.g., -- applying an incremental high-pass filter to refine the upper boundary of the threshold search area improves the robustness to anomalous features between pupil and iris histographic features.--, in [0010]; and, -- the method further comprises: prior to identifying a pupil region in the eye image histogram: processing the eye image histogram using an incremental high-pass filter… the pupil region in the eye image histogram includes an upper pupil region boundary and a lower pupil region boundary and the iris region in the eye image histogram includes an upper iris region boundary and a lower iris region boundary.
… wherein the eye image is a grayscale eye image and the histogram represents a frequency distribution of pixel gray values in the grayscale eye image. --, in [0019]-[0021]; and, -- [0080] FIG. 5A is a block diagram illustrating an example architecture of an eye feature extractor 240, in accordance with examples of the present disclosure. In examples, a filter block 510 of the eye feature extractor 240 may receive the eye image histogram 230 and may output an interpolated eye image histogram 520. In examples, the filter block 510 may serve to remove noise or other undesirable values from the eye image histogram, or to more effectively expose eye features in the histogram, for example, the pupil region 405 or the iris region 410, among other benefits. In examples, the filter block 510 may also include a linear interpolation operation to help smooth the eye image histogram 230. An example interpolated eye image histogram 520 is presented in FIG. 5B, in accordance with examples of the present disclosure. In examples, a histogram features block 530 may determine upper and lower boundaries for both the pupil region 405 and the iris region 410, where the boundaries correspond to a gray value in the interpolated eye image histogram 520. In examples, the boundaries of the pupil region 405 and the iris region 410 may be used be used by the histogram features block 530 to define a pupil-iris window 540 that spans a gray value corresponding to the pupil region 405 and the iris region 410 in the interpolated eye image histogram 520.--, in [0080]-[0081]; and, also see DU: e.g., -- eye image are converted from an initial red, green, blue (RGB) color space, to an intermediate luma, red-difference, blue-difference (YCrCb) color space, and then into a hue, saturation, brightness value (HSV) color space (block 312) using transformations listed below.
(17) .function..times..times..times..times..times. ##EQU00001##
(18) In Equation 1, the red (R), green (G), and blue (B) numeric values corresponding to each pixel undergo a cross product matrix transform with the weighting matrix of equation 1. The numeric RGB values in equation 1 are in a range of 0 to 255, but larger or smaller ranges using modified matrix coefficients are possible for alternative binary image formats. The resulting matrix is then adjusted by adding 128 to each of the Cr and Cb values, resulting in a YCrCb matrix. The YCrCb matrix is then transformed into an HSV matrix using the transformations listed in equation 2. The transformations listed above are repeated for each pixel in the eye image, producing an image in the HSV color space. Both the original RGB and the transformed HSV color space values are used in different steps of the segmentation process of FIG. 3.--, in col. 3 through col. 4; and, -- (31) Returning to the process of FIG. 2, the segmented sclera image contains features such as vein patterns, vein lines, and extrema points that are enhanced to improve feature detection (block 212). The surface of an eye is often highly reflective, which makes focusing of an imaging device photographing the eye difficult. The result is that features within the sclera region may be blurred, and have a low contrast ratio in the image. In order to enhance feature patterns, a Gabor filtering technique may be used to differentiate the features from the surrounding sclera. The Gabor filtering process applies the Gabor filter of equation 11 using the transformation of equation 12 on the segmented sclera image.--, in lines 18-67, col. 6).
Re Claim 11, RUBENS as modified by LEE and DU further disclose wherein the filtering step further includes guided image filtering the hue channel image after the median filtering (see RUBENS: e.g., -- applying an incremental high-pass filter to refine the upper boundary of the threshold search area improves the robustness to anomalous features between pupil and iris histographic features.--, in [0010]; and, -- the method further comprises: prior to identifying a pupil region in the eye image histogram: processing the eye image histogram using an incremental high-pass filter… the pupil region in the eye image histogram includes an upper pupil region boundary and a lower pupil region boundary and the iris region in the eye image histogram includes an upper iris region boundary and a lower iris region boundary.
… wherein the eye image is a grayscale eye image and the histogram represents a frequency distribution of pixel gray values in the grayscale eye image. --, in [0019]-[0021]; and, -- [0080] FIG. 5A is a block diagram illustrating an example architecture of an eye feature extractor 240, in accordance with examples of the present disclosure. In examples, a filter block 510 of the eye feature extractor 240 may receive the eye image histogram 230 and may output an interpolated eye image histogram 520. In examples, the filter block 510 may serve to remove noise or other undesirable values from the eye image histogram, or to more effectively expose eye features in the histogram, for example, the pupil region 405 or the iris region 410, among other benefits. In examples, the filter block 510 may also include a linear interpolation operation to help smooth the eye image histogram 230. An example interpolated eye image histogram 520 is presented in FIG. 5B, in accordance with examples of the present disclosure. In examples, a histogram features block 530 may determine upper and lower boundaries for both the pupil region 405 and the iris region 410, where the boundaries correspond to a gray value in the interpolated eye image histogram 520. In examples, the boundaries of the pupil region 405 and the iris region 410 may be used be used by the histogram features block 530 to define a pupil-iris window 540 that spans a gray value corresponding to the pupil region 405 and the iris region 410 in the interpolated eye image histogram 520.--, in [0080]-[0081]; and, also see DU: e.g., -- eye image are converted from an initial red, green, blue (RGB) color space, to an intermediate luma, red-difference, blue-difference (YCrCb) color space, and then into a hue, saturation, brightness value (HSV) color space (block 312) using transformations listed below.
(17) .function..times..times..times..times..times. ##EQU00001##
(18) In Equation 1, the red (R), green (G), and blue (B) numeric values corresponding to each pixel undergo a cross product matrix transform with the weighting matrix of equation 1. The numeric RGB values in equation 1 are in a range of 0 to 255, but larger or smaller ranges using modified matrix coefficients are possible for alternative binary image formats. The resulting matrix is then adjusted by adding 128 to each of the Cr and Cb values, resulting in a YCrCb matrix. The YCrCb matrix is then transformed into an HSV matrix using the transformations listed in equation 2. The transformations listed above are repeated for each pixel in the eye image, producing an image in the HSV color space. Both the original RGB and the transformed HSV color space values are used in different steps of the segmentation process of FIG. 3.--, in col. 3 through col. 4; and, -- (31) Returning to the process of FIG. 2, the segmented sclera image contains features such as vein patterns, vein lines, and extrema points that are enhanced to improve feature detection (block 212). The surface of an eye is often highly reflective, which makes focusing of an imaging device photographing the eye difficult. The result is that features within the sclera region may be blurred, and have a low contrast ratio in the image. In order to enhance feature patterns, a Gabor filtering technique may be used to differentiate the features from the surrounding sclera. The Gabor filtering process applies the Gabor filter of equation 11 using the transformation of equation 12 on the segmented sclera image.--, in lines 18-67, col. 6).
Re Claim 12, RUBENS as modified by LEE and DU further disclose wherein the step of detecting the pupil boundary and the pupil center of the eye includes:
binarizing the filtered hue channel image to obtain a binary image; determining a boundary of an image region formed by black pixels that are connected to each other and located in a center region of the binary image; and determining a centroid of the image region (see RUBENS: e.g.,-- Methods and systems are described for estimating an optimal pupil binarization threshold using adaptive binarization thresholding. The disclosed methods and systems are designed to determine a pupil binarization threshold from a histogram of an eye image. An eye image is obtained and an eye image histogram is computed from the eye image. A pupil region and an iris region are identified in the eye image histogram and the second derivative of the eye image histogram is computed. The pupil binarization threshold is determined based on the second derivative of the eye image histogram, the identified pupil region and the identified iris region and then used to generate a binarized eye image. A pupil contour may be determined from the binarized eye image. The disclosed system may help to overcome challenges associated with robust and precise detection of a pupil contour, for example, in images of varying quality.--, in abstract; and,
also see: --the binarized eye image 270 may be input to an optional pupil geometry estimator 280 to generate a pupil contour 290 and a pupil center 295. In examples, a variety of methods may be used to estimate a pupil contour 290 and a pupil center 295 from the binarized eye image 270. FIG. 2 illustrates the pupil geometry estimator 280 as being external to the pupil segmentation system 200. In other examples, the pupil geometry estimator 280 may be part of the pupil segmentation system 200.
[0087] FIG. 6A illustrates an example embodiment of the pupil segmentation system 200, in accordance with examples of the present disclosure. In the example embodiment, an artifact binarization threshold 255 may also be generated by the eye feature extractor 240 and input to the binarization block 260. In some examples, the eye image 210 may include areas of the image that appear to be darker than the pupil 301, for example, when the eye image 210 includes pigments associated with eye makeup 305 worn by an individual, or due to insufficient lighting conditions, among others. In examples, dark areas of an image may be visible as a dark artifact region 605 feature adjacent to a pupil region 405 in a corresponding interpolated eye image histogram 520. In this regard, dark artifacts that remain visible in a binarized eye image 270 may interfere with pupil contour detection or other processing of the binarized eye image 270.--, in [0086]-[0087]).
Re Claim 13, RUBENS as modified by LEE and DU further disclose wherein the step of determining the pupil boundary and the pupil center of the eye includes:
binarizing the filtered hue channel image to obtain a binary image (see RUBENS: e.g.,-- Methods and systems are described for estimating an optimal pupil binarization threshold using adaptive binarization thresholding. The disclosed methods and systems are designed to determine a pupil binarization threshold from a histogram of an eye image. An eye image is obtained and an eye image histogram is computed from the eye image. A pupil region and an iris region are identified in the eye image histogram and the second derivative of the eye image histogram is computed. The pupil binarization threshold is determined based on the second derivative of the eye image histogram, the identified pupil region and the identified iris region and then used to generate a binarized eye image. A pupil contour may be determined from the binarized eye image. The disclosed system may help to overcome challenges associated with robust and precise detection of a pupil contour, for example, in images of varying quality.--, in abstract; and,
also see: --the binarized eye image 270 may be input to an optional pupil geometry estimator 280 to generate a pupil contour 290 and a pupil center 295. In examples, a variety of methods may be used to estimate a pupil contour 290 and a pupil center 295 from the binarized eye image 270. FIG. 2 illustrates the pupil geometry estimator 280 as being external to the pupil segmentation system 200. In other examples, the pupil geometry estimator 280 may be part of the pupil segmentation system 200.
[0087] FIG. 6A illustrates an example embodiment of the pupil segmentation system 200, in accordance with examples of the present disclosure. In the example embodiment, an artifact binarization threshold 255 may also be generated by the eye feature extractor 240 and input to the binarization block 260. In some examples, the eye image 210 may include areas of the image that appear to be darker than the pupil 301, for example, when the eye image 210 includes pigments associated with eye makeup 305 worn by an individual, or due to insufficient lighting conditions, among others. In examples, dark areas of an image may be visible as a dark artifact region 605 feature adjacent to a pupil region 405 in a corresponding interpolated eye image histogram 520. In this regard, dark artifacts that remain visible in a binarized eye image 270 may interfere with pupil contour detection or other processing of the binarized eye image 270.--, in [0086]-[0087]);
fitting an image region formed by black pixels that are connected to each other and located in a center region of the image to a circle or an ellipse; and determining a center of the circle or the ellipse (see --[0085] Returning to FIG. 2, the binarization threshold 250 may be input to a binarization block 260 to generate a binarized eye image 270. In examples, the binarization threshold 250 may be applied to the pixels of an eye image 210 to generate a binarized eye image 270 in which the gray value associated with each pixel is compared to the binarization threshold 250 and where the gray value of those pixels situated below the binarization threshold value may be set equal to zero (e.g. black) and the gray value of the remaining pixels situated above the binarization threshold value may be set equal to 1 (e.g. white). In this regard, the binarized image 270 may present a clearly exposed pupil where the shape of the pupil may be preserved very well across a variety of image conditions.
[0086] In some embodiments, for example, the binarized eye image 270 may be input to an optional pupil geometry estimator 280 to generate a pupil contour 290 and a pupil center 295. In examples, a variety of methods may be used to estimate a pupil contour 290 and a pupil center 295 from the binarized eye image 270. FIG. 2 illustrates the pupil geometry estimator 280 as being external to the pupil segmentation system 200. In other examples, the pupil geometry estimator 280 may be part of the pupil segmentation system 200.
[0087] FIG. 6A illustrates an example embodiment of the pupil segmentation system 200, in accordance with examples of the present disclosure. In the example embodiment, an artifact binarization threshold 255 may also be generated by the eye feature extractor 240 and input to the binarization block 260. In some examples, the eye image 210 may include areas of the image that appear to be darker than the pupil 301, for example, when the eye image 210 includes pigments associated with eye makeup 305 worn by an individual, or due to insufficient lighting conditions, among others. In examples, dark areas of an image may be visible as a dark artifact region 605 feature adjacent to a pupil region 405 in a corresponding interpolated eye image histogram 520. In this regard, dark artifacts that remain visible in a binarized eye image 270 may interfere with pupil contour detection or other processing of the binarized eye image 270.--, in [0085]-[0087]; and, see Lee: e.g., --A candidate area detection unit(110) detects a candidate area for detection of a pupil center from an eye image by selective application of an AdaBoost algorithm and a CAMShift algorithm. A pupil center detection unit(120) applies a circle detection algorithm to the candidate area. The pupil center detection unit detects the pupil center by application of a local binarization process, a labeling process, and a centroid computation process. --,in abstract; and, --Referring to FIG. 4, the pupil center detection unit 120 according to an embodiment of the present invention may have a region of an iris size in an eye image 410 of a user 140 based on a center coordinate value of the detected candidate region. The operation of canceling the reflected light can be performed.
The image 420 represents an image from which the reflected light is canceled from the eye image 410.
In addition, the pupil center detector 120 may detect the initial pupil center 431 by applying a circular detection algorithm to the detected candidate region of the image 420 from which the reflected light is erased. In addition, the pupil center detector 120 may detect a radius of the pupil. In addition, the pupil center detector 120 may detect a gray histogram of a region having a predetermined size based on the pupil center.
The image 430 represents an image in which the initial pupil center 431 is detected by applying a circular detection algorithm.
The shape of the pupil has an irregular shape to simplify to the shape of a circle, and can be observed in the form of an ellipse according to the position of the pupil relative to the position of the camera. In addition, when the reflected light of the illumination is located at the boundary of the pupil, the center position of the pupil may not be extracted accurately.
Accordingly, the pupil center detection unit 120 sequentially detects the pupil center of the user 140 by sequentially applying a local binarization process, a labeling process, and a weight center calculation process to the detected initial pupil center 431.--, in page 5/11 of English version of KR-20120049605-A, as provided in the Office Action).
Claims 9, and 21-26 are rejected under 35 U.S.C. 103 as being unpatentable over RUBENS as modified by LEE and DU, and further in view of HOMER (US 20220304850 A1).
Re Claim 9, RUBENS as modified by LEE and DU further disclose wherein the step of obtaining the color image of the eye includes: the patient interface device applanates a surface area of the eye (see RUBENS: e.g., --a camera is used to capture eye images. Cameras may be infrared (IR) cameras (which capture IR spectrum data) or RGB cameras (which capture visible spectrum data). Automatic pupil detection or segmentation in captured eye images--, in [0003], [0066], and [0073]); and
capturing the color image of the eye through the patient interface using a camera (see RUBENS: e.g., --a camera is used to capture eye images. Cameras may be infrared (IR) cameras (which capture IR spectrum data) or RGB cameras (which capture visible spectrum data). Automatic pupil detection or segmentation in captured eye images--, in [0003], [0066], and [0073]);
RUBENS as modified by LEE and DU however do not explicitly disclose the camera coupling the eye to an ophthalmic laser system by a patient interface device
HOMER discloses camera coupling the eye to an ophthalmic laser system by a patient interface device, and the camera captures the eye image (see HOMER: e.g., -- imaging the iris with an image sensor prior to the color alteration procedure to generate an image of the iris. A mapping of the iris may be generated from the image. The mapping may include a number of regions corresponding to varying absorption coefficients of a treatment wavelength in the stromal pigment of the iris. A laser system may be set, based on the mapping, to deliver laser light at a laser power sufficient to cause elimination of at least a portion of stromal pigment in the iris. The laser light may then be delivered with the laser system.--, in abstract, and, -- This mapping allows determination of spatially varying absorption coefficients (of the laser light that is for treatment) in the iris. Also, to ensure that the laser light is accurately delivered to all regions of the eye needed for treatment, a scanning pattern for the delivery of the laser light is determined. Optical tracking of the eye is done during the procedure to ensure that laser light is delivered according to the scanning pattern. Another aspect that improves accurate delivery of laser light is monitoring the iris for unacceptable changes in tilt (e.g., due to patient motion).--, in [0006], and, -- [0041] Galvos systems 216 (also referred to as the x-y beam guidance system) may be included in the laser system and may include adjustable mirrors to provide a means of delivering the laser light to various locations on an X-Y plane (typically the plane of the iris where the laser light usually focused). Further implementations of the laser system may include, for example rangefinders and/or optical tracking systems, which may include cameras to determine an X-Y deviation of the center of the eye relative to the optical axis of the laser system.--, in [0041], [0068]; and, -- [0121] FIG. 3 shows a simplified diagram of a laser system 210 and image sensor 310 for use in mapping the iris in accordance with one or more embodiments. Determination of the proper laser power may depend on variations in the absorption of the delivered laser power due to inhomogeneities in regions 330, 332, 334 of the stromal pigment layer. Such variations may be caused by, for example, varying density of the stromal pigment, varying sizes of stromal pigment cells, types and compositions of the stromal pigment, etc. As such, regions of the iris where the stromal pigment has a higher absorption coefficient reach a higher temperature (or a target temperature faster) for a given laser power. These differences, if not accounted for, may result in uneven color alteration or possibly even damage to the eye. To address this problem, some implementations of the disclosed methods may include imaging the iris with an image sensor operatively connected to a computer 312 prior to the procedure to generate images of the iris. Examples of image sensors may include a CCD, COMS, or camera used in conjunction with an illumination source 320, wherein the wavelength range of the sensor includes the wavelength of the illumination source. Exemplary wavelengths include near and mid-infrared, visible light, or the specific wavelength of the treatment laser beam. An embodiment might also include software programs capable of creating a digital color model from the captured images and mapping or otherwise analyzing the stromal pigment coefficients for the treatment wavelength based on the model. Exemplary digital color models include RGB (which stands for red-green-blue), HSI (for hue-saturation-intensity), HSL (for hue-saturation-lightness), HSV (for hue-saturation-value), CMY (for cyan-magenta-yellow), and YIQ (luminance-inphase-quadrature).
[0122] To facilitate integration of the image sensor with existing laser system, the image sensor may incorporate a dichroic optic 314 (e.g., a dichroic lens, mirror, or prism) to divert incoming light reflected from the iris the reflective or refractive side of the optic and directing it to the image sensor, while allowing outgoing laser light to pass through the optic to the iris surface for treatment. Such implementations have the advantage that the light may be collected on the same optical axis as the laser system. This has the advantage of both simplifying and making more accurate the generation of the mapping relative to the geometry of laser system because it avoids the need to account for an off-axis image sensor.
[0123] Based on the images, a mapping of the iris may be generated by the system and may contain regions corresponding to varying absorption coefficients of a treatment wavelength in the stromal pigment of the iris.--, in [0121]-[0123]);
RUBENS (as modified by LEE and DU) and HOMER are combinable as they are in the same field of endeavor: image processing and segmentation eye image. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify RUBENS’s method using HOMER’s teachings by including camera coupling the eye to an ophthalmic laser system by a patient interface device, and the camera captures the eye image to RUBENS (as modified by LEE and DU)’s eye image capturing and segmentations in order to add the usage and system that detection of pupil, pupil boundary, center of pupil and other near eye regions, such as making more accurate the generation of the mapping relative to the geometry of laser system (see HOMER: e.g. in [0041], [0068]; and [0121]-[0123]);
Re Claims 21-26, claims 21-26 are corresponding system claim to claims 8-13, respectively. Claims 21-26 thus are rejected for the similar reasons for claims 8-13. See above discussions with regard to claims 8-13 respectively. Furthermore, RUBENS as modified by LEE and DU and HOMER further disclose n ophthalmic laser system for treating a patient's eye, comprising: a laser source configured to generate a pulsed laser beam; an optical delivery system coupled to the laser source, configured to receive and direct the pulsed laser beam; a camera coupled to the optical delivery system, configured to capture a color image of the eye, the color image being represented in an RGB color space (see HOMER: e.g., -- imaging the iris with an image sensor prior to the color alteration procedure to generate an image of the iris. A mapping of the iris may be generated from the image. The mapping may include a number of regions corresponding to varying absorption coefficients of a treatment wavelength in the stromal pigment of the iris. A laser system may be set, based on the mapping, to deliver laser light at a laser power sufficient to cause elimination of at least a portion of stromal pigment in the iris. The laser light may then be delivered with the laser system.--, in abstract, and, -- This mapping allows determination of spatially varying absorption coefficients (of the laser light that is for treatment) in the iris. Also, to ensure that the laser light is accurately delivered to all regions of the eye needed for treatment, a scanning pattern for the delivery of the laser light is determined. Optical tracking of the eye is done during the procedure to ensure that laser light is delivered according to the scanning pattern. Another aspect that improves accurate delivery of laser light is monitoring the iris for unacceptable changes in tilt (e.g., due to patient motion).--, in [0006], and, -- [0041] Galvos systems 216 (also referred to as the x-y beam guidance system) may be included in the laser system and may include adjustable mirrors to provide a means of delivering the laser light to various locations on an X-Y plane (typically the plane of the iris where the laser light usually focused). Further implementations of the laser system may include, for example rangefinders and/or optical tracking systems, which may include cameras to determine an X-Y deviation of the center of the eye relative to the optical axis of the laser system.--, in [0041], [0068]; and, -- [0121] FIG. 3 shows a simplified diagram of a laser system 210 and image sensor 310 for use in mapping the iris in accordance with one or more embodiments. Determination of the proper laser power may depend on variations in the absorption of the delivered laser power due to inhomogeneities in regions 330, 332, 334 of the stromal pigment layer. Such variations may be caused by, for example, varying density of the stromal pigment, varying sizes of stromal pigment cells, types and compositions of the stromal pigment, etc. As such, regions of the iris where the stromal pigment has a higher absorption coefficient reach a higher temperature (or a target temperature faster) for a given laser power. These differences, if not accounted for, may result in uneven color alteration or possibly even damage to the eye. To address this problem, some implementations of the disclosed methods may include imaging the iris with an image sensor operatively connected to a computer 312 prior to the procedure to generate images of the iris. Examples of image sensors may include a CCD, COMS, or camera used in conjunction with an illumination source 320, wherein the wavelength range of the sensor includes the wavelength of the illumination source. Exemplary wavelengths include near and mid-infrared, visible light, or the specific wavelength of the treatment laser beam. An embodiment might also include software programs capable of creating a digital color model from the captured images and mapping or otherwise analyzing the stromal pigment coefficients for the treatment wavelength based on the model. Exemplary digital color models include RGB (which stands for red-green-blue), HSI (for hue-saturation-intensity), HSL (for hue-saturation-lightness), HSV (for hue-saturation-value), CMY (for cyan-magenta-yellow), and YIQ (luminance-inphase-quadrature).
[0122] To facilitate integration of the image sensor with existing laser system, the image sensor may incorporate a dichroic optic 314 (e.g., a dichroic lens, mirror, or prism) to divert incoming light reflected from the iris the reflective or refractive side of the optic and directing it to the image sensor, while allowing outgoing laser light to pass through the optic to the iris surface for treatment. Such implementations have the advantage that the light may be collected on the same optical axis as the laser system. This has the advantage of both simplifying and making more accurate the generation of the mapping relative to the geometry of laser system because it avoids the need to account for an off-axis image sensor.
[0123] Based on the images, a mapping of the iris may be generated by the system and may contain regions corresponding to varying absorption coefficients of a treatment wavelength in the stromal pigment of the iris.--, in [0121]-[0123]); and
a processor coupled to the laser source, the optical delivery system, and the camera, configured to perform a process to detect a pupil of the eye (see HOMER: e.g., -- imaging the iris with an image sensor prior to the color alteration procedure to generate an image of the iris. A mapping of the iris may be generated from the image. The mapping may include a number of regions corresponding to varying absorption coefficients of a treatment wavelength in the stromal pigment of the iris. A laser system may be set, based on the mapping, to deliver laser light at a laser power sufficient to cause elimination of at least a portion of stromal pigment in the iris. The laser light may then be delivered with the laser system.--, in abstract, and, -- This mapping allows determination of spatially varying absorption coefficients (of the laser light that is for treatment) in the iris. Also, to ensure that the laser light is accurately delivered to all regions of the eye needed for treatment, a scanning pattern for the delivery of the laser light is determined. Optical tracking of the eye is done during the procedure to ensure that laser light is delivered according to the scanning pattern. Another aspect that improves accurate delivery of laser light is monitoring the iris for unacceptable changes in tilt (e.g., due to patient motion).--, in [0006], and, -- [0041] Galvos systems 216 (also referred to as the x-y beam guidance system) may be included in the laser system and may include adjustable mirrors to provide a means of delivering the laser light to various locations on an X-Y plane (typically the plane of the iris where the laser light usually focused). Further implementations of the laser system may include, for example rangefinders and/or optical tracking systems, which may include cameras to determine an X-Y deviation of the center of the eye relative to the optical axis of the laser system.--, in [0041], [0068]; and, -- [0121] FIG. 3 shows a simplified diagram of a laser system 210 and image sensor 310 for use in mapping the iris in accordance with one or more embodiments. Determination of the proper laser power may depend on variations in the absorption of the delivered laser power due to inhomogeneities in regions 330, 332, 334 of the stromal pigment layer. Such variations may be caused by, for example, varying density of the stromal pigment, varying sizes of stromal pigment cells, types and compositions of the stromal pigment, etc. As such, regions of the iris where the stromal pigment has a higher absorption coefficient reach a higher temperature (or a target temperature faster) for a given laser power. These differences, if not accounted for, may result in uneven color alteration or possibly even damage to the eye. To address this problem, some implementations of the disclosed methods may include imaging the iris with an image sensor operatively connected to a computer 312 prior to the procedure to generate images of the iris. Examples of image sensors may include a CCD, COMS, or camera used in conjunction with an illumination source 320, wherein the wavelength range of the sensor includes the wavelength of the illumination source. Exemplary wavelengths include near and mid-infrared, visible light, or the specific wavelength of the treatment laser beam. An embodiment might also include software programs capable of creating a digital color model from the captured images and mapping or otherwise analyzing the stromal pigment coefficients for the treatment wavelength based on the model. Exemplary digital color models include RGB (which stands for red-green-blue), HSI (for hue-saturation-intensity), HSL (for hue-saturation-lightness), HSV (for hue-saturation-value), CMY (for cyan-magenta-yellow), and YIQ (luminance-inphase-quadrature).
[0122] To facilitate integration of the image sensor with existing laser system, the image sensor may incorporate a dichroic optic 314 (e.g., a dichroic lens, mirror, or prism) to divert incoming light reflected from the iris the reflective or refractive side of the optic and directing it to the image sensor, while allowing outgoing laser light to pass through the optic to the iris surface for treatment. Such implementations have the advantage that the light may be collected on the same optical axis as the laser system. This has the advantage of both simplifying and making more accurate the generation of the mapping relative to the geometry of laser system because it avoids the need to account for an off-axis image sensor.
[0123] Based on the images, a mapping of the iris may be generated by the system and may contain regions corresponding to varying absorption coefficients of a treatment wavelength in the stromal pigment of the iris.--, in [0121]-[0123]), and the process including the method (see RUBENS: e.g.,-- Methods and systems are described for estimating an optimal pupil binarization threshold using adaptive binarization thresholding. The disclosed methods and systems are designed to determine a pupil binarization threshold from a histogram of an eye image. An eye image is obtained and an eye image histogram is computed from the eye image. A pupil region and an iris region are identified in the eye image histogram and the second derivative of the eye image histogram is computed. The pupil binarization threshold is determined based on the second derivative of the eye image histogram, the identified pupil region and the identified iris region and then used to generate a binarized eye image. A pupil contour may be determined from the binarized eye image. The disclosed system may help to overcome challenges associated with robust and precise detection of a pupil contour, for example, in images of varying quality.--, in abstract).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEIWEN YANG whose telephone number is (571)270-5670. The examiner can normally be reached on Monday-Friday 8:30am-4:30pm east.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on 571-272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEI WEN YANG/Primary Examiner, Art Unit 2662
Prosecution Insights