DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 8-9 and 19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Regarding claim 8, applicant claims “…a plurality of images of second pupils….” It is unclear if there are a plurality of second pupils (e.g. multiple people or a history of second pupil images from one person) or if there is a plurality of images of a second pupil. That is, “plurality” is used to describe multiple images and then “pupils” suggests several different pupils.
For the purposes of this action the office will interpret the claim such that there is a plurality of images of a second pupil.
Claim 9 depends from claim 8 and has identical issues.
Regarding claim 19, the applicant claims “…haptic feedback comprises outputting a vibration using at least one haptic actuator, the vibration to indicate to the subject to move their head towards an instructed direction…” It is unclear how haptic feedback on its own could indicate an “instructed direction” as claimed.
For the purposes of this action the office will interpret the claim such that the haptic feedback is not required to indicate any specific direction.
Allowable Subject Matter
Claim 13 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-3, 7, 12, 14-16, 18 and 20-21 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Khan et al. (PGPUB 20220151489, of record).
Regarding claim 1, Khan discloses a method for triggering fundus imaging, comprising:
detecting a pupil of a subject in an image ([0094] and/or [0142]);
controlling at least one actuator to align an imaging path of a fundus camera with the pupil after detecting the pupil in the image ([0132]); and
capturing a fundus image of a retina of the subject at a gaze angle after determining that the pupil is oriented towards an intended fixation target direction based on the gaze angle associated with the pupil in the image ([0132]-[0133]).
Regarding claim 2, Khan discloses wherein the image is a stereo image, and the method further comprising (Fig. 16A shows a pair of cameras, one for each eye, which would result in left and right images):
capturing a first image from a first camera and a second image from a second camera; processing the first image and the second image to form the stereo image (Fig. 16A shows a pair of cameras, 1400); and
detecting the pupil in the first image and the second image to detect the pupil of the subject in the stereo image ([0142]).
Regarding claim 3, Khan discloses wherein the image comprises a first frame and a second frame, and the method further comprising performing at least one of edge detection or thresholding to determine whether the pupil is detected in at least one of the first frame or the second frame (Fig. 14A where there are two cameras, which gives two image frames and [0142] where the pupil is detected).
Regarding claim 7, Khan discloses further comprising performing one or more pre-processing operations on the image, and wherein the one or more pre-processing operations comprise at least one of removing glare from a portion of the image, correcting illumination on the portion of the image, or cropping the portion of the image ([0168] glare removal).
Regarding claim 12, Khan discloses further comprising:
measuring a first value of a dimension of the pupil ([0132], pupil location determination must be done in order to provide the fundus camera location information);
determining at least one first coordinate of the pupil in a coordinate system associated with the fundus camera after determining that the first value satisfies a threshold ([0132], any data that is used to move an object to a particular position necessarily defines a coordinate system); and
determining a correction to at least one second coordinate of the fundus camera after determining that the at least one first coordinate is not associated with a three-dimensional target zone for the pupil ([0132], automatic positioning would necessarily require a first position and a second position and a fundus is a 3D object); and
wherein the controlling of the at least one actuator is based on the correction to the at least one second coordinate ([0132]).
Regarding claim 14, Khan discloses further comprising:
determining that the pupil is not oriented towards the target direction ([0129]);
changing the intended fixation target direction towards a direction in which the pupil is gazing ([0129] where a correction is applied to the gaze angle data and results in changing the intended fixation target location by way of said correction); and
capturing the fundus image of the retina after the changing in the intended fixation target direction ([0129]).
Regarding claim 15, Khan discloses wherein capturing the fundus image of the retina is in response to detecting that an eyelid of the subject reopened (Acquiring a fundus image using a visible light camera requires that the eye be open. Further, all the processes in Khan for detecting a pupil require that the eye be open. As such, an open eye is necessarily the first step in both processes).
Regarding claim 16, Khan discloses wherein the at least one actuator comprises a first motor, a second motor, and a third motor, and the method further comprising at least one of (Fig. 16A and [0161]-[0162] where there may be a total of four motors that include moving the left and right fundus cameras):
controlling the first motor to move the fundus camera in a first direction; controlling the second motor to move the fundus camera in a second direction orthogonal to the first direction; or controlling the third motor to move the fundus camera orthogonal to at least one of the first direction or the second direction (Fig. 16A and [0161]-[0162]).
Regarding claim 18, Khan discloses wherein the image comprises a first image and a second image, and the method further comprising:
determining the pupil is not in at least one of the first image or the second image; and outputting feedback to the subject to cause the subject to move such that the pupil is detected in at least one of the first image or the second image ([0120] determination of whether a pupil is visible, such as when the eye is closed, is a clear requirement for the invention to Khan to function and would necessarily require communication to the user).
Regarding claim 20, Khan discloses at least one non-transitory computer-readable storage medium comprising instructions that, when executed, cause at least one processor to perform a method for triggering fundus imaging (600), comprising:
detecting a pupil of a subject in an image ([0094] and/or [0142]); controlling at least one actuator to align an imaging path of a fundus camera with the pupil after detecting the pupil in the image ([0132]); and capturing a fundus image of a retina of the subject at a gaze angle after determining that the pupil is oriented towards an intended fixation target direction based on the gaze angle associated with the pupil in the image ([0132]-[0133]).
Regarding claim 21, Khan discloses a fundus camera system comprising a three-dimensional visualization system, at least one memory storing machine-readable instructions, and at least one processor configured to execute the machine-readable instructions to perform at least a method for triggering fundus imaging (600), comprising:
detecting a pupil of a subject in an image ([0094] and/or [0142]); controlling at least one actuator to align an imaging path of a fundus camera with the pupil after detecting the pupil in the image ([0132]); and capturing a fundus image of a retina of the subject at a gaze angle after determining that the pupil is oriented towards an intended fixation target direction based on the gaze angle associated with the pupil in the image ([0132]-[0133]).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 4 and 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Khan in view of Arar et al. (PGPUB 20210182609, of record).
Regarding claim 4, Khan discloses the use of machine learning ([0154]), but does not disclose wherein the image comprises a first frame and a second frame, and the method further comprising:
executing a first machine-learning model using the first frame as at least one first input to generate at least one first output, the at least one first output representative of whether the pupil is detected in the first frame; and
executing a second machine-learning model, substantially in parallel with the executing of the first machine-learning model, using the second frame as at least one second input to generate at least one second output, the at least one second output representative of whether the pupil is detected in the second frame.
However, Arar teaches a method of tracking and imaging a user’s eyes ([0023]) wherein the method includes executing a first machine-learning model using the first frame as at least one first input to generate at least one first output, the at least one first output representative of whether the pupil is detected in the first frame ([0092]); and
executing a second machine-learning model, substantially in parallel with the executing of the first machine-learning model, using the second frame as at least one second input to generate at least one second output, the at least one second output representative of whether the pupil is detected in the second frame ([0113]-[0114] where multiple neural networks may work in parallel).
It would have been obvious to one having ordinary skill in the art as of the effective filing date of the invention to combine Khan and Arar such that machine learning was used to generate a tracking output from the left and right eye images motivated by improving system adaptability.
Regarding claim 10, Khan discloses determining a gaze angle with respect to a fundus camera ([0019]), but does not disclose wherein the image comprises a first frame, and the method further comprising:
executing a machine-learning model using the first frame as at least one input to generate at least one output, the at least one output representative of a determination of the gaze angle, the gaze angle being an angle with respect to the pupil and the fundus camera.
However, Arar teaches a method of tracking and imaging a user’s eyes ([0023]) wherein the image comprises a first frame, and the method further comprising:
executing a machine-learning model using the first frame as at least one input to generate at least one output, the at least one output representative of a determination of the gaze angle, the gaze angle being an angle with respect to the pupil and a target ([0034]).
It would have been obvious to one having ordinary skill in the art as of the effective filing date of the invention to combine Khan and Arar such that machine learning was used to determine a gaze angle motivated by improving system adaptability.
Claims 5-6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Khan in view of Rougeaux et al. (PGPUB 20190156100, of record).
Regarding claim 5, Khan does not disclose further comprising executing a machine-learning model using at least part of the image as at least one input to generate at least one output, the at least one output representative of whether the pupil is detected in the image.
However, Rougeaux teaches a gaze tracking method comprising executing a machine-learning model ([0087]) using at least part of the image as at least one input to generate at least one output, the at least one output representative of whether the pupil is detected in the image (Fig. 6 and [0094]).
It would have been obvious to one having ordinary skill in the art as of the effective filing date of the invention to combine Khan and Rougeaux such that a machine learning algorithm was used to output pupil detection motivated by improved accuracy ([0089]).
Regarding claim 6, modified Khan teaches wherein the executing of the machine-learning model comprises performing at least one of (i) concentric ellipse detection to detect at least one of an iris of the subject or the pupil in the image or (ii) performing line detection to detect an eyelid in the image, and the detection of the eyelid to be representative of whether an eye of the subject is closed ([0098] of Rougeaux).
Claim 8-9, 11 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Khan in view of Qin et al. (PGPUB 20210004619, of record).
Regarding claim 8, as best understood, Khan does not disclose wherein the pupil is a first pupil, and the method further comprising:
obtaining a plurality of images of second pupils, the plurality of images labeled with metadata; training a machine-learning model using the plurality of images and the metadata;
compiling the machine-learning model into at least one of an executable file, machine-readable instructions, or a configuration image after determining that an accuracy of the machine-learning model satisfies a threshold; and
at least one of executing the executable file, executing the machine-readable instructions, or instantiating the configuration image to detect the first pupil of the subject in the image.
However, Qin teaches a method of gaze detection ([0086]) wherein the pupil is a first pupil, and the method further comprising:
obtaining a plurality of images of second pupils ([0106]), the plurality of images labeled with metadata; training a machine-learning model using the plurality of images and the metadata ([0079]-[0080]);
compiling the machine-learning model into at least one of an executable file, machine-readable instructions, or a configuration image after determining that an accuracy of the machine-learning model satisfies a threshold ([0079]-[0080]); and
at least one of executing the executable file, executing the machine-readable instructions, or instantiating the configuration image to detect the first pupil of the subject in the image ([0079]-[0080]).
It would have been obvious to one having ordinary skill in the art as of the effective filing date of the invention to combine Khan and Qin such that machine learning was used in conjunction with metadata motivated by improved accuracy ([0080]).
Regarding claim 9, modified Khan discloses wherein the metadata is representative of at least one of an indication whether the second pupils are detected in respective ones of the plurality of images, a degree to which corresponding eyes of the second pupils are open or closed, or a gaze angle of respective ones of the second pupils ([0079]-[0080] of Qin).
Regarding claim 11, modified Khan does not disclose further comprising:
measuring a first value of a dimension of the pupil; and generating a command to cause the subject to increase the dimension of the pupil after determining that the first value does not satisfy a threshold, the command comprising at least one of audible, tactile, or visual feedback to the subject.
However, Qin teaches a method of gaze detection ([0086]) comprising measuring a first value of a dimension of the pupil ([0077]); and generating a command to cause the subject to increase the dimension of the pupil after determining that the first value does not satisfy a threshold, the command comprising at least one of audible, tactile, or visual feedback to the subject ([0077] and [0007], alarm).
It would have been obvious to one having ordinary skill in the art as of the effective filing date of the invention to combine Khan and Qin such that an eye open/closed stated was measured and alert the user to open their eyes further motivated by improving image quality.
Regarding claim 19, Khan does not disclose wherein the feedback is at least one of audio feedback, haptic feedback, or visual feedback to the subject, wherein the audio feedback comprises outputting audio using at least one speaker, the audio comprising audible instructions for the subject to move their head towards an instructed direction, wherein the haptic feedback comprises outputting a vibration using at least one haptic actuator, the vibration to indicate to the subject to move their head towards an instructed direction, and wherein the visual feedback comprises projecting an image using at least one display device, the image comprising instructions in natural language text for the subject to move their head towards an instructed direction.
However, Qin teaches a method of gaze detection ([0086]) comprising the feedback is at least one of audio feedback, haptic feedback, or visual feedback to the subject, wherein the audio feedback comprises outputting audio using at least one speaker, the audio comprising audible instructions for the subject to move their head towards an instructed direction, wherein the haptic feedback comprises outputting a vibration using at least one haptic actuator, the vibration to indicate to the subject to move their head towards an instructed direction, and wherein the visual feedback comprises projecting an image using at least one display device, the image comprising instructions in natural language text for the subject to move their head towards an instructed direction ([0077] and [0007], alarm).
It would have been obvious to one having ordinary skill in the art as of the effective filing date of the invention to combine Khand and Qin such that an audio feedback was provided to a user to look in a particular direction motivated by automating a feedback to a user.
Claim(s) 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Khan in view of Moon et al. (PGPUB 20220079436).
Regarding claim 17, Khan discloses the use of a plurality of prisms (1942 and 1944) for use in a first and second image (left and right eyes), but does not disclose the method further comprising:
processing the first image portion and the second image portion to form a three- dimensional image; and detecting the pupil in the three-dimensional image.
However, Moon teaches an eye imaging method using machine learning ([0004]) wherein the method comprises processing the first image portion and the second image portion to form a three- dimensional image; and detecting the pupil in the three-dimensional image ([0016]).
It would have been obvious to one having ordinary skill in the art as of the effective filing date of the invention to combine Khan and Moon such that the optical system generated a 3D image motivated by improving data quality for analysis.
Examiner Notes
Examiner cites particular columns and line numbers in the references as applied to the claims below for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested that, in preparing responses, the applicant fully consider the references in entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TRAVIS S FISSEL whose telephone number is (313)446-6573. The examiner can normally be reached on 9AM-5PM.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephone Allen can be reached on (571) 272-2434. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TRAVIS S FISSEL/Primary Examiner, Art Unit 2872
Prosecution Insights