DETAILED ACTION
Claims 1-20 have been examined. Claims 1, 4-9, 12 and 14-20 are rejected. Claims 2-3, 10-11 and 13 are objected.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 1, 9 and 17 are objected to because of the following informalities: claims 1, 9 and 17 recite “time of flight TOF”. The term or abbreviation “TOF” should be enclosed in parentheses (e.g. “a time of flight (TOF)”).
Claims 1, 9 and 17 are objected to because of the following informalities: claims 1, 9 and 17 recite “an RGB image sensor”. In regard to the term "RGB", any abbreviation or shortened representations of a term should be spelled out completely upon its first use in each claim branch (e.g., “red, green and blue (RGB) image sensor”). Appropriate correction is required.
Claims 2-3, 10-11, 13 and 18-19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. Additionally, claims 18-19 also need to be amended to overcome the rejections under 35 U.S.C. 101 and 35 U.S.C. 112(b) to be allowable.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 17-20 are rejected under 35 U.S.C. 101 because claims 17-20 are directed towards transitory propagating signals. A broad but reasonable interpretation of a claim drawn to a computer readable medium (also called machine readable medium and other such variations) typically covers forms of non-transitory tangible media and also transitory propagating signals in view of the ordinary and customary meaning of computer readable media. When the broadest reasonable interpretation of a claim covers a signal, the claim must be rejected under 35 U.S.C. § 101 as covering non-statutory subject matter. A claim drawn to such a computer readable medium or readable storage medium that covers both transitory and non-transitory embodiments may be amended to narrow the claim to cover only statutory embodiments to avoid a rejection under 35 U.S.C. § 101 by adding the limitation “non-transitory” to the claim. Claims 18-20 fail to remedy the deficiency of claim 17 and are rejected.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 17-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claims 17-20 recite “A readable storage medium”. However, the claims do not recite that the “storage medium” is computer-readable. Therefore, it is unclear how the “computer instructions” can be run on an electronic device if the storage medium cannot be read by the electronic device or a computer. The rejection may be overcome by amending the claims to recite “A non-transitory computer-readable storage medium, comprising computer instructions…”.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4-5, 9, 12, 17 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (CN 111685724, see the English translated copy) in view of Lehman et al. (U.S. PGPub 2016/0162020).
Regarding claims 1, 9 and 17, Wang teaches An electronic device, wherein the electronic device comprises a display, a time of flight TOF image sensor, an RGB image sensor, a memory, and one or more processors; the display, the time of flight TOF image sensor, the RGB image sensor, and the memory are coupled to the processor; the display is configured to display an image generated by the processor, the image sensor is configured to collect image data, the memory is configured to store computer program code, and the computer program code comprises computer instructions; and when the processor executes the computer instructions, the electronic device is enabled to perform: (Wang, see page 7, paragraph 5 comprise a depth sensor, or the image sensor is replaced by a depth sensor. Wherein, the depth sensor can be TOF (Time of Flight) sensor. Specifically, the TOF sensor emits modulated near infrared light...; see page 8, paragraphs 9-10 image sensor includes an RGB image pickup unit and/or an infrared image pickup unit...)
collecting first image data by using a target image sensor, wherein the first image data comprises an eye image of a user; determining first eye movement data of the user based on the first image data of the user, (Wang, see page 7, paragraphs 5-7 the image sensor...detect the eye depth information of the user to be detected, namely the eye depth information, can according to the eye depth information to determine whether to generate eye movement and eye movement of the specific condition...)
wherein the collecting first image data by using a target image sensor comprises: obtaining current ambient light luminance; and (Wang, see page 8, paragraphs 9-10 image sensor includes an RGB image pickup unit and/or an infrared image pickup unit…detecting the current ambient light intensity by the light intensity sensor; under the condition that the current ambient light intensity is greater than or equal to the preset light intensity threshold value, generating a starting instruction to the RGB image pickup unit, so as to open the RGB image pickup unit; under the condition that the light intensity of the current environment is less than the preset light intensity threshold value, generating a starting instruction to the infrared camera unit, so as to open the infrared camera unit.)
controlling, in response to that the current ambient light luminance is greater than a first preset luminance threshold, the RGB image sensor to collect first RGB image data; or controlling, in response to that the current ambient light luminance is less than or equal to a first preset luminance threshold, the TOF image sensor to collect first TOF image data. (Wang, see page 8, paragraphs 9-10 image sensor includes an RGB image pickup unit and/or an infrared image pickup unit…detecting the current ambient light intensity by the light intensity sensor; under the condition that the current ambient light intensity is greater than or equal to the preset light intensity threshold value, generating a starting instruction to the RGB image pickup unit, so as to open the RGB image pickup unit; under the condition that the light intensity of the current environment is less than the preset light intensity threshold value, generating a starting instruction to the infrared camera unit, so as to open the infrared camera unit.)
However, Wang does not explicitly teach displaying a first interface, wherein the first interface comprises a notification message;
wherein the first eye movement data comprises coordinates of a first gaze point and first gaze duration; and
in response to that the coordinates of the first gaze point are located in a display area of the notification message, and the first gaze duration is greater than a first preset duration threshold, displaying a second interface,
wherein the second interface is a display interface of an application corresponding to the notification message;
Lehman teaches displaying a first interface, wherein the first interface comprises a notification message; (Lehman, see figs. 2-5; see paragraph 0023 the preview duration is recognized (e.g., “YES” at 208), the method proceeds to 218 to display an application launcher. For example, the positional preview may be replaced with the application launcher as indicated at 220, such that at least an edge, center, or other portion of the application launcher is placed at a respective location of an edge, center, or other portion of the positional preview (e.g., displayed at 206 and/or in the new location at 216)...; see abstract; see paragraphs 0028-0029 viewable alongside displayed augmentation elements, such as notification icons 402 (e.g., showing a weather and a new message indication). As represented by gaze line 404, the user's gaze may be directed toward gaze target 406,...Before time T2, a user trigger is recognized, indicating an intent to display an application launcher at the gaze target 406. )
wherein the first eye movement data comprises coordinates of a first gaze point and first gaze duration; and (Lehman, see figs. 2-5; see paragraph 0016 location (e.g., a pixel coordinate) at which the vector of the gaze direction intersects the display may be determined to be the gaze target …; see paragraph 0059 an eye-tracking camera, a gaze target, responsive to a user trigger, displaying, via a display, an application launcher at the gaze target...via the eye-tracking camera, that a user gaze is directed to the gaze target for a duration that is greater than a trigger threshold...; see a paragraph 0017 recognizing a user trigger indicating a request to display an application launcher...the eye-tracking camera (e.g., as used for recognizing the gaze target)...; see paragraph 0019 displaying a positional preview of an application launcher (e.g., application launcher 109 of FIG. 1) at the gaze target responsive to a user trigger...)
in response to that the coordinates of the first gaze point are located in a display area of the notification message, and the first gaze duration is greater than a first preset duration threshold, displaying a second interface, (Lehman, see figs. 2-5; see paragraph 0059 an eye-tracking camera, a gaze target, responsive to a user trigger, displaying, via a display, an application launcher at the gaze target...via the eye-tracking camera, that a user gaze is directed to the gaze target for a duration that is greater than a trigger threshold...; see a paragraph 0017 recognizing a user trigger indicating a request to display an application launcher...the eye-tracking camera (e.g., as used for recognizing the gaze target)...; see paragraph 0019 displaying a positional preview of an application launcher (e.g., application launcher 109 of FIG. 1) at the gaze target responsive to a user trigger...)
wherein the second interface is a display interface of an application corresponding to the notification message; (Lehman, see figs. 2-5; see paragraph 0059 an eye-tracking camera, a gaze target, responsive to a user trigger, displaying, via a display, an application launcher at the gaze target...via the eye-tracking camera, that a user gaze is directed to the gaze target for a duration that is greater than a trigger threshold...; see a paragraph 0017 recognizing a user trigger indicating a request to display an application launcher...the eye-tracking camera (e.g., as used for recognizing the gaze target)...; see paragraph 0019 displaying a positional preview of an application launcher (e.g., application launcher 109 of FIG. 1) at the gaze target responsive to a user trigger...)
It would have been obvious to one of ordinary skill in the art, at the time the invention was filed, to combine Wang and Lehman to provide the techniques of displaying a first interface, wherein the first interface comprises a notification message, the first eye movement data comprises coordinates of a first gaze point and first gaze duration, in response to that the coordinates of the first gaze point are located in a display area of the notification message, and the first gaze duration is greater than a first preset duration threshold, displaying a second interface, and the second interface is a display interface of an application corresponding to the notification message of Lehman in the system of Wang in order to further improve user experience by allowing Wang's system to run applications and provide a seamless application launching experience for a user of a computing device (Lehman, see paragraph 0033).
Regarding claims 4, 12 and 20, Wang-Lehman teaches wherein the method further comprises: in response to that the coordinates of the first gaze point are located outside a display area of the notification message, displaying a fifth interface after first preset duration, (Lehman, see figs. 2-5; see paragraphs 0028-0029 viewable alongside displayed augmentation elements, such as notification icons 402 (e.g., showing a weather and a new message indication)...Before time T2, a user trigger is recognized, indicating an intent to display an application launcher at the gaze target 406. For example, the user trigger may be performed and/or recognized at some time between times T1 and T2. Accordingly, a positional preview 408 of the application launcher may be displayed as an augmentation to the real-world environment, such that the preview is centered at the gaze target...; note gaze target 406 is outside of display area of the notification icon 402)
wherein the fifth interface does not comprise the notification message. (Lehman, see figs. 2-5; see paragraphs 0029-0030 the user trigger may be performed and/or recognized at some time between times T1 and T2. Accordingly, a positional preview 408 of the application launcher may be displayed as an augmentation to the real-world environment...the positional preview 408 is replaced with an application launcher 410. The application launcher 410 includes a plurality of icons 412, each icon identifying a different selectable application for launching.) The motivation regarding to the obviousness to claims 1, 9 and 17 is also applied to claims 4, 12 and 20.
Regarding claim 5, Wang-Lehman teaches wherein after the controlling, in response to that the current ambient light luminance is greater than a first preset luminance threshold, the RGB image sensor to collect first RGB image data, the method further comprises: if it is determined, based on the first RGB image data, that eyes of the user are not covered by a reflective object, controlling the RGB image sensor to be turned off, and controlling the TOF image sensor to be turned on and then collect the first image data. (Wang, see page 8, paragraphs 9-10 image sensor includes an RGB image pickup unit and/or an infrared image pickup unit…detecting the current ambient light intensity by the light intensity sensor; under the condition that the current ambient light intensity is greater than or equal to the preset light intensity threshold value, generating a starting instruction to the RGB image pickup unit, so as to open the RGB image pickup unit; under the condition that the light intensity of the current environment is less than the preset light intensity threshold value, generating a starting instruction to the infrared camera unit, so as to open the infrared camera unit.; Note that claim 5 is a method claim and contains the contingent limitation "...if it is determined...". The broadest reasonable interpretation of a method (or process) claim having contingent limitations requires only those steps that must be performed and does not include steps that are not required to be performed because the condition(s) precedent are not met. The limitation "it is determined, based on the first RGB image data, that eyes of the user are not covered by a reflective object, controlling the RGB image sensor to be turned off, and controlling the TOF image sensor to be turned on and then collect the first image data" is interpreted as the condition is not being met, and therefore, the limitation following the condition "if it is determined..." is not required by the broadest reasonable interpretation. The rejection of claim 5 may be overcome by replacing the limitation "if it is determined,..." with "in response to determining, based on the first RGB image data, that...")
Claims 6-7 and 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Wang-Leman in view of Williams et al. (U.S. PGPub 2019/0324528).
Regarding claims 6 and 14, Wang-Leman teaches all of the features of claims 1 and 9. However, Wang-Leman does not explicitly teach wherein the method further comprises: displaying a sixth interface, wherein the sixth interface comprises at least one calibration location; collecting fourth image data by using the target image sensor;
determining coordinates of a fourth gaze point of the user based on the fourth image data; and determining an eye movement calibration result based on an error between coordinates of the calibration location and the coordinates of the fourth gaze point.
Williams teaches wherein the method further comprises: displaying a sixth interface, wherein the sixth interface comprises at least one calibration location; collecting fourth image data by using the target image sensor; (Williams, see figs. 8-9; see paragraph 0086 a gaze point is received. The gaze point 177 may be received by the adjustment engine 165. The gaze point 177 may be associated with coordinates on the near eye display system 110 of the HMD device 100... because of an error in calibration of the eye tracking component 175, the gaze point 177 may not accurately reflect a true gaze point of the wearer...; see paragraph 0092 based on coordinates associated with the received gaze point and the adjustment model, an offset adjustment is determined. The offset adjustment 168 may be determined by the offset adjuster 315. The offset adjustment 168 may be a vector or a set of values that may be added to the gaze point 177 to create an adjusted gaze point 169 that corrects for the miscalibration of the eye tracking component 175...)
determining coordinates of a fourth gaze point of the user based on the fourth image data; and determining an eye movement calibration result based on an error between coordinates of the calibration location and the coordinates of the fourth gaze point. (Williams, see figs. 8-9; see paragraph 0086 a gaze point is received. The gaze point 177 may be received by the adjustment engine 165. The gaze point 177 may be associated with coordinates on the near eye display system 110 of the HMD device 100... because of an error in calibration of the eye tracking component 175, the gaze point 177 may not accurately reflect a true gaze point of the wearer...; see paragraph 0092 based on coordinates associated with the received gaze point and the adjustment model, an offset adjustment is determined. The offset adjustment 168 may be determined by the offset adjuster 315. The offset adjustment 168 may be a vector or a set of values that may be added to the gaze point 177 to create an adjusted gaze point 169 that corrects for the miscalibration of the eye tracking component 175...)
It would have been obvious to one of ordinary skill in the art, at the time the invention was filed, to combine Wang-Lehman and Williams to provide the technique of displaying a sixth interface, wherein the sixth interface comprises at least one calibration location; collecting fourth image data by using the target image sensor, determining coordinates of a fourth gaze point of the user based on the fourth image data, and determining an eye movement calibration result based on an error between coordinates of the calibration location and the coordinates of the fourth gaze point of Williams in the system of Wang-Lehman in order to correct for calibration errors without having to recalibrate the eye tracking components every time there is a change to a lighting condition or pupil size for a user (Williams, see paragraphs 0003).
Regarding claims 7 and 15, Wang-Leman-Williams wherein the electronic device comprises the time of flight TOF image sensor and the RGB image sensor; and (Wang, see page 7, paragraph 5 comprise a depth sensor, or the image sensor is replaced by a depth sensor. Wherein, the depth sensor can be TOF (Time of Flight) sensor. Specifically, the TOF sensor emits modulated near infrared light...; see page 8, paragraphs 9-10 image sensor includes an RGB image pickup unit and/or an infrared image pickup unit...; see page 8, paragraphs 9-10 image sensor includes an RGB image pickup unit and/or an infrared image pickup unit..)
the collecting fourth image data by using the target image sensor comprises: controlling the TOF image sensor to collect second TOF image data, and (Wang, see page 7, paragraph 5 comprise a depth sensor, or the image sensor is replaced by a depth sensor. Wherein, the depth sensor can be TOF (Time of Flight) sensor. Specifically, the TOF sensor emits modulated near infrared light...; see page 8, paragraphs 9-10 image sensor includes an RGB image pickup unit and/or an infrared image pickup unit...)
controlling the RGB image sensor to collect second RGB image data. (Wang, see page 8, paragraphs 9-10 image sensor includes an RGB image pickup unit and/or an infrared image pickup unit…detecting the current ambient light intensity by the light intensity sensor; under the condition that the current ambient light intensity is greater than or equal to the preset light intensity threshold value, generating a starting instruction to the RGB image pickup unit, so as to open the RGB image pickup unit; under the condition that the light intensity of the current environment is less than the preset light intensity threshold value, generating a starting instruction to the infrared camera unit, so as to open the infrared camera unit.)
Claims 8 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Wang-Leman-Williams in view of Ollila et al. (U.S. PGPub 2021/0327030) further in view of Price et al. (U.S. PGPub 2019/0058859).
Regarding claims 8 and 16, Wang-Leman-Williams wherein the electronic device comprises the time of flight TOF image sensor and the RGB image sensor; (Wang, see page 7, paragraph 5 comprise a depth sensor, or the image sensor is replaced by a depth sensor. Wherein, the depth sensor can be TOF (Time of Flight) sensor. Specifically, the TOF sensor emits modulated near infrared light...; see page 8, paragraphs 9-10 image sensor includes an RGB image pickup unit and/or an infrared image pickup unit...; see page 8, paragraphs 9-10 image sensor includes an RGB image pickup unit and/or an infrared image pickup unit..)
However, Wang-Leman-Williams does not explicitly teach the TOF image sensor and the RGB image sensor are connected through a connection cable; and
Ollila teaches the TOF image sensor and the RGB image sensor are connected through a connection cable; and (Ollila, see paragraph 0044 a given camera is at least one of: a digital camera, a RGB-D camera, a RGB-IR camera, a RGBW camera, a Time-of-Flight (ToF) camera; see paragraph 0046 coupled to various components (such as the first camera and the second camera) of the imaging system…in a wired manner...)
It would have been obvious to one of ordinary skill in the art, at the time the invention was filed, to combine Wang-Lehman-Williams and Ollila to provide the technique of the TOF image sensor and the RGB image sensor are connected through a connection cable of Ollila in the system of Wang-Lehman-Williams in order to facilitates an increase in overall efficiency of the system (Ollila, see paragraph 0033).
However, Wang-Lehman-Williams-Ollila does not explicitly teach the connection cable is configured to transmit a synchronization signal, so that the TOF image sensor and the RGB image sensor synchronously collect image data based on the synchronization signal.
Price teaches the connection cable is configured to transmit a synchronization signal, so that the TOF image sensor and the RGB image sensor synchronously collect image data based on the synchronization signal. (Price, see paragraph 0031 different camera types may be used together, such as an RGB color camera in combination with an IR camera or a ToF camera…; see a paragraph 0025 The system 100 may be configured to synchronize frames captured by the first and second cameras 120 and 130. For example, current frames 142 and 146 may both be for the same frame period, or overlapping frame periods...)
It would have been obvious to one of ordinary skill in the art, at the time the invention was filed, to combine Wang-Lehman-Williams-Ollila and Price to provide the technique of transmitting a synchronization signal, so that the TOF image sensor and the RGB image sensor synchronously collect image data based on the synchronization signal of Price in the system of Wang-Lehman-Williams-Ollila in order to improve performance of object detection and/or depth estimation (Price, see paragraph 0030).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. This includes:
U.S. PGPub 2022/0236797, which describes systems and method to estimate user gaze;
U.S. PGPub 2022/0285828, which describes a wearable electronic device including a gesture camera, an eye tracking camera, a distance measuring camera (a depth camera), and/or a red, green, and blue (RGB) camera; and
U.S. PGPub 2022/0391012, which describes devices, methods, and processor-readable media for eye tracking using statistically derived linear functions.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MENG VANG whose telephone number is (571)270-7023. The examiner can normally be reached M-F 8AM-2PM, 3PM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, NICHOLAS TAYLOR can be reached at (571) 272-3889. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MENG VANG/Primary Examiner, Art Unit 2443
Prosecution Insights