Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed on 25 March, 2024.
Response to Amendment
The amendment filed 6 April, 2026 has been entered.
The amendment of claim 16 has been acknowledged.
Response to Arguments
Applicant’s arguments, see page 1, section “Claim Rejections – 35 U.S.C. § 101”, filed 6 April, 2026 with respect to the rejection of claim 16 have been fully considered and are persuasive. The rejection of claim 16 under 35 U.S.C. § 101 has been withdrawn.
Applicant’s arguments, see page 1, section “Claim Rejections – 35 U.S.C. § 103”, filed 6 April, 2026 with respect to the rejection of claims 1 - 16 have been fully considered but they are not persuasive.
First, applicant states on page 1 that independent claim 1 requires “selecting at least one gaze point from the plurality of determined gaze points based on one or more attention criteria”, with claim 2 further defining this attention criteria to mean “a threshold fixation time” and “a threshold number of instances”. Applicant states that the plain meaning of “threshold” requires a filtering operation and that Koo et al (KR2022/0060163A, Employed using the provided machine translation, hereinafter “Koo”) fails to teach a threshold as Koo merely acquires or records gaze frequency and fixation time, but does not rely on thresholds (and therefore filtering operations) to select gaze points. Applicant states the action of “selecting” in claim 1 “implies an active filtering process and that there is no teaching in Koo of applying a threshold (e.g. fixation time > X seconds) to filter out gaze points that do not meet an attention criterion before proceeding to further processing steps.”. The examiner respectfully disagrees.
Independent claim 1 recites the limitation of “selecting at least one gaze point from the plurality of determined gaze points based on the one or more attention criteria”. Independent claim 1 does not recite the use of any thresholding techniques in the step of “selecting”. The step of “selecting at least one gaze point from the plurality of determined gaze points based on one or more attention criteria” merely requires any step of choosing any gaze points based on one or more attention criteria. Koo ¶ 0053 teaches “The user’s gaze point information includes a gaze point position on a space in which the user is located, and a gaze distance from the user to the gaze target. The user’s gaze information may be acquired using not only information on the position of the pupil, but also Purkinje images, gaze convergence information, and the like. The user’s gaze point information may include not only the gaze point location and gaze distance, but also information on the user’s gaze frequency… whether the gaze is fixed, the gaze fixation time, and whether the gaze is tracked.”. Under broadest reasonable interpretation, it is understood by the examiner that Koo teaches this claim limitation fully as Koo uses information from the user, such as pupil location and gaze convergence information, to identify the gaze position of the user. The pupil location and gaze convergence information serve as the attention criteria in this case, as they directly indicate where the user’s pupils are directed and therefor giving attention to. The gaze position is also one which is selected based on a plurality of candidate positions, these candidate positions being any position within the headset a user could view.
Secondly, the applicant states claim 1 further requires “receiving at least one scene image from the outward-facing image sensor at a time corresponding to the timestamp of the at least one selected gaze point”. Applicant alleges on page 2 that “Koo neither discloses nor suggests a two-stage process where (1) gaze points are first selected based on attention criteria, and then (2) a scene image is retrieved based on the timestamp of that selected gaze point.”. Again, the examiner respectfully disagrees.
Regarding Koo’s failure to teach “(1) gaze points are first selected based on attention criteria, and then (2) a scene image is retrieved based on the timestamp of that selected gaze point”, Koo explicitly teaches what applicant describes at the first step corresponding to the claim limitation of “selecting at least one gaze point from the plurality of gaze points based on the one or more attention criteria (see paragraphs above). Regarding “a scene image is retrieved based on the timestamp of that selected gaze point” corresponding to the claim limitation of “receiving at least one scene image from the outward-facing image sensor at a time corresponding to the timestamp of the at least one selected gaze point”, the claim language of “receiving at least one image” corresponds to Koo ¶ 0044 “Further, the wearable device may include a front camera 1 configured to capture an image comprising at least a portion of the user’s field of view”. Koo discloses a video passthrough headset which provides images to the user in real time. Additionally, this headset collects images of the user’s pupils and eye movements to detect the gaze information of the user. ¶ 0055 of Koo states “First, the gaze point information estimator 10 may detect the Purkinje image and the pupil from the pupil images acquired from the left or right pupil cameras 2 and 3 (s2, s3) (s11). In addition, image calibration between the pupil-captured image and the front camera may be performed (s12), and a convergence point may be estimated using the detected pupil information and the calibrated gaze image of the front camera (s13). (emphasis added)”. This paragraph describes the process of Koo consisting of collecting gaze information from the user, then calibrating the pupil captured image with the image collected from the front camera at the exact same time. The gaze information is then used in conjunction with this image in step s12 and s13 to estimate the convergence point (i.e. direction/location of user’s gaze in the front camera image) of the user. This process would distinctly require the use of the image collected by the front camera that was captured at the same time as the pupil-captured image(s) so as to distinctly identify the position in the front camera image where the users gaze is fixated. As such the examiner believes that Koo does teach the claim limitations of “selecting at least one gaze point from the plurality of determined gaze points based on one or more attention criteria”, and “receiving at least one scene image from the outward-facing image sensor at a time corresponding to the timestamp of the at least one selected gaze point”.
Finally, the applicant states on page 2 that the applicants claim “requires that the Area of Interest (AOI) is identified in the scene image that was received at the timestamp of the selected gaze point”, however in Koo, “the ROI is generated in real-time from the current gaze point and current scene image”. The applicant further specifies that the scene image is not necessarily the same as the image used to determine the gaze point. Per the applicant “The claimed process creates a clear temporal and causal separation: (a) gaze point are determined, (b) certain gaze points are selected based on attention criteria, and (c) for each selected gaze point, the scene image from that specific moment is retrieved and used to identify AOIs”. Again, the examiner respectfully disagrees.
Claim 1 as written has the following limitation: “identifying one or more areas of interest “AOIs” in the scene image based on the at least one selected gaze point.”. This limitation under broadest reasonable interpretation encompasses identifying a region of interest within the scene image based on the users gaze information. Koo teaches explicitly a step of calibrating a pupil-captured image with a front camera image captured at the same time (see ¶ 0055). The convergence point disclosed in ¶ 0055 of Koo is the location within the image that the user’s gaze is directed/focused. ¶ 0061 of Koo discloses a region of interest (ROI) generation step, which generates an ROI based on the user’s gaze information and uses this information in tandem with front images to determine a region of interest (See Koo Fig. 5). There is nothing in this claim limitation that states “the scene image is not necessarily the same as the image used to determine the gaze point”. Additionally, there is nothing that distinguishes the applicant’s claim limitations from what is disclosed in Koo. As such the examiner upholds the prior 35 U.S.C. 35 § 102 rejections under Koo.
Applicant’s arguments, see page 2 - 3, section “Claim Rejections – 35 U.S.C. § 103”, filed 6 April, 2026 with respect to the rejection of claims 11, 12, 14, and 15 have been fully considered but they are not persuasive.
Applicant states that the stated motivation of “the system of Lohr can work communicatively with remote processing and storage, thus providing additionally processing capabilities than what is possible to be installed directly on a head mounted display” is generic and that there is no teaching or suggestion in either reference that adding remote processing would lead to the specific, ordered steps of the applicant’s method, nor a reason why any person of ordinary skill in the art would combine the references to achieve the applicant’s claimed invention. The examiner respectfully disagrees.
The assertion that the motivation to combine is a generic improvement does not preclude the fact that it is still an improvement that would motivate one of ordinate skill in the art to utilize Lohr et al (U.S. Patent Publication No 2024/0265650 A1, hereinafter “Lohr”) for the benefits it provides. As both Koo and Lohr pertain to the same field of endeavor, that being head mounted display image processing and detection, it would be obvious for one skilled in the art to modify Koo with the teaching of Lohr pertaining to the benefit it provides such that remote computing resources are utilized, thus far exceeding the computational capacity of the head mounted display. One skilled in the art would be aware of the benefits that are associated with an increase in computational capacity and with respect to image processing allows for a higher quality, more thorough processing, of the images while reducing the load directly associated with the head mounted display.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of pre-AIA 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a) the invention was known or used by others in this country, or patented or described in a printed publication in this or a foreign country, before the invention thereof by the applicant for a patent.
(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on sale in this country, more than one year prior to the date of application for patent in the United States.
Claims 1 – 10 and 13 are rejected under pre-AIA 35 U.S.C. 102(a)(1) as being anticipated by KR2022/0060163A Koo et al (Employed using the provided machine translation, hereinafter “Koo”).
Regarding claim 1, Koo teaches a method for identifying at least one area of interest in an environment around a wearable device (¶ 0012: The first aspect of the present invention for solving the above problems is a three-dimensional gaze that receives and processes an image of an external environment including a plurality of objects from a camera in the direction of the user's gaze and information on the location of the user's pupil), wherein the wearable device comprises an eye tracking arrangement and an outward-facing image sensor (¶ 0044: Further, the wearable device may include a front camera 1 configured to capture an image comprising at least a portion of the user's field of view.; ¶ 0045: Referring to FIG. 2 (b) , the wearable device used in the present invention may include left and right pupil cameras 2 and 3.), the method comprising:
determining a plurality of gaze points of a user of the wearable device and corresponding timestamps using at least the eye tracking arrangement (¶ 0052: The gaze point information estimator 10 communicates with the external environment through the image of the external environment obtained from the front camera 1 and the image information of the user's binocular pupils obtained from the left and right pupil cameras 2 and 3) Gaze point information according to the user's gaze is estimated (s10).; ¶ 0053: The user's gaze point information includes a gaze point position on a space in which the user is located, and a gaze distance from the user to the gaze target.; ¶ 0124: The gaze point information estimated by the gaze point information estimator 10 includes the user's gaze point location and gaze time. The gaze point information estimator 10 may also acquire information on the gaze frequency and gaze point formation time for the object of interest.);
selecting at least one gaze point from the plurality of determined gaze points based on one or more attention criteria (¶ 0053: not only information on the position of the pupil, but also Purkinje images, gaze convergence information, and the like. The user's gaze point information may include not only the gaze point location and gaze distance, but also information on the user's gaze frequency, whether the pupil is saccades, whether the gaze is fixed, the gaze fixation time, and whether the gaze is tracked.; ¶ 0126: Metadata generator 40 receives the user's gaze point information from the gaze point information estimator 10, and obtains visual information at which the user gazes at each object and location information of each object in the form of an object of interest information log by time.);
receiving at least one scene image from the outward-facing image sensor at a time corresponding to the timestamp of the at least one selected gaze point (¶ 0052: The gaze point information estimator 10 communicates with the external environment through the image of the external environment obtained from the front camera 1 and the image information of the user's binocular pupils obtained from the left and right pupil cameras 2 and 3) Gaze point information according to the user's gaze is estimated (s10).; ¶ 0055: First, the gaze point information estimator 10 may detect the Purkinje image and the pupil from the pupil images acquired from the left or right pupil cameras 2 and 3 (s2, s3) (s11). In addition, image calibration between the pupil-captured image and the front camera may be performed (s12), and a convergence point may be estimated using the detected pupil information and the calibrated gaze image of the front camera (s13).); and
identifying one or more areas of interest “AOIs” in the scene image based on the at least one selected gaze point (¶ 0061: Referring to FIG. 5, the region of interest generating unit 20 generates an ROI corresponding to the user's gaze point according to the user's gaze point information and a predetermined criterion among the images of the external environment obtained from the front camera 1.).
Regarding claim 2, Koo teaches the method of claim 1.
Additionally, Koo teaches wherein the one or more attention criteria comprises at least one of a threshold fixation time, a threshold number of instances of the gaze point, and a classification of an object corresponding to the gaze point (¶ 0019: According to the embodiment of the present invention, the user's gaze point information estimator acquires gaze frequency and gaze point formation time information for the object of interest, and the metadata generator classifies the object of interest by object type, and the Gaze frequency and gaze point formation time information for the object of interest may be recorded for each object of interest classified by object type.; ¶ 0053: The user's gaze point information includes a gaze point position on a space in which the user is located, and a gaze distance from the user to the gaze target. The user's gaze point information may be acquired using not only information on the position of the pupil, but also Purkinje images, gaze convergence information, and the like. The user's gaze point information may include not only the gaze point location and gaze distance, but also information on the user's gaze frequency, whether the pupil is saccades, whether the gaze is fixed, the gaze fixation time, and whether the gaze is tracked.).
Regarding claim 3, Koo teaches the method of claim 2.
Additionally, Koo teaches comprising selecting a gaze point from the plurality of determined gaze points if the gaze point is viewed for more than the threshold fixation time, is viewed for more than the threshold number of instances, and/or corresponds to an object having a predetermined classification (¶ 0019: According to the embodiment of the present invention, the user's gaze point information estimator acquires gaze frequency and gaze point formation time information for the object of interest, and the metadata generator classifies the object of interest by object type, and the Gaze frequency and gaze point formation time information for the object of interest may be recorded for each object of interest classified by object type. (emphasis added)).
Regarding claim 4, Koo teaches the method of claim 1.
Additionally, Koo teaches wherein identifying one or more AOIs in the scene image comprises determining an image patch around the at least one gaze point (¶ 0065: The region of interest generator 20 adjusts the size of the region of interest according to the user's gaze distance. When the user's gaze distance is relatively long, the region of interest generator 20 sets the size of the region of interest to be small in order to focus the user on the corresponding object… relatively close, the region of interest generator 20 sets the size of the region of interest to be large in order to focus the user on the corresponding object. That is, the user's gaze distance can be a parameter defining the ROI of the object.).
Regarding claim 5, Koo teaches the method of claim 4.
Additionally, Koo teaches wherein determining an image patch comprises at least one of:
determining an image patch having a fixed size;
determining an image patch having a size dependent on a vergence distance associated with the gaze point (¶ 0065: The region of interest generator 20 adjusts the size of the region of interest according to the user's gaze distance. When the user's gaze distance is relatively long, the region of interest generator 20 sets the size of the region of interest to be small in order to focus the user on the corresponding object… relatively close, the region of interest generator 20 sets the size of the region of interest to be large in order to focus the user on the corresponding object. That is, the user's gaze distance can be a parameter defining the ROI of the object.);
determining an image patch using an edge detection algorithm; and
determining an image patch using a neural network (¶ 0016: According to the embodiment of the present invention, the object information detection unit sets a comparison characteristic between the region of interest generated by the region of interest generator and the image of the external environment as input information, learns the comparison characteristic. An object of interest can be extracted by constructing a variable neural network that selectively combines feature map layers. (emphasis added)).
Regarding claim 6, Koo teaches the method of claim 1.
Additionally, Koo teaches comprising:
selecting a plurality of gaze points from the plurality of determined gaze points (¶ 0132: The metadata providing unit 50 receives the user's object of interest change over time);
identifying a plurality of AOIs in the scene image (¶ 0132: The metadata providing unit 50 receives the user's object of interest change over time and the gaze frequency information for each object of interest updated from the metadata generator,); and
clustering the identified AOIs into one or more clusters (Figures 12 and 14; ¶ 0132: The metadata providing unit 50 receives the user's object of interest change over time and the gaze frequency information for each object of interest updated from the metadata generator, and classifies and provides the objects of interest in the order of the user's highest degree of interest. (emphasis added)).
Regarding claim 7, Koo teaches the method of claim 6.
Additionally, Koo teaches wherein clustering the identified AOIs into one or more clusters comprises using at least one of a matching algorithm, a computer vision feature detection algorithm and a machine-learning clustering algorithm (¶ 0016: An object of interest can be extracted by constructing a variable neural network that selectively combines feature map layers.; ¶ 0132: The metadata providing unit 50 receives the user's object of interest change over time and the gaze frequency information for each object of interest updated from the metadata generator, and classifies and provides the objects of interest in the order of the user's highest degree of interest.).
Regarding claim 8, Koo teaches the method of claim 1.
Additionally, Koo teaches comprising determining a gaze time associated with each identified AOI (Figure 12 – 14; ¶ 0053: The user's gaze point information may include not only the gaze point location and gaze distance, but also information on the user's gaze frequency, whether the pupil is saccades, whether the gaze is fixed, the gaze fixation time, and whether the gaze is tracked.).
Regarding claim 9, Koo teaches the method of claim 8.
Additionally, Koo teaches wherein the gaze time comprises:
a total time for which the AOI was viewed by the user (Figure 12 – 14; ¶ 0053: The user's gaze point information may include not only the gaze point location and gaze distance, but also information on the user's gaze frequency, whether the pupil is saccades, whether the gaze is fixed, the gaze fixation time, and whether the gaze is tracked. (emphasis added));
and/or a ratio between the total time for which the AOI was viewed by the user and the total time for which the AOI was present in the scene image.
Regarding claim 10, Koo teaches the method of claim 1.
Additionally, Koo teaches wherein determining a gaze point of the user comprises determining at least one of a gaze direction of the user, a position of the user, a vergence distance, and a viewing angle (¶ 0013: According to the embodiment of the present invention, the user's gaze point information may include a gaze point location in a space where the user is located, and a gaze distance from the user to the gaze target).
Regarding claim 13, claim 13 has been analyzed with regard to claim 1 and is rejected for the same reasons of obviousness as used above as well as in accordance with Koo’s further teaching on:
At least one processor configured to (¶ 0005: a processor for controlling the touch screen…; ¶ 0080: … and the amount of computation of the processor can be reduced…)…
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 11, 12 and 14 - 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over KR2022/0060163A Koo et al (Employed using the provided machine translation, hereinafter “Koo”) et al in view of Lohr et al (U.S. Patent Publication No. 2024/0265650 A1, hereinafter “Lohr”).
Regarding claim 11, Koo teaches the method of claim 1.
Koo does not explicitly teach wherein the at least one scene image comprises a video.
However, Lohr does teach wherein the at least one scene image comprises a video (¶ 0022: That is, the HMD may include one or more modes, such as a pass-through mode where the real-world environment is presented to the user… In turn, the display may present images of the real-world environment to allow a user to locate his or her glass of water without taking off the HMD.; ¶ 0023: the images captured by the camera may be presented on an entirety of the display…; Examiner’s note: pass-through of a wearable headset is known to be a video capture method which displays images of the headset’s surroundings in real time.).
Koo and Lohr are considered to be analogous art as both pertain to head mounted display image processing and object detection. Therefore, it would have been obvious to one of ordinary skill in the art to combine the apparatus for detecting objects of interest based on 3D gaze point information (as taught by Koo) and the head-mounted display with pass-through imaging (as taught by Lohr) before the effective filing date of the claimed invention. The motivation for this combination of references would be the system of Lohr can work communicatively with remote processing and storage, thus providing additional processing capabilities than what is possible to be installed directly on a head mounted display (See ¶ 0046).
This motivation for the combination of Koo and Lohr is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Regarding claim 12, Koo teaches the method of claim 1.
Additionally, Lohr teaches wherein the wearable device comprises a 3D sensor and/or a positional device (¶ 0018: For example, the HMD (head mounted display) may include a depth sensor to determine distances between the user and the objects in the real-world environment…).
Regarding claim 14, Koo teaches the system of claim 13.
Additionally, Lohr teaches wherein the wearable device comprises a control unit, and the at least one processor comprises:
A first processor comprised in the control unit and configured to perform the determining step (¶ 0034: As illustrated, the HMD 104 may include processor(s) 114 that carry out or otherwise perform operations associated with the HMD 104. For example, the processor(s) 114 cause the first camera 110 and/or the second camera 112 to capture images, and subsequently, may receive images captured by the first camera 110 and/or the second camera 112, compare the images (or image data), and determine differences therebetween.); and
A second processor comprised in the control unit and configured to perform the selecting, receiving, and identifying steps (¶ 0034: As illustrated, the HMD 104 may include processor(s) 114 that carry out or otherwise perform operations associated with the HMD 104. For example, the processor(s) 114 cause the first camera 110 and/or the second camera 112 to capture images, and subsequently, may receive images captured by the first camera 110 and/or the second camera 112, compare the images (or image data), and determine differences therebetween.).
Regarding claim 15, Koo teaches the system of claim 13.
Additionally, Lohr teaches wherein the wearable device comprises a control unit, and the at least one processor comprises:
A first processor comprised in the control unit and configured to perform the determining step (¶ 0034: As illustrated, the HMD 104 may include processor(s) 114 that carry out or otherwise perform operations associated with the HMD 104. For example, the processor(s) 114 cause the first camera 110 and/or the second camera 112 to capture images, and subsequently, may receive images captured by the first camera 110 and/or the second camera 112, compare the images (or image data), and determine differences therebetween.); and
A second processor remote from the wearable device and configured to perform the selecting, receiving, and identifying steps (¶ 0048: In some instances, the remote computing resources 142 may be remote from the environment 102 and the HMD 104 may communicatively couple to the remote computing resources 142 via the network interfaces 126 and over the network 124. The remote computing resources 142 may be implemented as one or more servers and may in some instances, form a portion of a network-accessible computing platform implemented as a computing infrastructure of processors, storage, software, data access, and so forth that is maintained and accessible via a network such as the Internet.).
Regarding claim 16, the Koo and Lohr combination teaches a computer-readable medium having stored thereon instructions that, when executed by one or more processors cause execution of the method steps according to claim 1 (¶ 0051: The memory may be implemented as computer-readable storage media (“CRSM”), which may be any available physical media accessible by the processor(s) to execute instructions stored on the memory.).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW JONES whose telephone number is (703)756-4573. The examiner can normally be reached Monday - Friday 8:00-5:00 EST, off Every Other Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW B. JONES/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights