DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 9/16/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered and attached by the examiner.
Claim Objections
Claim 1 is objected to because of the following informalities:
“generating, by a neural network, a first gaze prediction based the given image and a mirror gaze prediction based on the mirror image” should be corrected to “generating, by a neural network, a first gaze prediction based on the given image and a mirror gaze prediction based on the mirror image” Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2 and 7-8 are rejected under 35 U.S.C. 103 as being unpatentable over Linden (U.S. Patent Pub. No. 2019/0303723) in view of Mustikovela (U.S. Patent Pub. No. 2021/0150757).
Regarding Claim 1, Linden teaches a method for gaze prediction, the method comprising (¶1 The present application relates to gaze detection systems and methods. In an example, such systems and methods rely on deep learning systems, such as neural networks to detect three dimensional (3D) gaze:) Linden 2019/0303723
generating a mirror image from a given image, the given image obtained from a database of training images (¶52 one of the eye warped images 642 is mirrored such that the resulting mirrored image aligns with the other (un-mirrored) eye warped image. For example, the warped image around the left eye is mirrored… By having this alignment, the eye warped image input to the neural network is the same in terms of orientation, thereby simplifying the architecture and the training of the neural network;)
generating, by a neural network, a first gaze prediction based the given image and a mirror gaze
prediction based on the mirror image (¶86 Once the training images (normal and mirrored image) are generated, they are input to the neural network for the training. In particular, the neural network predicts the gaze angles and the user eye to-camera distances from these training images;)
generating a symmetry loss value based on a comparison between the first gaze prediction and
the mirror gaze prediction (Fig. 12-14; ¶86 The loss function can be defined based relative to gaze angles and user eye to-camera distances. In the interest of clarity, consider the first training image described herein above (e.g., generated when the user eye was gazing at the gaze point 1220). Based on that training image (and the second training image), the neural network predicts the first solution (e.g., the first angle “a1” 1232 and the first distance “d1” 1234);)
updating at least some parameters of the neural network based on a combination of the symmetry loss value (Fig. 12-14; ¶80 Generally, the training is iterative across the training images to minimize a loss function and, accordingly, update the parameters of the neural network through back-propagation (e.g., one that uses gradient descent); ¶86 The goal of the training is to update the parameters of the neural network such that its loss function is minimized;)
using the updated neural network to provide a further gaze prediction based on a further image,
the further image obtained from a user device; and (Fig. 6; ¶47 this 2D image 610 is generated with a camera; ¶74 At operation 1110, the eye tracking system inputs to a neural network the warped image of the user eye. If the other two warped images (e.g., of the second user eye and the user face) are generated, they are also input to the neural network. The neural network is already trained and predicts (i) a distance correction c and (ii) a 2D gaze origin and a 2D gaze direction per eye in the associated warped eye image.)
outputting the further gaze prediction to an application utilizing the further gaze prediction (¶26 that same trained neural network can be used across different eye tracking systems including ones integrated with different types of smartphones, tablets, laptops, wearable headset devices (e.g., virtual reality and augmented reality headsets), and standalone eye tracking systems. Further, because 3D gaze is detected, stereoscopic displays and 3D applications can be supported.)
Linden does not explicitly disclose generating a symmetry loss value based on a comparison between the first image and a generated image; generating, by the neural network, a reconstructed image based on at least one of the given image and the mirror image; generating a reconstruction loss value based on a comparison between the reconstructed image and the at least one of the given image and the mirror image; updating at least some parameters of the neural network based on a combination of the symmetry loss value and the reconstruction loss value.
Mustikovela is in the same field of art of image analysis. Further, Mustikovela teaches generating a symmetry loss value based on a comparison between the first image and a generated image (Fig. 7; ¶114 In at least one embodiment, symmetry loss 736 is computed based on first generated image 708 and second generated image 726.)
generating, by the neural network, a reconstructed image based on at least one of the given image and the mirror image (¶107 generator 706 accepts first viewpoint 702 and set of appearance parameters 704 as inputs and creates a first generated image 708. In at least one embodiment, first generated image 708 is a synthetic image with appears generated based on set of appearance parameters 704 and oriented according to first viewpoint 702;)
generating a reconstruction loss value based on a comparison between the reconstructed image
and the at least one of the given image and the mirror image (Fig. 7; ¶110 In at least one embodiment, Z reconstruction loss 720 is computed. In at least one embodiment, Z reconstruction loss refers to a difference or distance between set of appearance parameters 704 and a predicted first set of appearance parameters 716;)
updating at least some parameters of the neural network based on a combination of the symmetry loss value and the reconstruction loss value (¶108 loss functions are used to compute gradients (e.g., using gradient descent) and update parameters for discriminator 710 while fixing parameters of generator 706 constant.)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Linden by generating a symmetry and reconstruction loss and updating parameters of the network based on those losses that is taught by Mustikovela; thus, one of ordinary skilled in the art would be motivated to combine the references to improve amounts of memory, time, and/or computing resources used to train neural networks (Mustikovela ¶2).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 2, Linden in view of Mustikovela discloses the method of claim 1, wherein the neural network comprises a convolutional neural network (Linden, ¶54 the neural network 650 is a convolutional neural network that includes multiple subnetworks (e.g., along parallel branches of the neural network 650).)
Regarding claim 7, claim 7 has been analyzed with regard to claim 1 and is rejected for the same reasons of obviousness as used above as well as in accordance with Linden further teaching on: A system comprising: at least one processor; and a memory storing computer program instructions executable by the at least one processor (Linden, Claim 8: A computer system comprising: a processor; and a memory storing computer-readable instructions that, upon execution by the processor, configure the computer system to perform operations)
Claim 8 recites limitations similar to claim 2 and is rejected under the same rationale and reasoning.
Claims 3 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Linden (U.S. Patent Pub. No. 2019/0303723) in view of Mustikovela (U.S. Patent Pub. No. 2021/0150757) in view of Zhu (U.S. Patent Pub. No. 2023/0274531).
Regarding Claim 3, Linden in view of Mustikovela teaches the method of claim 1 wherein the neural network provides gaze predictions (see rejection of claim 1.)
Linden in view of Mustikovela does not explicitly disclose wherein the neural network comprises a backbone providing backbone output to a decoder for generating reconstructed images and to a gaze estimation layer for providing gaze predictions.
Zhu is in the same field of art of image analysis. Further, Zhu teaches wherein the neural network comprises a backbone providing backbone output to a decoder for generating reconstructed images and to a gaze estimation layer for providing (¶67 S101: Input a to-be-detected image into a pretrained global and local feature reconstruction network; ¶68 where the global and local feature reconstruction network GLFRNet includes a feature encoding module, a global feature reconstruction GFR module, and a feature decoding module based on a local feature reconstruction LFR module, and the global feature reconstruction GFR module is embedded in a skip connection between the feature encoding module and the feature decoding module; ¶69 The feature encoding module uses ImageNet pretrained ResNet34 t as a backbone network with a last global pooling layer and a fully connected layer removed.)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Linden in view of Mustikovela by implementing a backbone that feeds into a decoder and estimation layers that is taught by Zhu; thus, one of ordinary skilled in the art would be motivated to combine the references to overcome the problems such as insufficient global feature extraction (Zhu ¶6).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 9 recites limitations similar to claim 3 and is rejected under the same rationale and reasoning.
Claims 6 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Linden (U.S. Patent Pub. No. 2019/0303723) in view of Mustikovela (U.S. Patent Pub. No. 2021/0150757) in view of Zhu (U.S. Patent Pub. No. 2023/0274531) in view of Wu (U.S. Patent Pub. No. 2025/0022100).
Regarding Claim 3, Linden in view of Mustikovela in view of Zhu teaches the method of claim 1 wherein the neural network includes a backbone, decoder, and gaze estimation (see rejection of claim 1 and 3.)
Linden in view of Mustikovela does not explicitly disclose wherein in updating at least some parameters of the neural network comprises freezing pre-trained parameters.
Zhu is in the same field of art of image analysis. Further, Zhu teaches wherein in updating at least some parameters of the neural network comprises freezing pre-trained parameters (¶39 To perform transfer learning and/or fine-tuning operations, the system 100 can retrieve or otherwise identify (a first instance of) a given machine learning model 116, freeze (e.g., maintain values of) one or more weights, biases, or parameters of one or more layers of the given machine learning model 116 (e.g., an output layer), and perform training of the given machine learning model while the parameters are frozen (e.g., using training data instances 108 selected for fine-tuning and/or transfer learning).)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Linden in view of Mustikovela in view of Zhu by freezing parameters that is taught by Zhu; thus, one of ordinary skilled in the art would be motivated to combine the references to optimize the updating of machine learning models (Wu ¶39).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 12 recites limitations similar to claim 6 and is rejected under the same rationale and reasoning.
Allowable Subject Matter
Claims 4-5 and 10-11 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claims 4 and 10, no prior art teaches wherein updating at least some parameters of the neural network comprises updating a meta prompt that determines characteristics of padding applied to the given image and the mirror image.
Regarding claims 5 and 11, no prior art teaches wherein the neural network is a convolutional neural network comprising m layers and the updating at least some parameters of the neural network comprises updating a meta prompt that determines characteristics of padding applied to a first n output feature maps from a preceding layer of the m layers of the convolution neural network, wherein n is greater than 2 and less than m.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DUSTIN BILODEAU whose telephone number is (571)272-1032. The examiner can normally be reached 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUSTIN BILODEAU/Examiner, Art Unit 2664
/JENNIFER MEHMOOD/Supervisory Patent Examiner, Art Unit 2664