DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 50-62 and 67-69 are rejected under 35 U.S.C. 103 as being unpatentable over Hu (NPL: "A new, feasible, and convenient method based on semantic segmentation and deep learning for hemoglobin monitoring.”) in view of Veturi (NPL: "SynthEye: investigating the impact of synthetic data on artificial intelligence-assisted gene diagnosis of inherited retinal disease.”).
With respect to claim 50, Hu discloses a method for training a machine learning model to analyze an image of an eye of a subject (see Abstract; and see §2.1, eye is photographed, further see §2.3-2.5 train mask R-CNN and MobileNetV3 for a number hemoglobin (Hb) estimate), comprising:
obtaining, by one or more processors, a first training set comprising a first plurality of images, wherein said first plurality of images comprises said image of said eye of said subject (see §2.1 where eye images are collected via processors; and see §2.6 which describes training/test data and computational implementation in §2.3-2.5); and
training, by said one or more processors, a machine learning model to predict a hemoglobin level of said subject from said eye of said subject using said first training set (see §2.4-2.5 which describes the training of MobileNetV3 regression from eye/palpebral images linked to blood gas Hb; and see §2.6, evaluate vs measured values of Hb).
Hu does not disclose creating, by said one or more processors, a second training set comprising a second plurality of images, wherein said second plurality of images comprises one or more generated images, wherein said one or more generated images are generated by a generative learning algorithm trained on a plurality of images each comprising an eye; and training, by said one or more processors, said machine learning model to predict said hemoglobin level using said second training set.
Veturi teaches creating, by said one or more processors, a second training set comprising a second plurality of images, wherein said second plurality of images comprises one or more generated images, wherein said one or more generated images are generated by a generative learning algorithm trained on a plurality of images each comprising an eye (see Methods: training of StyleGan2 on gene labeled eye-fundus images, generates synthetic images and formed mixed or rebalanced training sets); and training, by said one or more processors, said machine learning model to predict said hemoglobin level using said second training set (see Methods: training CNN on mixed real and synthetic data).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the trained invention to have modified Hu with the teachings of Veturi to apply Veturi’s training procedure to Hu’s eye-image corpus because it would have resulted in the predictable result of generating images that contain the required eye region to estimate hemoglobin levels (Veturi: see Methods).
With respect to claim 51, all limitations of claim 50 apply in which Hu further discloses wherein said one or more generated images correspond to a low hemoglobin level or a high hemoglobin level (see §2.6 assesses Hb thresholds of 10, 11 and 12 g/dL to correspond to low or high levels).
With respect to claim 52, all limitations of claim 50 apply in which Hu further discloses wherein said image of said eye of said subject comprises image data of a lower eyelid portion and a sclera portion (see §2.1 and 2.3 which uses uncropped eye images that show conjunctival target alongside the rest of the exposed eye).
With respect to claim 53, all limitations of claim 50 apply in which Hu further discloses further processing said image of said eye of said subject (see §2.2), wherein processing said image of said eye of said subject comprises:
filtering said image of said eye of said subject based on one or more of a yellow value, a red value, a brightness value, or a proportion contribution of said eye of said subject to said image of said eye of said subject;
denoising said image of said eye of said subject (see §2.2 denoises bright spots using K-means); or
adjusting a white balance of said image of said eye of said subject using a sclera portion of said eye of said subject.
With respect to claim 54, all limitations of claim 50 apply in which Hu further discloses generating said first plurality of images by:
determining that each of said first plurality of images contains an eye using a classification model and removing any image that does not contain an eye (see §2.1: selected eye images and §2.3 using mask R-CNN to recognize conjunctiva), and
segmenting each of said first plurality of images using a segmentation model to identify a lower eyelid portion and a sclera portion (see §2.1-2.3: segmentation).
With respect to claim 55, all limitations of claim 50 apply in which Hu further discloses training a classification model to predict the presence of an eye in each image of said first plurality of images (see §2.3: classifies and segments conjunctiva).
With respect to claim 56, all limitations of claim 50 apply in which Hu further discloses further comprising training a segmentation model to identify a plurality of pixels corresponding to a lower eyelid portion of each said first plurality of images (see §2.3-2.5: trains mask R-CNN to delineate palpebral conjunctiva from eye images).
With respect to claim 57, all limitations of claim 50 apply in which Hu further discloses wherein each image of said first training set is associated with a known hemoglobin level (see §2.1 and 2.6: pairs eye images with arterial blood gab Hb and trains against those measured values).
With respect to claim 58, all limitations of claim 50 apply in which Veturi further teaches wherein said one or more generated image comprises a generated lower eyelid portion (see Methods: synthetic fundus images).
With respect to claim 59, all limitations of claim 50 apply in which Veturi further teaches wherein said second plurality of images comprises a near uniform distribution of possible labels in said first training set (see Methods: rebalances discrete gene classes).
With respect to claim 60, all limitations of claim 50 apply in which Veturi further teaches wherein said one or more generated image comprises a new image outside said first training set (see Methods: synthetic fundus images and assesses whether they merely copy training images).
With respect to claim 61, all limitations of claim 60 apply in which Veturi further teaches wherein said generative learning algorithm comprises one or both of a variational autoencoder (VAE) or a generative adversarial network (GAN) (see Methods: StyleGan2 generative adversarial network).
With respect to claim 62, all limitations of claim 50 apply in which the combination of Hu and Veturi further teaches performing operations (a)-(d) until said machine learning model is trained to predict hemoglobin levels to within about 1.56 grams per deciliter of a true hemoglobin levels (Hu: see §2.1-2.6 mean absolute error of 1.6 g/dL).
With respect to claim 67, all limitations of claim 50 apply in which Hu further discloses providing said predicted hemoglobin level to an interface of a user device (see §2.1-2.6: reports Hb predictions to smartphone of user).
With respect to claim 68, all limitations of claim 50 apply in which Hu further discloses further comprising determining a hemoglobin level for a second subject based on an image of an eye of said second subject (see §2.5-2.6: evaluated model predictions on held-out eye images with independently measured Hb).
With respect to claim 69, all limitations of claim 68 apply in which Hu further discloses providing an indication of an anemic condition to said second subject (see §2.1-2.6: assessing anemia at thresholds) and recommending a dietary change to said second subject to treat said anemic condition (see §2.1-2.6).
Claims 63-66 are rejected under 35 U.S.C. 103 as being unpatentable over Hu in view of Veturi as applied to claim 50 above, and further in view of Zhao (NPL: "Deep-learning-based hemoglobin concentration prediction and anemia screening using ultra-wide field fundus images.”).
With respect to claim 63, all limitations of claim 50 apply in which Hu and Veturi do not disclose wherein said second plurality of images comprises an augmented image, wherein said augmented image is based on an image of said first plurality of images.
Zhao teaches an augmented image, wherein said augmented image is based on an image of said first plurality of image (see Materials and Methods: routine augmentation of images based on training dataset).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hu and Veturi with the teachings of Zhang to have included augmented images because it would have resulted in the predictable result of increasing the training dataset and reducing the possibility of overfitting (Zhao: see Materials and Methods).
With respect to claim 64, all limitations of claim 63 apply in which Zhao further teaches further comprising generating said augmented image by rotating (see Materials and Methods: augmentation methods include rotation), scaling, shifting, translating, or color adjusting said image of said first plurality of images.
With respect to claim 65, all limitations of claim 50 apply in which Hu and Veturi do not disclose wherein one or both of said first training set or second training set comprise medical information of said subject.
Zhao teaches wherein one or both of said first training set or second training set comprise medical information of said subject (see Materials and Methods: integration of demographic information such as age and gender to training dataset).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hu and Veturi with the teachings of Zhao to include medical information of a subject to training data because it would have resulted in the predictable result of screening for anemia with the prediction of hemoglobin concentration (Zhao: see Materials and Methods).
With respect to claim 66, all limitations of claim 65 apply in which Zhao further discloses wherein said known medical information comprises one or both of demographic data or medical history (see Materials and Methods: integration of demographic information such as age and gender to training dataset).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NIDHI PATEL whose telephone number is (571)272-2379. The examiner can normally be reached Mondays to Fridays 9AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Robertson can be reached at (571) 272-5001. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.N.P./Examiner, Art Unit 3791
/JENNIFER ROBERTSON/Supervisory Patent Examiner, Art Unit 3791
