DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference character “1305” has been used to designate both vitamin D production and type and quantity of sun screen cream in fig. 13. Type and quantity of sun screen cream should be designated by 1306 in accordance with para. [0112] of the published application. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description: 702 & 704 in fig. 7 1206 in fig. 12 & 1303 in fig. 13. Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
Claim 7 is objected to because of the following informalities: “a extent” in claim 7 line 4 should recite “an extent”. Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “electronic device” in claims 1-2, 4, and 7-8.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
The electronic device is defined in para. [0062-0067] as a laptop, a palmtop, a desktop, a mobile phone, a smartphone, Personal Digital Assistant (PDA), a tablet, a wearable device, an Internet of Things (IoT) device, a virtual reality device, a foldable device, a flexible device, an immersive system.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-15 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 line 1 and Claim 9 line 1 (claims 2-8 and 10-15 by virtue of dependency) recites the limitation "the skin". There is insufficient antecedent basis for this limitation in the claims. The limitation is suggested to recite “a skin” or “information of skin”.
Claim 1 line 10 and Claim 9 line 12 (claims 2-8 and 10-15 by virtue of dependency) recites the limitation “the wavelength bands”. There is insufficient antecedent basis for this limitation in the claims. The limitation is suggested to recite “the at least one wavelength band”.
Claim 4 line 2 and Claim 12 line 2 recites the limitation “the wavelength bands”. There is insufficient antecedent basis for this limitation in the claims. The limitation is suggested to recite “the at least one wavelength band”.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-15 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
The claims recite:
capturing, by the electronic device, a Red, Green, and Blue (RGB) image of a skin of a user;
converting, by the electronic device, the RGB image into a hyper spectral image;
determining, by the electronic device, at least one wavelength band by applying a wavelength reflectance model on the hyper spectral image; and
determining, by the electronic device, information of the skin by applying a neural network model on the wavelength bands.
The above claim limitations (determining … wavelength band …) constitute an abstract idea that is part of the Mathematical Concepts and/or Mental Processes group identified in the 2019 Revised Patent Subject Matter Eligibility Guidance published in the Federal Register (84 FR 50) on January 7, 2019.
“A mathematical relationship is a relationship between variables or numbers. A mathematical relationship may be expressed in words ….” October 2019 Update: Subject Matter Eligibility, II. A. i. “[T]here are instances where a formula or equation is written in text format that should also be considered as falling within this grouping.” Id. at II. A. ii. “[A] claim does not have to recite the word “calculating” in order to be considered a mathematical calculation.” Id. at II. A. iii. See for example, SAP Am., Inc. v. InvestPic, LLC, 898 F.3d 1161, 1163-65 (Fed. Cir. 2018).
The claimed steps of determining, by the electronic device, at least one wavelength band by applying a wavelength reflectance model on the hyper spectral image recite a mathematical concept (i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations).
The step of “determining, by the electronic device, at least one wavelength band by applying a wavelength reflectance model on the hyper spectral image” in independent Claims 1 and 9 is a mathematical relationship between the concentration of pigments in a tissue and wavelengths (skin, hair, blood) at which reflection data is recorded.
Referring to para. [0094-0097] of the specification, determining the concentrations of pigments be computed from the Beer Lambert law equation.
The claimed steps of determining can be practically performed in the human mind using mental steps or basic critical thinking, which are types of activities that have been found by the courts to represent abstract ideas.
“[T]he ‘mental processes’ abstract idea grouping is defined as concepts performed in the human mind, and examples of mental processes include observations, evaluations, judgments, and opinions.” MPEP 2106.04(a)(2) III. The pending claims merely recite steps for determining skin information that include observations, evaluations, and judgments.
Examples of ineligible claims that recite mental processes include:
a claim to “collecting information, analyzing it, and displaying certain results of the collection and analysis,” where the data analysis steps are recited at a high level of generality such that they could practically be performed in the human mind, Electric Power Group, LLC v. Alstom, S.A.;
claims to “comparing BRCA sequences and determining the existence of alterations,” where the claims cover any way of comparing BRCA sequences such that the comparison steps can practically be performed in the human mind, University of Utah Research Foundation v. Ambry Genetics Corp.
a claim to collecting and comparing known information, which are steps that can be practically performed in the human mind, Classen Immunotherapies, Inc. v. Biogen IDEC.
See p. 7-8 of October 2019 Update: Subject Matter Eligibility.
Regarding the dependent claims, the dependent claims are directed to either 1) steps that are also abstract (claims 2-8 and 10-15 are directed to more abstract ideas) or 2) additional data gathering/output that is well-understood, routine and previously known to the industry. Although the dependent claims are further limiting, they do not recite significantly more than the abstract idea. A narrow abstract idea is still an abstract idea and an abstract idea with additional well-known equipment/functions is not significantly more than the abstract idea.
This judicial exception (abstract idea) in Claims 1-15 is not integrated into a practical application because:
The abstract idea amounts to simply implementing the abstract idea on a computer. For example, the recitations regarding the generic computing components for determining merely invoke a computer as a tool.
The data-gathering step (capturing, converting) and the data-output step do not add a meaningful limitation to the method as they are insignificant extra-solution activity.
There is no improvement to a computer or other technology. “The McRO court indicated that it was the incorporation of the particular claimed rules in computer animation that "improved [the] existing technological process", unlike cases such as Alice where a computer was merely used as a tool to perform an existing process.” MPEP 2106.05(a) II. The claims recite a computer that is used as a tool for determining.
The claims do not apply the abstract idea to affect a particular treatment or prophylaxis for a disease or medical condition. Rather, the abstract idea is utilized for determining skin information.
The claims do not apply the abstract idea to a particular machine. “Integral use of a machine to achieve performance of a method may provide significantly more, in contrast to where the machine is merely an object on which the method operates, which does not provide significantly more.” MPEP 2106.05(b). II. “Use of a machine that contributes only nominally or insignificantly to the execution of the claimed method (e.g., in a data gathering step or in a field-of-use limitation) would not provide significantly more.” MPEP 2106.05(b) III. The pending claims utilize a computer for providing, determining. The claims do not apply the obtained calculation to a particular machine. Rather, the data is merely output in a post-solution step.
The additional elements are identified as follows: electronic device; memory; processor; skin details detector/circuitry.
Those in the relevant field of art would recognize the above-identified additional elements as being well-understood, routine, and conventional means for data-gathering and computing, as demonstrated by
Applicant’s specification (e.g., paragraphs [0062-0063]) which discloses that the electronic device, processor(s)/memory comprise generic computer components that are configured to perform the generic computer functions (e.g., determining) that are well-understood, routine, and conventional activities previously known to the pertinent industry.
Applicant’s Background in the specification;
The Non-Patent Literature of record:
Gevaux L, Gierschendorf J, Rengot J, et al. Real-time skin chromophore estimation from hyperspectral images using a neural network. Skin Res Technol. 2021; 27: 163–177. https://doi.org/10.1111/srt.12927;
Kim et al., Data-driven imaging of tissue inflammation using RGB-based hyperspectral reconstruction toward personal monitoring of dermatologic health. Biomed Opt Express. 2017 Oct 26;8(11):5282-5296. doi: 10.1364/BOE.8.005282;
HE et al., "Hyperspectral imaging enabled by an unmodified smartphone for analyzing skin morphological features and monitoring hemodynamics", February 2020, 16 pages;
Gevaux et al., (2020). Multispectral and Hyperspectral Imaging for Skin Acquisition and Analysis. In: Fimiani, M., Rubegni, P., Cinotti, E. (eds) Technology in Practical Dermatology. Springer, Cham. https://doi.org/10.1007/978-3-030-45351-0_26;
SHARMA et al., "Hyperspectral Reconstruction from RGB Images for Vein Visualization", June 8-11, 2020, 11 pages;
SEROUL et al., "Hyperspectral imaging system for in-vivo quantification of skin pigments", October 2014, 11 pages.
Thus, the claimed additional elements “are so well-known that they do not need to be described in detail in a patent application to satisfy 35 U.S.C. § 112(a).” Berkheimer Memorandum, III. A. 3.
Furthermore, the court decisions discussed in MPEP § 2106.05(d)(lI) note the well-understood, routine and conventional nature of such additional generic computer components as those claimed. See option III. A. 2. in the Berkheimer memorandum.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the units associated with the steps do not add meaningful limitation to the abstract idea. A computer, processor, memory, or equivalent hardware is merely used as a tool for executing the abstract idea(s). The process claimed does not reflect an improvement in the functioning of the computer.
When considered in combination, the additional elements (i.e., the generic computer functions and conventional equipment/steps) do not amount to significantly more than the abstract idea. Looking at the claim limitations as a whole adds nothing that is not already present when looking at the elements taken individually. There is no indication that the combination of elements improves the functioning of a computer or improves any other technology. Their collective functions merely provide conventional computer implementation.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-4, 6, 9-12, and 14 are under 35 U.S.C. 103 as being unpatentable over He (He et al., "Hyperspectral imaging enabled by an unmodified smartphone for analyzing skin morphological features and monitoring hemodynamics", February 2020, 16 pages) in view of Gevaux (Gevaux et al. Real-time skin chromophore estimation from hyperspectral images using a neural network. Skin Res Technol. 2021; 27: 163–177. https://doi.org/10.1111/srt.12927).
Regarding claim 1, He discloses a method for determining information of the skin by an electronic device using hyper spectral reconstruction (pages 895-896, Introduction, “unmodified smartphone … RGB images … reconstruct … melanin absorption etc., within the skin”), wherein the method comprises: capturing, by the electronic device (“smartphone”, page 896, Introduction), a Red, Green, and Blue (RGB) image of a skin of a user (page 896, Introduction, “RGB images captured by the built-in camera … smartphone”); converting, by the electronic device, the RGB image into a hyper spectral image (pages 896-897, 2.1., Reconstruction principle from RGB images to hyperspectral images, “Wiener estimation algorithm to perform hyperspectral reconstruction from RGB images”); determining at least one wavelength band (page 898-899, 2.3. Hyperspectral reconstruction and post-processing, “extracted spatial absorption information of skin chromophores, e.g. melanin and hemoglobin, through a series of processing steps on images representing different wavebands … weighted subtraction … red light wavebands”, see equation 9), and determining by the electronic device, information of the skin based on the wavelength bands (extraction of blood and melanin information content from hyperspectral reconstruction with the RGB images captured by a smartphone under the fluorescent lamp illumination … Blood and melanin absorption maps, fig. 5 & 3.2. Skin morphological feature analysis).
He does not disclose determining, by the electronic device, at least one wavelength band applying a wavelength reflectance model; and determining, by the electronic device, information of the skin by applying a neural network model on the wavelength bands.
However, Gevaux directed to real-time skin chromophore estimation from hyperspectral images using a neural network discloses determining, by the electronic device (page 170, “CPU … computer”), at least one wavelength band applying a wavelength reflectance model (figs. 3 & 6, page 166-168, 3 Skin Analysis Using Optic Al Modeling And Optimization, “Beer-Lambert-Bouguer law … Kubelka-Munk theory … direct relationship between skin spectral reflectanceRskin, skin chromophore concentrations {cmel, cHbO2, cHb, cbi}, and epidermis thickness h” & fig. 3 (Examiner note: see para. [0094] of the instant application specification)); and determining information of the skin by applying a neural network model on the wavelength bands (page 169, 4.2 Neural network architecture, “neural network … wavelength … melanin concentration, epidermis thickness” & page 174, figs. 5-7). Gevaux further discloses that the implementation of a neural network method for the estimation of skin parameter maps can drastically reduce calculation time, allowing for the visualization of results immediately after acquisition (page 176, Conclusion).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify He such that the method further comprises determining, by the electronic device, at least one wavelength band applying a wavelength reflectance model; and determining, by the electronic device, information of the skin by applying a neural network model on the wavelength bands, in view of the teachings of Gevaux for the obvious advantage of reducing the calculation time of skin parameter maps to allow for the visualization of results immediately after acquisition using a skin analysis algorithm.
Regarding claim 2, He, as modified by Gevaux hereinabove, discloses the method as claimed in claim 1.
He does not disclose wherein determining, by the electronic device, the at least one wavelength band by applying the wavelength reflectance model on the hyper spectral image comprises: segmenting, by the electronic device, different tissues under the skin non-invasively from the hyper spectral image using the wavelength reflectance model; extracting, by the electronic device, spectra of each individual tissue of the different tissues under the skin by analysing multiple pixels on the hyper spectral image using the wavelength reflectance model; and determining, by the electronic device, the at least one wavelength band comprising a concentration of pigments in the different tissues based on the extracted spectra of the individual tissues.
However, Gevaux discloses wherein determining, by the electronic device (page 170, “CPU … computer”), the at least one wavelength band by applying the wavelength reflectance model on the hyper spectral image (page 166-168, 3 Skin Analysis Using Optic Al Modeling And Optimization, “Beer-Lambert-Bouguer law … Kubelka-Munk theory … direct relationship between skin spectral reflectanceRskin, skin chromophore concentrations {cmel, cHbO2, cHb, cbi}, and epidermis thickness h” & figs. 3 & 6) comprises: segmenting, by the electronic device, different tissues under the skin non- invasively from the hyper spectral image using the wavelength reflectance model (figs. 3-4, page 166-168, 3 Skin Analysis Using Optic Al Modeling And Optimization, “epidermis and dermis … functions of the layers’ absorption”); extracting, by the electronic device, spectra of each individual tissue of the different tissues under the skin by analysing multiple pixels on the hyper spectral image using the wavelength reflectance model (figs. 3-4 & 6, page 166-168, page 165-166, “Figure 3 … melanin spots … blue wavelengths (420 nm), blood vessels … 490 nm and the skin appearing very uniform in red wavelengths (700 nm)”; “for each pixel, the parameters … determined”); and determining, by the electronic device, the at least one wavelength band comprising a concentration of pigments in the different tissues based on the extracted spectra of the individual tissues (figs. 3-4, page 166-168, 3 Skin Analysis Using Optic Al Modeling And Optimization “skin chromophore is characterized … concentration: the quantities {cmel, cHbO2, cHb, cbi}”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify He, as modified by Gevaux hereinabove, such that determining, by the electronic device, the at least one wavelength band by applying the wavelength reflectance model on the hyper spectral image comprises segmenting, by the electronic device, different tissues under the skin non-invasively from the hyper spectral image using the wavelength reflectance model; extracting, by the electronic device, spectra of each individual tissue of the different tissues under the skin by analysing multiple pixels on the hyper spectral image using the wavelength reflectance model; and determining, by the electronic device, the at least one wavelength band comprising a concentration of pigments in the different tissues based on the extracted spectra of the individual tissues, in view of the teachings of Gevaux, as such a modification would have yielded predictable results, namely characterizing each skin chromophore by its concentration of melanin, oxyhemoglobin, deoxyhemoglobin, bilirubin, and its epidermis thickness.
Regarding claim 3, He, as modified by Gevaux hereinabove, discloses the method as claimed in claim 2, wherein the concentration of the pigments in the different tissues under the skin comprises information related to at least one of thickness of the skin, melanin concentration in the skin, bilirubin concentration, hair thickness under the skin, blood vessel thickness under the skin, a hemoglobin (Hb) concentration under the skin, and oxygenated hemoglobin (HBO2) concentration under the skin (He, page 901-902 & 906-907, 3.2. Skin morphological feature analysis, “hemoglobin absorption information”; 4. Discussions, “blood, melanin absorption maps and oxygen saturation” & Gevaux, page 165, 3. Skin Analysis Using Optic Al Modeling and Optimization, “oxygen rate, blood volume fraction, melanin concentration, bilirubin concentration, and epidermis thickness”).
Regarding claim 4, He, as modified by Gevaux hereinabove, discloses the method as claimed in claim 2. He does not disclose wherein determining, by the electronic device, the information of the skin by applying the neural network model on the wavelength bands comprises: inputting, by the electronic device, concentration of the pigments in the different tissues under the skin to the neural network model; and obtaining, by the electronic device, the information of the skin from the neural network model.
However, Gevaux discloses wherein determining, by the electronic device (page 170, “CPU … computer”), the information of the skin by applying the neural network model on the wavelength bands comprises: inputting, by the electronic device, concentration of the pigments in the different tissues under the skin to the neural network model (pages 167-168, 4. Skin Analysis Using A Neural Network Method, “measured skin spectral reflectance on one pixel corresponds to the input data of the neural network”); and obtaining, by the electronic device, the information of the skin from the neural network model (pages 167-168, 4. Skin Analysis Using A Neural Network Method, “output values … correspond to the calculated skin parameters for the pixel, that is, oxygen rate, blood volume fraction, melanin concentration, and epidermis thickness”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify He such that determining, by the electronic device, the information of the skin by applying the neural network model on the wavelength bands comprises: inputting, by the electronic device, concentration of the pigments in the different tissues under the skin to the neural network model; and obtaining, by the electronic device, the information of the skin from the neural network model, in view of the teachings of Gevaux for the obvious advantage of reducing the calculation time of skin parameter maps to allow for the visualization of results immediately after acquisition using a skin analysis algorithm.
Regarding claim 6, He, as modified by Gevaux hereinabove, discloses the method as claimed in 1, wherein the RGB image is captured by at least one of an imaging apparatus with limited spectral resolution (Abstract, “built-in RGB camera … unmodified smartphone”).
Regarding claim 9, He discloses an electronic device configured to determine information of skin using hyper spectral reconstruction (pages 895-896, Introduction, “unmodified smartphone … RGB images … reconstruct … melanin absorption etc., within the skin”), the electronic device (Abstract, unmodified smartphone) comprising: a memory; a processor (Abstract, “smartphone” (Examiner note: a smartphone comprises a memory and a processor)); and a skin details detector comprising circuitry (Abstract, “smartphone … built-in RGB camera”), operably coupled to the memory and the processor (Abstract), configured to: capture a Red, Green, and Blue (RGB) image of a skin (page 896, Introduction, “RGB images captured by the built-in camera … smartphone”); convert the RGB image into a hyper spectral image (pages 896-897, 2.1., Reconstruction principle from RGB images to hyperspectral images, “Wiener estimation algorithm to perform hyperspectral reconstruction from RGB images”); determine at least one wavelength band (page 898-899, 2.3. Hyperspectral reconstruction and post-processing, “extracted spatial absorption information of skin chromophores, e.g. melanin and hemoglobin, through a series of processing steps on images representing different wavebands … weighted subtraction … red light wavebands”, see equation 9), and determine information of the skin based on the wavelength bands (extraction of blood and melanin information content from hyperspectral reconstruction with the RGB images captured by a smartphone under the fluorescent lamp illumination … Blood and melanin absorption maps, fig. 5 & 3.2. Skin morphological feature analysis).
He does not disclose determining at least one wavelength band by applying a wavelength reflectance model on the hyper spectral image; and determining information of the skin by applying a neural network model on the wavelength bands.
However, Gevaux directed to real-time skin chromophore estimation from hyperspectral images using a neural network on a computer (page 170, “CPU … computer”) discloses determining at least one wavelength band by applying a wavelength reflectance model on the hyper spectral image (figs. 3 & 6, page 166-168, 3 Skin Analysis Using Optic Al Modeling And Optimization, “Beer-Lambert-Bouguer law … Kubelka-Munk theory … direct relationship between skin spectral reflectanceRskin, skin chromophore concentrations {cmel, cHbO2, cHb, cbi}, and epidermis thickness h” & fig. 3 (Examiner note: see para. [0094] of the instant application specification)); and determining information of the skin by applying a neural network model on the wavelength bands (page 169, 4.2 Neural network architecture, “neural network … wavelength … melanin concentration, epidermis thickness” & page 174, figs. 5-7). Gevaux further discloses that the implementation of a neural network method for the estimation of skin parameter maps can drastically reduce calculation time, allowing for the visualization of results immediately after acquisition (page 176, Conclusion).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify He such that the skin details detector is further configured to determine at least one wavelength band by applying a wavelength reflectance model on the hyper spectral image; and determine information of the skin by applying a neural network model on the wavelength bands, in view of the teachings of Gevaux for the obvious advantage of reducing the calculation time of skin parameter maps to allow for the visualization of results immediately after acquisition using a skin analysis algorithm.
Regarding claim 10, He, as modified by Gevaux hereinabove, discloses the electronic device as claimed in claim 9. He, as modified by Gevaux hereinabove, does not disclose wherein determining the at least one wavelength band by applying the wavelength reflectance model on the hyper spectral image comprises: segmenting different tissues under the skin non-invasively from the hyper spectral image using the wavelength reflectance model; extracting spectra of each individual tissue of the different tissues under the skin by analysing multiple pixels on the hyper spectral image using the wavelength reflectance model; and determining the at least one wavelength band comprising a concentration of pigments in the different tissues based on the extracted spectra of the individual tissues.
However, Gevaux discloses wherein determining the at least one wavelength band by applying the wavelength reflectance model on the hyper spectral image (page 166-168, 3 Skin Analysis Using Optic Al Modeling And Optimization, “Beer-Lambert-Bouguer law … Kubelka-Munk theory … direct relationship between skin spectral reflectanceRskin, skin chromophore concentrations {cmel, cHbO2, cHb, cbi}, and epidermis thickness h” & figs. 3 & 6) comprises: segmenting different tissues under the skin non-invasively from the hyper spectral image using the wavelength reflectance model (figs. 3-4, page 166-168, 3 Skin Analysis Using Optic Al Modeling And Optimization, “epidermis and dermis … functions of the layers’ absorption”); extracting spectra of each individual tissue of the different tissues under the skin by analysing multiple pixels on the hyper spectral image using the wavelength reflectance model (figs. 3-4 & 6, page 166-168, page 165-166, “Figure 3 … melanin spots … blue wavelengths (420 nm), blood vessels … 490 nm and the skin appearing very uniform in red wavelengths (700 nm)”; “for each pixel, the parameters … determined”); and determining the at least one wavelength band comprising a concentration of pigments in the different tissues based on the extracted spectra of the individual tissues (figs. 3-4, page 166-168, 3 Skin Analysis Using Optic Al Modeling And Optimization “skin chromophore is characterized … concentration: the quantities {cmel, cHbO2, cHb, cbi}”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify He, as modified by Gevaux hereinabove, such that determining the at least one wavelength band by applying the wavelength reflectance model on the hyper spectral image comprises: segmenting different tissues under the skin non-invasively from the hyper spectral image using the wavelength reflectance model; extracting spectra of each individual tissue of the different tissues under the skin by analysing multiple pixels on the hyper spectral image using the wavelength reflectance model; and determining the at least one wavelength band comprising a concentration of pigments in the different tissues based on the extracted spectra of the individual tissues, in view of the teachings of Gevaux, as such a modification would have yielded predictable results, namely characterizing each skin chromophore by its concentration of melanin, oxyhemoglobin, deoxyhemoglobin, bilirubin, and its epidermis thickness.
Regarding claim 11, He, as modified by Gevaux hereinabove, discloses the electronic device as claimed in claim 10, wherein the concentration of the pigments in the different tissues under the skin comprises information related to at least one of a thickness of the skin, a Melanin concentration in the skin, a Bilirubin concentration, a hair thickness under the skin, a Blood vessel thickness under the skin, a hemoglobin (Hb) concentration under the skin, and a oxygenated hemoglobin (HBO2) concentration under the skin (He, page 901-902 & 906-907, 3.2. Skin morphological feature analysis, “hemoglobin absorption information”; 4. Discussions, “blood, melanin absorption maps and oxygen saturation” & Gevaux, page 165, 3. Skin Analysis Using Optic Al Modeling and Optimization, “oxygen rate, blood volume fraction, melanin concentration, bilirubin concentration, and epidermis thickness”).
Regarding claim 12, He, as modified by Gevaux hereinabove, discloses the electronic device as claimed in claim 10. He, as modified by Gevaux hereinabove, does not disclose wherein determining information of the skin by applying the neural network model on the wavelength bands comprises: inputting concentration of the concentration of pigments in the different tissues under the skin to the neural network model; and obtaining the information of the skin from the neural network model.
However, Gevaux discloses wherein determining information of the skin by applying the neural network model on the wavelength bands comprises: inputting concentration of the concentration of pigments in the different tissues under the skin to the neural network model (pages 167-168, 4. Skin Analysis Using A Neural Network Method, “measured skin spectral reflectance on one pixel corresponds to the input data of the neural network”); and obtaining the information of the skin from the neural network model (pages 167-168, 4. Skin Analysis Using A Neural Network Method, “output values … correspond to the calculated skin parameters for the pixel, that is, oxygen rate, blood volume fraction, melanin concentration, and epidermis thickness”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify He such that determining information of the skin by applying the neural network model on the wavelength bands comprises: inputting concentration of the concentration of pigments in the different tissues under the skin to the neural network model; and obtaining the information of the skin from the neural network model, in view of the teachings of Gevaux for the obvious advantage of reducing the calculation time of skin parameter maps to allow for the visualization of results immediately after acquisition using a skin analysis algorithm.
Regarding claim 14, He, as modified by Gevaux hereinabove, discloses the electronic device as claimed in claim 9, wherein the RGB image is captured by at least one of an imaging apparatus with limited spectral resolution (Abstract, “built-in RGB camera … unmodified smartphone”).
Claims 5, 8, and 13 are under 35 U.S.C. 103 as being unpatentable over He in view of Gevaux, as applied to claims 1 and 9 above, and further in view of Bandic (US 20100185064 A1).
Regrading claim 5, He, as modified by Gevaux hereinabove, discloses the method as claimed in claim 1. He, as modified by Gevaux hereinabove, does not disclose wherein the information of the skin includes at least one of skin tone, ultraviolet exposure risk, pigmentation, psoriasis, eczema and skin abnormalities.
However, Bandic directed to an image processing technique for determining a skin photo type of a captured image in a Red Green Blue (RGB) color imaging system and classification of other skin characteristics (para. [0007, 00749]) and an algorithm 150/neural network to perform the determining of a skin state (para. [0035, 0312, 0315]) discloses wherein the information of the skin includes at least one of skin tone, ultraviolet exposure risk, pigmentation, psoriasis, eczema and skin abnormalities (“properties … psoriasis … pigmentation, tone”; “abnormal skin condition”, “sun damage”, para. [0028, 0256, 0315, 0416, 0440, 0752]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify He, as modified by Gevaux hereinabove, such that the information of the skin includes at least one of skin tone, ultraviolet exposure risk, pigmentation, psoriasis, eczema and skin abnormalities, in view of the teachings of Bandic, as such a modification would have yielded predictable results, namely obtaining dermal biophysical properties by performing a spectral analysis of image data using an algorithm/neural network.
Regrading claim 8, He, as modified by Gevaux hereinabove, discloses the method as claimed in claim 1. He, as modified by Gevaux hereinabove, does not disclose wherein the method comprises: generating, by the electronic device, a skin health and disorder report by applying the wavelength reflectance model and neural network model; and performing, by the electronic device, at least one of: displaying changes in health of the skin based on the skin health and disorders report; and recommending products specific to the skin based on the skin health and disorder report.
However, Bandic directed to an image processing technique for determining a skin photo type of a captured image in a Red Green Blue (RGB) color imaging system and classification of other skin characteristics (para. [0007, 00749]) and an algorithm 150/neural network to perform the determining of a skin state (para. [0035, 0312, 0315]) discloses wherein the method comprises: generating, by the electronic device, a skin health and disorder report (“objective skin health assessment report”; pre-diagnosis 162 & skin state 185”, para. [0060, 0305-0307, 0315]) by applying the algorithm (“algorithms … analysis”, para. [0298-0300, 0305-0307]); and performing, by the electronic device (fig. 1), at least one of: displaying changes in health of the skin based on the skin health and disorders report (“skin condition … tracked … displayed”; “report of the images and skin state 158”; “observe measurable changes in skin health”, para. [0266, 0311, 0333, 0386], figs. 5-7); and recommending products specific to the skin based on the skin health and disorder report (figs. 5-7, “regimen recommendation … personalized”; “recommendations for skin care … products” para. [0262, 0411-0413]). Bandic further discloses that the skin care regiment recommendation is personalized to the skin condition of each person (para. [0262]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify He, as modified by Gevaux hereinabove, such that the method comprises: generating, by the electronic device, a skin health and disorder report by applying the wavelength reflectance model and neural network model; and performing, by the electronic device, at least one of: displaying changes in health of the skin based on the skin health and disorders report; and recommending products specific to the skin based on the skin health and disorder report, in view of the teachings of Bandic, for the obvious advantage of providing personalized skin care regimen/product recommendations based on the skin condition.
Regrading claim 13, He, as modified by Gevaux hereinabove, discloses the electronic device as claimed in claim 9. He, as modified by Gevaux hereinabove, does not disclose wherein the information of the skin includes at least one of a skin tone, an ultraviolet exposure risk, a pigmentation, a psoriasis, an eczema and a skin abnormalities.
However, Bandic directed to an image processing technique for determining a skin photo type of a captured image in a Red Green Blue (RGB) color imaging system and classification of other skin characteristics (para. [0007, 0749]) and an algorithm 150/neural network to perform the determining of a skin state (para. [0035, 0312, 0315]) discloses wherein the information of the skin includes at least one of a skin tone, an ultraviolet exposure risk, a pigmentation, a psoriasis, an eczema and a skin abnormalities (“properties … psoriasis … pigmentation, tone”; “abnormal skin condition”, “sun damage”, para. [0028, 0256, 0315, 0416, 0440, 0752]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify He, as modified by Gevaux hereinabove, such that the information of the skin includes at least one of a skin tone, an ultraviolet exposure risk, a pigmentation, a psoriasis, an eczema and a skin abnormalities, in view of the teachings of Bandic, as such a modification would have yielded predictable results, namely obtaining dermal biophysical properties by performing a spectral analysis of image data using an algorithm/neural network.
Claims 7 and 15 are under 35 U.S.C. 103 as being unpatentable over He in view of Gevaux, as applied to claims 1 and 9 above, further in view of Demirli (US 20080212894 A1), and further in view of Bandic.
Regarding claim 7, He, as modified by Gevaux hereinabove, discloses the method as claimed in claim 1. He, as modified by Gevaux hereinabove, does not disclose wherein the method further comprises: generating, by the electronic device, a hyper pigmentation report by applying the wavelength reflectance model and neural network model containing information of a extent of pigmentation; determining, by the electronic device, whether the extent of pigmentation is improving based on the hyper pigmentation report.
However, Demirli directed to the generation of images depicting the simulated aging or de-aging of skin discloses generating, by the electronic device (para. [0035], fig. 11), a hyper pigmentation report (“severity score”, para. [0057]) by applying the information of a extent of pigmentation (“degree of hyperpigmentation”, para. [0057]); determining, by the electronic device (para. [0035], fig. 11), whether the extent of pigmentation is improving based on the hyper pigmentation report (“score … used to monitor worsening or improvement of pigmentation”, para. [0057]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify He, as modified by Gevaux hereinabove, such that the method further comprises: generating, by the electronic device, a hyper pigmentation report by applying the wavelength reflectance model and neural network model containing information of a extent of pigmentation; determining, by the electronic device, whether the extent of pigmentation is improving based on the hyper pigmentation report, in view of the teachings of Demirli, as such a modification would have yielded predictable results, namely monitoring the worsening or improvement of pigmentation based on a severity score associated with the degree of hyperpigmentation.
He, as modified by Gevaux and Demirli hereinabove, does not disclose performing, by the electronic device, at least one of: recommending to the user not to change a prescription in response to determining that the extent of pigmentation is improving; recommending to the user to change the prescription in response to determining that the extent of pigmentation is not improving; and recommending to the user to stop medication and consult a doctor in response to determining that the extent of pigmentation is declining.
However, Bandic directed to an image processing technique for determining a skin photo type of a captured image in a Red Green Blue (RGB) color imaging system and classification of other skin characteristics (para. [0007, 0749]) and an algorithm 150/neural network to perform the determining of a skin state/pigmentation (para. [0035, 0312, 0315]) discloses performing, by the electronic device (fig. 1), at least one of: recommending to the user not to change a prescription in response to determining that the skin state is improving; recommending to the user to change the prescription in response to determining that the skin state is not improving; and recommending to the user to stop medication and consult a doctor in response to determining that the skin state is declining (“skin state … pigmentation”; “skin health assessment … comparing … advice on continuing, modifying, or terminating a regimen 118 … skin state 158 changed over time … healthier … shared with a practitioner … consultation”, para. [0315, 0420-0422], figs. 16-17).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify He, as modified by Gevaux and Demirli hereinabove, such that the method further comprises performing, by the electronic device, at least one of: recommending to the user not to change a prescription in response to determining that the extent of pigmentation is improving; recommending to the user to change the prescription in response to determining that the extent of pigmentation is not improving; and recommending to the user to stop medication and consult a doctor in response to determining that the extent of pigmentation is declining, in view of the teachings of Bandic, as such a modification would have yielded predictable results, namely tracking the effectiveness of a skin care product or regimen based on the skin state/pigmentation.
Regrading claim 15, He, as modified by Gevaux hereinabove, discloses the electronic device as claimed in claim 9. He, as modified by Gevaux hereinabove, does not disclose wherein the electronic device is configured to: generate a hyper pigmentation report by applying the wavelength reflectance model and neural network model containing information of an extent of pigmentation; and determine whether the extent of pigmentation is improving or in optimal range based on the hyper pigmentation report.
However, Demirli directed to the generation of images depicting the simulated aging or de-aging of skin discloses wherein the electronic device (para. [0035], fig. 11) is configured to generate a hyper pigmentation report (“severity score”, para. [0057]) by applying the information of an extent of pigmentation (“degree of hyperpigmentation”, para. [0057]); and determine whether the extent of pigmentation is improving or in optimal range based on the hyper pigmentation report (“score … used to monitor worsening or improvement of pigmentation”, para. [0057]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify He, as modified by Gevaux hereinabove, such that the method further comprises: generating, by the electronic device, a hyper pigmentation report by applying the wavelength reflectance model and neural network model containing information of a extent of pigmentation; determining, by the electronic device, whether the extent of pigmentation is improving based on the hyper pigmentation report, in view of the teachings of Demirli, as such a modification would have yielded predictable results, namely monitoring the worsening or improvement of pigmentation based on a severity score associated with the degree of hyperpigmentation.
He, as modified by Gevaux and Demirli hereinabove, does not disclose the electronic device configured to perform at least one of: recommend not changing the prescription in response to determining that the extent of pigmentation is improving or in optimal range; recommend changing the prescription in response to determining that the extent of pigmentation is not improving; and recommend stopping medication and consulting a doctor in response to determining that the extent of pigmentation is declining.
However, Bandic directed to an image processing technique for determining a skin photo type of a captured image in a Red Green Blue (RGB) color imaging system and classification of other skin characteristics (para. [0007, 0749]) and an algorithm 150/neural network to perform the determining of a skin state/pigmentation (para. [0035, 0312, 0315]) discloses the electronic device (fig. 1) configured to perform at least one of: recommend not changing the prescription in response to determining that the skin state is improving or in optimal range; recommend changing the prescription in response to determining that the skin state is not improving; and recommend stopping medication and consulting a doctor in response to determining that the skin state is declining (“skin state … pigmentation”; “skin health assessment … comparing … advice on continuing, modifying, or terminating a regimen 118 … skin state 158 changed over time … healthier … shared with a practitioner … consultation”, para. [0315, 0420-0422], figs. 16-17).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify He, as modified by Gevaux and Demirli hereinabove, such that the electronic device is configured to perform at least one of: recommend not changing the prescription in response to determining that the extent of pigmentation is improving or in optimal range; recommend changing the prescription in response to determining that the extent of pigmentation is not improving; and recommend stopping medication and consulting a doctor in response to determining that the extent of pigmentation is declining, in view of the teachings of Bandic, as such a modification would have yielded predictable results, namely tracking the effectiveness of a skin care product or regimen based on the skin state/pigmentation.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: Gevaux et al., (2020). Multispectral and Hyperspectral Imaging for Skin Acquisition and Analysis. In: Fimiani, M., Rubegni, P., Cinotti, E. (eds) Technology in Practical Dermatology. Springer, Cham. https://doi.org/10.1007/978-3-030-45351-0_26 directed to spectral imaging techniques and methods for analysis using spectral images to determine information pertaining to skin structure and composition;
Shaked (US 20200323480 A1) directed to hyperspectral imaging maps 1000 of patient tissue each representing a map of counts of a specific detected wavelength from a hyperspectral camera 10 and using a machine learning algorithm to determine which of the hyperspectral reflectance maps 1000 and their corresponding wavelengths are to be processed (para. [0066-0068]);
Kim et al., Data-driven imaging of tissue inflammation using RGB-based hyperspectral reconstruction toward personal monitoring of dermatologic health. Biomed Opt Express. 2017 Oct 26;8(11):5282-5296. doi: 10.1364/BOE.8.005282 directed to RGB-based hyperspectral reconstruction toward personal monitoring of dermatologic health; Mohamad Hani (US 20150057552 A1) directed to determining the concentration of the types of melanin based on image analysis of hyperspectral sensing;
Gala (US 20220358755 A1) directed to a hyperspectral imaging system configured to automatically recognize drugs by using an artificial intelligence algorithm.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW ELI HOFFPAUIR whose telephone number is (571)272-4522. The examiner can normally be reached Monday-Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at
http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Charles Marmor II can be reached at (571) 272-4730. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHARLES A MARMOR II/Supervisory Patent Examiner
Art Unit 3791
/A.E.H./Examiner, Art Unit 3791