DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment / Arguments
Applicant’s response and amendment to the independent claims have been fully considered. In response, the examiner find that the addition of “greyscale” images to the claims is both taught by Applicant’s Admitted Prior Art, and a broad use of Applicant’s written description as support.
To Applicant’s written description: greyscale is found literally only two times, and in the same context, the same words: to respectfully admit that” “images in the IR and/or NIR domain are generally represented in greyscale (with different shades of grey representing different IR and/or NIR frequencies)” (quoting para. 51 or 115 as filed, which are the two places in Applicant’s specification, as filed, that have “greyscale”).
Applicant did not respectfully invent NIR or IR images, the fact they are typically displayed in grayscale/greyscale, or electromagnetic frequencies and light. Accordingly, the claims stand rejected under 103. Also, to Applicant’s newly amended: wherein the one or more trained machine learning models select user-specific colors corresponding to each of the plurality of body parts in the color representation based on the different shades of grey within the one or more greyscale images and the prior user-specific training of the one or more trained machine learning models, the support for this in Applicant’s specification is more closely related to non-elected invention of Species D (systems having a first, a second or a third set of machine learning models). Nevertheless, the prior art (Saragih) teaches training using both IR and color images. See remainder of this official action for more details.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 4, 5, 6, 8-13, 20, 23, 24 and 26-28 are rejected under 35 U.S.C. 103 as being unpatentable over Saragih (U.S. Patent App. Pub. No. 2020/0402284) in view of Sachs (U.S. Patent App. Pub. No. 2019/0122411) and AAPA (Applicant’s Admitted Prior Art)
Regarding claim 1:
Saragih teaches: an apparatus for imaging, the apparatus comprising: at least one memory; and one or more processors coupled to the at least one memory, the one or more processors configured (claim 1) to:
receive, from an image sensor (i.e. para. 31-32, receiving training images from from IR and/or RGB cameras on a headset), one or more greyscale images of a plurality of body parts of a user, wherein the image sensor captures the one or more greyscale images in a an electromagnetic (EM) frequency domain other than a visible light frequency domain (AAPA, specification as filed, paras. 51, 115, which admits that images in IR and/or NIR are generally represented in greyscale), and wherein different shades of grey within the one or more grayscale images represent different frequences in the EM frequency domain (claim 18, which teaches captured images of a user in a first EM domain (the IR domain, in combination with AAPA, paras. 51, 115), See also Saragih, claims 32-35; para. 32 describes that the image can be domains other than IR or visible light “in which images may be captured by cameras”. The images can be facial images, which have a plurality of body parts (eye, skin, nose, eyebrow, mouth, cheek, lips, etc.) (see Summary));
generate a color representation of the plurality of body parts of the user in the visible light frequency domain and in color at least in part by inputting at least the one or more greyscale images into one or more trained machine learning models (see e.g. claim 1, the “generate” function) (alternatively, see Fig.5., this can be the rendered avatar image, in combination with claim 18, second spectral domain is visible or color),
wherein the one or more trained machine learning models are specific to the user based on prior user-specific training of the one or more trained machine learning models using training data having at least one color image of the user in the visible light frequency domain and at least one greyscale image of the user in the EM frequency domain (e.g. paras. 4, 5, 29-32, training the model using images of a user’s face. See also Figs. 1A, 1B, 2 and 5 with related descriptions (for example, at para. 35: “In particular embodiments, the ML models, such as the domain-transfer ML model 114, that are used in generating an avatar for a particular user 102 may be trained based on that particular user's facial expressions.” Training data being color and NIR or greyscale is also taught and mapped above))…and
output the color representation of the plurality of body parts of the user in the visible light frequency domain (claim 1, outputting the representation is complete or met, once the generation function (see mapping above) is finished. See also Fig. 5).
Regarding: wherein the one or more trained machine learning models select user-specific colors corresponding to each of the plurality of body parts in the color representation based on the different shades of grey within the one or more greyscale images and the prior user-specific training of the one or more trained machine learning models, consider the following.
Saragih teaches a model that is trained to learn representations of body parts (i.e. facial body parts) from training images (see e.g. Fig. 5, and paras. 31-33, 44, 54, whereby para. 54 specifically describes training landmark detectors (example landmarks are shown in Fig. 7). These landmarks are facial body parts, and the training data or images can be both color and IR or greyscale. AAPA admits that greyscale images of IR/NIR are different shades of grey (original specification, paras. 51, 115)).
Likewise, Sachs also teaches generating a 3D model of a user’s face/head (corresponding to Applicant’s claimed outputted representation), using Machine Learning (see “Generating a 3D Model Using Machine Learning” section, beginning at para. 156). The trained machine learning model of Sachs is trained to learn a representation of facial images that includes distinguishing between facial body parts by learning and therefore being able to later identify taxonomy attributes from input images (see claim 1). Taxonomy attributes include global and local attributes (claim 2) such as facial body parts (claim 9 and paras. 163, 166, 183-84) and color of same (para. 184). Likewise, Sachs teaches that its machine learned model can generate the final 3D model (corresponding to Applicant’s claimed outputted representation) by identifying said taxonomy attributes (i.e. identifying a first and second user specific color for respective body parts). Colors of two different body parts being different from each other is a design feature of the facial images. See also Sachs, claim 3, texture synthesis based on taxonomy attributes; this also teaches user-specific colors, as the texture processing related to final color of body part (for more detailed exemplary description, see paras. 108-119).
Sachs also teaches many body parts, including but not limited to: a patch of hair (e.g. Fig. 12 : hair belonging to a hairline of a user’s face is a patch of hair; another example: Fig. 19: patch of hair identified as covering an eye; another example: para. 88: facial hair is another example of a patch of hair; another example: para. 103; another example: para. 106: strands of hair). And likewise, Sachs teaches non-hair body parts (see same mapping for hair patches, these sections also teach skin or areas of the face that aren’t hair; another example: para. 109: eye; another example: para. 21; another example: para. 105, skin across nose bridge; para. 106-07, skin; para. 121: eyes, lip corners); another example: para. 127, iris of eyes)).
Modifying the applied refs, in view of Saragih and Sachs, such that the in the training using images of a user’s face (per Saragih), colors representative of the user’s body parts (Saragih, and more specifically in Sachs), for training as per both references, and using color and IR images, is all of taught and suggested by the prior art, and would have been obvious and predictable to one of ordinary skill as of the effective filing date of Applicant’s claims. See MPEP §2143(A).
The prior art included each element recited in claim 1, although not necessarily in a single embodiment, with the only difference being between the claimed element and the prior art being the lack of actual combination of certain elements in a single prior art embodiment, as described above.
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention.
Regarding claim 4:
Saragih teaches: the apparatus of claim 1, wherein, to output the representation, the one or more processors are configured to input the color representation into the second set of one or more machine learning models,
wherein the second set of one or more machine learning models are configured to generate a three-dimensional mesh for an avatar of the user and a texture to apply to the three-dimensional mesh for the avatar of the user based on input of the representation into the second set of one or more machine learning models (Fig. 5 and related description. The second trained model is 206, the first can be 114).
Modifying Saragih, in view of same, to have included the above, would have been obvious for one of ordinary skill in the art as of the effective filing date of the claimed invention, motivated to have flexibility with regard to spectral domain imaging and transfer.
Regarding claim 5:
It would have been obvious for one of ordinary skill in the art to have further modified the applied reference(-s), in view of same, to have obtained: the apparatus of claim 1, wherein the frequency domain includes at least one of an ultraviolet (UV) frequency domain or a near-ultraviolet (NUV) frequency domain, and the results of the modification would have been obvious and predictable to one of ordinary skill in the art as of the effective filing date of the claimed invention. See MPEP §2143(A).
Saragih isn’t limited to what the domains are with regard to transferring images between different spectral domains. In fact, Saragih teaches that the “Spectral domains may include infrared, visible light, or other domains in which images may be captured by cameras” (quoting part of para. 32). This includes the UV or NUV frequency domain, as any other domain in which images may be captured by cameras. Modifying Saragih, in view of itself, to have included the above, is all of taught and suggested by Saragih, and would have been obvious and predictable to one of ordinary skill in the art.
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention.
Regarding claim 6:
Saragih teaches: the apparatus of claim 5, wherein the EM frequency domain includes least one of an infrared (IR) frequency domain or a near-infrared (NIR) frequency domain (see e.g. claim 18).
Modifying Saragih, in view of same, to have included the above, would have been obvious for one of ordinary skill in the art as of the effective filing date of the claimed invention, motivated to have flexibility with regard to spectral domain imaging and transfer.
Regarding claim 8:
Saragih teaches: the apparatus of claim 1, wherein the one or more greyscale images of the user depict the user in a pose (paras. 30-36, see also Fig. 5),
wherein the color representation represents the user in the pose (paras. 30-36; see also Fig. 5), and
wherein the pose includes at least one of a position of at least a part of the user, an orientation of at least the part of the user, or a facial expression of the user (see above mappings).
Regarding claim 9:
Saragih teaches: the apparatus of claim 1, wherein the color representation of includes a color texture in the visible light frequency domain (e.g. para. 35, Fig. 5: 516, 506 and related description),
wherein the color texture is configured to apply to a three-dimensional mesh representation of the user (Fig. 5: 516, 514, 115, 506 and related description).
Modifying Saragih, in view of same, to have included the above, would have been obvious for one of ordinary skill in the art as of the effective filing date of the claimed invention, motivated to have flexibility with regard to spectral domain imaging and transfer.
Regarding claim 10:
Saragih teaches: the apparatus of claim 1, wherein the color representation includes a three-dimensional model of the user that is textured using a color texture that includes a plurality of colors in the visible light frequency domain (see e.g. paras. 28 and Fig. 5 and related description).
Modifying Saragih, in view of same, to have included the above, would have been obvious for one of ordinary skill in the art as of the effective filing date of the claimed invention, motivated to have flexibility with regard to spectral domain imaging and transfer.
Regarding claim 11:
Saragih teaches: the apparatus of claim 1, wherein the color representation includes a color rendered image of a three-dimensional model of the user and from a predetermined perspective, wherein the color rendered image includes a plurality of colors in the visible light frequency domain (see e.g. para. 35 and Fig. 5 and related description).
Modifying Saragih, in view of same, to have included the above, would have been obvious for one of ordinary skill in the art as of the effective filing date of the claimed invention, motivated to have flexibility with regard to spectral domain imaging and transfer.
Regarding claim 12:
Saragih teaches: the apparatus of claim 1, wherein the color representation includes a color image of the user that includes a plurality of colors in the visible light frequency domain (e.g. paras. 32-35; see also Fig. 12: 1210, 1220).
Modifying Saragih, in view of same, to have included the above, would have been obvious for one of ordinary skill in the art as of the effective filing date of the claimed invention, motivated to have flexibility with regard to spectral domain imaging and transfer.
Regarding claim 13:
Saragih and Sachs teach: the apparatus of claim 1, wherein the trained machine learning model selects a first user-specific color for a patch of hair on the user and a second user-specific color for a non-hair body part of the user that includes at least one of skin or iris (Regarding specifically that the body parts are: (1) a patch of hair having a user-specific hair color; and (2) a non-hair body part having a second user-specific color, see mapping to claim 1 and Sachs reference. This is mapped in claim 1),
wherein the second user-specific color is one of a skin color of the user or an iris color of the user, (see mapping to claim 1, and Sachs reference), and wherein the plurality of body parts includes the patch of hair and the non-hair body part (see mapping to claim 1).
Modifying the applied references, in view of same, to have included the above, would have been obvious for one of ordinary skill in the art as of the effective filing date of the claimed invention, motivated to have flexibility with regard to spectral domain imaging and transfer.
Regarding claim 20: see also claim 1.
The method of claim 20 corresponds to what is performed by the apparatus of claim 1. The same rationale for rejection applies.
Regarding claim 23: see claim 4.
These claims are similar; the same rationale for rejection applies.
Regarding claim 24: see claim 5.
These claims are similar; the same rationale for rejection applies.
Regarding claim 26: see claim 10.
These claims are similar; the same rationale for rejection applies.
Regarding claim 27: see claim 11.
These claims are similar; the same rationale for rejection applies.
Regarding claim 28: see claim 13.
These claims are similar; the same rationale for rejection applies.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure relevant to imaging domains.
* * * * *
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Sarah Lhymn whose telephone number is (571)270-0632. The examiner can normally be reached M-F, 9:00 AM to 6:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached on 571-272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Sarah Lhymn
Primary Examiner
Art Unit 2613
/Sarah Lhymn/Primary Examiner, Art Unit 2613