DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 6th May, 2026 has been entered.
Response to Amendment
This action is in response to the amendment filed on 6th May, 2026. Claims 1, 7, 11, and 17 have been amended. Claims 2, 6, 12, and 16 have been cancelled. Claims 1, 3-5, 7-11, 13-15, and 17-20 remain rejected in the application.
Response to Arguments
Applicant's arguments with respect to Claims 1 and 11 filed on 6th May, 2026, with respect to the rejection under 35 U.S.C. § 103, regarding that the prior art does not teach the limitation(s): "acquire information about an exposure by the user to ultraviolet (UV) and sleep time, acquire information about an improvement effect of at least one cosmetic the user is using based on the information about the user's exposure to the UV and the sleep time, and control the display to display a simulation image representing the improvement effect of the at least one cosmetic the user is using according to the user's exposure to the UV and the sleep time" has been fully considered, but are moot because of new grounds for rejection. It has now been taught by the combination of Payonk, Xu, and Bandic.
Regarding arguments to Claims 3-5, 7-9, 13-15, and 17-19, they directly/indirectly depend on independent Claims 1 and 11 respectively. Applicant does not argue anything other than independent Claims 1 and 11. The limitations in those claims, in conjunction with combination, was previously established as explained.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-5, 7-9, 11, 13-15, and 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over Payonk et al. (US 20060092315 A1, previously cited), hereinafter referenced as Payonk, in view of Xu et al. (US 20220114838 A1, previously cited), hereinafter referenced as Xu, and further in view of Bandic et al. (US 20110301441 A1), hereinafter referenced as Bandic.
Regarding Claim 1, Payonk discloses a skin characteristic data visualization device (Payonk, [0105]: teaches a computer of an imaging station 200 <read on skin characteristic data visualization device> that displays the taken images of the user's face by a separate computer system) comprising:
a camera (Payonk, [0107]: teaches a camera video output);
a display configured to display a preview image of the camera (Payonk, FIG. 5 teaches a display of a user's face <read on preview image>, which is taken from the camera; [0106]: teaches the camera taking a picture at the same time a probe test is conducted); and
PNG
media_image1.png
399
456
media_image1.png
Greyscale
a processor, wherein the processor is configured to (Payonk, FIG. 14 teaches a system computer 652 <read on processor>):
acquire a measurement point which is a position on a skin where measurement is required by a measuring device (Payonk, [0103]: teaches using a probe <read on measuring device> to initiate testing over an area of interest, which is indicated by a focus box 226 <read on measurement point>, on a skin surface as shown in FIG. 5; [0106]: teaches "in the event that the user elects to select a probe by pressing button 228c, then the procedures necessary to capture and store the probe data 313 <read on required measurement> and display the probe data 315 are initiated"),
display the measurement point on the preview image by an augmented reality (AR) (Payonk, [0103]: teaches a monitor 270 displaying the subject S interactively positioning the probe over the area of interest, which is indicated by a focus box 226 <read on measurement point>),
recognize a position of the measuring device (Payonk, [0103]: teaches the area of interest being examined by positioning the probe over it and activating the probe to initiate testing; [0106]: teaches the probe test being conducted to correlate between the test result data and the position of the probe <read on recognize position of measuring device>),
display a measuring device indicator indicating the position of the measuring device (Payonk, FIG. 5 teaches a probe being displayed on a monitor, where an enlarged window indicates a region of interest that the probe is positioned at <read on measuring device indicator>; Note: it should be noted that the live video feed of the probe is being interpreted as the measuring device indicator),
change a color of the measurement point when the measuring device is positioned at the measurement point (Payonk, [0110]: teaches updating the pixel set <read on change color of measurement point> associated with focus box 226 when there has been a change to area of interest),
[[adjust a number of measurement points differently according to a size of a user's face on the preview image,]]
[[control the display to display a first number of measurement points if the size of the user's face on the preview image is smaller than a reference size, and]]
[[display a second number of measurement points if the size of the user's face on the preview image is larger than the reference size,]]
[[acquire information about an exposure by the user to ultraviolet (UV) and sleep time,]]
[[acquire information about an improvement effect of at least one cosmetic the user is using based on the information about the user's exposure to the UV and the sleep time, and]]
[[control the display to display a simulation image representing the improvement effect of the at least one cosmetic the user is using according to the user's exposure to the UV and the sleep time.]]
However, Payonk does not expressly disclose
adjust a number of measurement points differently according to a size of a user's face on the preview image,
control the display to display a first number of measurement points if the size of the user's face on the preview image is smaller than a reference size, and
display a second number of measurement points if the size of the user's face on the preview image is larger than the reference size,
acquire information about an exposure by the user to ultraviolet (UV) and sleep time,
acquire information about an improvement effect of at least one cosmetic the user is using based on the information about the user's exposure to the UV and the sleep time, and
control the display to display a simulation image representing the improvement effect of the at least one cosmetic the user is using according to the user's exposure to the UV and the sleep time.
Xu discloses
adjust a number of measurement points differently according to a size of a user's face on the preview image (Xu, [0015]: teaches a landmark engine 210 determining and detecting the locations of facial landmarks <read on adjust number of measurement points> based on the detected face <read on user's face> using machine learning; [0016]: teaches an alignment engine 205 selecting a size or an alignment of the face in the image <read on preview image> based on a size or an alignment of a neutral face),
control the display to display a first number of measurement points if the size of the user's face on the preview image is smaller than a reference size (Xu, [0016]: teaches the alignment engine 205 selecting a size or an alignment of the face in the image <read on control display> based on a size or an alignment of a neutral face, where "selecting the size and alignment to be the same may cause corresponding landmarks <read on first number of measurement points> between the face and neutral face to be in similar positions in the images"; [0016]: further teaches that "resizing the face in the image may include cropping the image, magnifying or shrinking <read on size of user's face on preview image being smaller than reference size> the image, or the like"), and
display a second number of measurement points if the size of the user's face on the preview image is larger than the reference size (Xu, [0016]: teaches the alignment engine 205 selecting a size or an alignment of the face in the image <read on control display> based on a size or an alignment of a neutral face, where "selecting the size and alignment to be the same may cause corresponding landmarks <read on second number of measurement points> between the face and neutral face to be in similar positions in the images"; [0016]: further teaches that "resizing the face in the image may include cropping the image, magnifying <read on size of user's face on preview image being larger than reference size> or shrinking the image, or the like"),
[[acquire information about an exposure by the user to ultraviolet (UV) and sleep time,]]
[[acquire information about an improvement effect of at least one cosmetic the user is using based on the information about the user's exposure to the UV and the sleep time, and]]
[[control the display to display a simulation image representing the improvement effect of the at least one cosmetic the user is using according to the user's exposure to the UV and the sleep time.]]
Xu is analogous art with respect to Payonk because they are from the same field of endeavor, namely face image capturing and analysis. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a landmark engine and an alignment engine as taught by Xu into the teaching of Payonk. The suggestion for doing so would allow the system to normalize facial images for more consistent landmark detection, thereby offering more reliable detected points. Therefore, it would have been obvious to combine Xu with Payonk.
However, the combination of Payonk and Xu does not expressly disclose
acquire information about an exposure by the user to ultraviolet (UV) and sleep time,
acquire information about an improvement effect of at least one cosmetic the user is using based on the information about the user's exposure to the UV and the sleep time, and
control the display to display a simulation image representing the improvement effect of the at least one cosmetic the user is using according to the user's exposure to the UV and the sleep time.
Bandic discloses
acquire information about an exposure by the user to ultraviolet (UV) and sleep time (Bandic, [0484]: teaches acquiring information 101 on users, "such as age, sex, location, personal lifestyle traits, smoking habits, sleep patterns <read on sleep time>, skin dryness/oiliness and moisture levels" through interactive tools, such as a user interface; [0471]: teaches "the user interface 102 may collect lifestyle data and also provide lifestyle (such as sleep, rest, exercise, and the like) and health (such as vitamins, food, products usage, and the like) recommendations based on the users particular skin state 158 and characteristics," where "the wellness and health module 142 may enable the user to obtain a personalized best fit health and wellness schedule and regimen 118"; [0421]: teaches analyzing and storing a skin state 158 to determine a regimen 118 based on "the current UV Index <read on user's exposure to UV>, time of day, location, kind of sun protection product the user prefers, and the like"),
acquire information about an improvement effect of at least one cosmetic the user is using based on the information about the user's exposure to the UV and the sleep time (Bandic, [0480]: teaches system 104 obtaining product information 190 and service oriented products, such as cosmetics, to improve <read on improvement effect> or maintain skin state 158 based on the user's individual preferences or conditions; Note: the user's condition is being interpreted as including UV exposure and user information, which includes sleep patterns), and
control the display to display a simulation image representing the improvement effect of the at least one cosmetic the user is using according to the user's exposure to the UV and the sleep time (Bandic, [0497]: teaches the user performing a simulation 132 <read on displayed simulation image> of future progress of skin state 158 based on skin care goal, product, and regimen; [0474]: teaches the user using simulation tools 132 to "change picture orientation, drag and drop, and the like"; Note: the regimen is based on the user's lifestyle data, which includes sleep and UV index).
Bandic is analogous art with respect to Payonk, in view of Xu because they are from the same field of endeavor, namely processing face image data of a user. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a user interface that includes a simulation tool that simulates the user's face based on a recommended regimen and input facial data of the user as taught by Bandic into the teaching of Payonk, in view of Xu. The suggestion for doing so would allow the simulation to use accurate data from probed facial regions, thereby achieving a more accurate simulation, enabling for better personalized recommendations and regimens, thereby improving the overall user experience. Therefore, it would have been obvious to combine Bandic with Payonk, in view of Xu.
Regarding Claim 11, it recites the limitations that are similar in scope to Claim 1, but in a skin characteristic data visualization system. As shown in the rejection, the combination of Payonk, Xu, and Bandic discloses the limitations of Claim 1. Additionally, Payonk discloses a skin characteristic data visualization system (Payonk, [0105]: teaches a computer system <read on skin characteristic data visualization system> that captures images of the user's face, which are then prepared for display and downloaded to the computer of imaging station 200) comprising:
a measuring device (Payonk, [0103]: teaches using a probe <read on measuring device> to initiate testing over an area of interest on a user's face); and
a skin characteristic data visualization device configured to communicate with the measuring device (Payonk, [0105]: teaches a computer of an imaging station 200 <read on skin characteristic data visualization device> that displays the taken images of the user's face by a separate computer system; [0106]: teaches the camera taking a picture at the same time the probe test is conducted to create a correlation <read on communication with measuring device> between the test result data and the position of the probe), wherein…
Thus, Claim 11 is met by Payonk according to the mapping presented in the rejection of Claim 1, given the skin characteristic data visualization device corresponds to a skin characteristic data visualization system.
Regarding Claims 3 and 13, the combination of Payonk, Xu, and Bandic discloses the skin characteristic data visualization device and the skin characteristic data visualization system of Claims 1 and 11 respectively. Additionally, Payonk further discloses wherein the processor is further configured to
display the position of the measuring device on the preview image by the AR (Payonk, FIG. 5 teaches a display of a user's face with a probe's position <read on measuring device position> being displayed).
Regarding Claims 4 and 14, the combination of Payonk, Xu, and Bandic discloses the skin characteristic data visualization device and the skin characteristic data visualization system of Claims 1 and 11 respectively. Additionally, Payonk further discloses wherein the processor is further configured to
activate a window for inputting a measurement value when the measuring device is positioned at the measurement point (Payonk, [0102]: teaches "button 228g being used to select a probe <read on measuring device> for examination for examination/analysis of the skin using a probe", where the user can select a probe from a menu along with a description of each probe's function; [0103]: teaches positioning the probe over an area of interest of the user's face defined by the focus box in a magnified image <read on measuring device positioned at measurement point> to aid the subject to place the probe in the correct position for testing and quantifying <read on measurement value input> skin conditions, where a live video thumbnail of the area of interest in the focus box is displayed <read on activate a window> to the user; [0103]: further teaches probe testing an area of irritation to "ascertain skin conductivity or redness weekly over the course of 4 weeks" through comparing quantified results) and
deactivate the window for inputting the measurement value when the measuring device is not positioned at the measurement point (Payonk, [0106]: teaches synchronizing the probe position with the live video; [0110]: teaches updating the pixel set associated with the focus box, where a "clear" signal <read on deactivate window> can be generated; Note: it should be noted that the focus box is updated when the probe's position is within the area of interest; should the position move out of the area of interest, the pixel set associated with the area of interest will no longer update).
Regarding Claims 5 and 15, the combination of Payonk, Xu, and Bandic discloses the skin characteristic data visualization device and the skin characteristic data visualization system of Claims 1 and 11 respectively. Additionally, Payonk further discloses wherein the processor is further configured to
display past data when there is a past measurement record at the measurement point where the measuring device is positioned (Payonk, [0099]: teaches the user reviewing images taken at different times <read on past data> regarding "the progress of their skin condition over time"; [0101]: teaches the user performing a side-by-side image comparison, where the position of the focus box <read on measurement point> in one image is synchronized with the other image; [0122]: teaches data obtained from skin in vivo on day 1 and 7 <read on past measurement record> as shown in FIGS. 17 and 18).
PNG
media_image2.png
451
453
media_image2.png
Greyscale
Regarding Claims 7 and 17, the combination of Payonk, Xu, and Bandic discloses the skin characteristic data visualization device and the skin characteristic data visualization system of Claims 1 and 11 respectively. The combination of Payonk and Xu does not expressly disclose the limitations of Claims 7 and 17; however, Bandic discloses wherein the processor is further configured to
display the simulation image further representing a change in skin characteristics according to a past measurement date and a past measurement value (Bandic, [0497]: teaches the user performing a simulation 132 <read on displayed simulation image> of future progress of skin state 158 based on skin care goal, product, and regimen; [0498]: teaches a summary screen of the skin care system, where "a report may include information on how the user's skin state 158 <read on skin characteristics> changed over time, if the user's skin is healthier <read on past measurement value> than when they started the regimen 118 <read on past measurement date>, if the product or regimen 118 met their initial goals, feedback on regimen 118/product effectiveness, and the like"; Note: it is being interpreted that the user's skin is being compared to a previously measured value).
Bandic is analogous art with respect to Payonk, in view of Xu because they are from the same field of endeavor, namely processing face image data of a user. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a simulation tool that simulates the user's face based on a recommended regimen, input current facial data, and past facial data of the user as taught by Bandic into the teaching of Payonk, in view of Xu. The suggestion for doing so would allow the simulation to use accurate data from probed areas of the face, thereby achieving a more accurate simulation, enabling for better personalized recommendations and regimens, thereby improving the overall user experience. Therefore, it would have been obvious to combine Bandic with Payonk, in view of Xu.
Regarding Claims 8 and 18, the combination of Payonk, Xu, and Bandic discloses the skin characteristic data visualization device and the skin characteristic data visualization system of Claims 7 and 17 respectively. The combination of Payonk and Xu does not expressly disclose the limitations of Claims 8 and 18; however, Bandic discloses wherein the processor is further configured to
display monitor information together with the simulation image (Bandic, [0466]: teaches the user interface 102 including simulation tools 132, where the user can "upload an image and model various skin parameters (such as moisture level in skin, collagen level, age, and the like.) and observe changes in the image <read on simulation image>" based on products and regimen 118; [0467]: teaches an included daily report 134 <read on monitor information> that is customized to the user based on their skin state 158), and
the monitor information includes external environment information and customer information (Bandic, [0467]: teaches the daily report 134 listing "skin care regimen 118 to be followed based on the environmental <read on external environment information> and lifestyle factors <read on customer information> relevant to the user").
Bandic is analogous art with respect to Payonk, in view of Xu because they are from the same field of endeavor, namely processing face image data of a user. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a simulation tool that simulates the user's face based on a recommended regimen, input current facial data, and past facial data of the user as taught by Bandic into the teaching of Payonk, in view of Xu. The suggestion for doing so would allow the simulation to use accurate data from probed areas of the face, thereby achieving a more accurate simulation, enabling for better personalized recommendations and regimens, thereby improving the overall user experience. Therefore, it would have been obvious to combine Bandic with Payonk, in view of Xu.
Regarding Claims 9 and 19, the combination of Payonk, Xu, and Bandic discloses the skin characteristic data visualization device and the skin characteristic data visualization system of Claims 7 and 17 respectively. The combination of Payonk and Xu does not expressly disclose the limitations of Claims 9 and 19; however, Bandic discloses wherein
when displaying the simulation image, the processor sets a transition effect between scenes (Bandic, [0423]: teaches using algorithm 150 for determining skin state 158 for prediction/simulation tools 132, where "a user may use a user interface 102 to access the simulation tools 132" and, in an example, "the user may like to perform a simulation of aging only, or a simulation of aging and treatment effects" as shown in FIG. 6; FIG. 6 teaches a simulation image of the user in the current day and 10 years later <read on scenes>, where skin parameters are adjustable, which is reflected <read on transition effect> in the right image; Note: the left and right images in FIG. 6 are being interpreted as scenes).
PNG
media_image3.png
407
561
media_image3.png
Greyscale
Bandic is analogous art with respect to Payonk, in view of Xu because they are from the same field of endeavor, namely processing face image data of a user. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a simulation tool that simulates the user's face based on a recommended regimen, input current facial data, and past facial data of the user as taught by Bandic into the teaching of Payonk, in view of Xu. The suggestion for doing so would allow the simulation to use accurate data from probed areas of the face, thereby achieving a more accurate simulation, enabling for better personalized recommendations and regimens, thereby improving the overall user experience. Therefore, it would have been obvious to combine Bandic with Payonk, in view of Xu.
Claims 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Payonk et al. (US 20060092315 A1, previously cited), hereinafter referenced as Payonk, in view of Xu et al. (US 20220114838 A1, previously cited), hereinafter referenced as Xu, and further in view of Bandic et al. (US 20110301441 A1), hereinafter referenced as Bandic as applied to Claims 1 and 11 above respectively, and further in view of Higashitsutsumi et al. (US 20150235359 A1, previously cited), hereinafter referenced as Higashitsutsumi.
Regarding Claims 10 and 20, the combination of Payonk, Xu, and Bandic discloses the skin characteristic data visualization device and the skin characteristic data visualization system of Claims 1 and 11 respectively. The combination of Payonk, Xu, and Bandic does not expressly disclose the limitations of Claims 10 and 20; however, Higashitsutsumi discloses
a communication unit wirelessly connected to the measuring device (Higashitsutsumi, [0164]: teaches a measuring instrument 21 <read on measuring device> containing a communication unit 21c that "wirelessly communicates with the smartphone 22"), wherein
PNG
media_image4.png
406
291
media_image4.png
Greyscale
the processor receives measurement information corresponding to the measurement point from the measuring device (Higashitsutsumi, [0196]: teaches the measuring instrument 21 <read on measuring device> measuring "a skin image at each of multiple different measurement positions <read on measurement points> while moving in close proximity to the user's skin"; [0197]: teaches measurement region 121 representing "a region measured <read on measurement information> by the measuring instrument 21").
Higashitsutsumi is analogous art with respect to the combination of Payonk, Xu, and Bandic because they are from the same field of endeavor, namely skin image analysis of faces. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a wireless measuring instrument that communicates with the main unit as taught by Higashitsutsumi into the combined teaching of Payonk, Xu, and Bandic. The suggestion for doing so would allow the user to measure their face without physical interference, thereby offering an overall better user experience. Therefore, it would have been obvious to combine Higashitsutsumi with the combination of Payonk, Xu, and Bandic.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Bandic et al. (US 20100185064 A1) discloses a skin care device that tracks the user's skin trait and offers recommendations;
Iglehart et al. (US 20170256084 A1) discloses guiding a user to manually apply cosmetics by overlaying graphical indicators on a real-time image of the user;
Kimura et al. (US 20180263551 A1) discloses a skin analysis system that measures the skin condition of input skin images;
Matsuda et al. (US 20200281340 A1) discloses a customizable cosmetic kit;
Nasu et al. (US 20240074563 A1) discloses applying makeup based on facial characteristics;
Yamanashi et al. (US 20160000209 A1) discloses a makeup assistance device that assists in makeup/skin treatment; and
Ye et al. (US 20140016823 A1) discloses a virtual makeup system that produces makeover effects to a 3D face model driven by facial tracking in real-time.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KARL TRUONG whose telephone number is (703)756-5915. The examiner can normally be reached 10:30 AM - 7:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/K.D.T./Examiner, Art Unit 2614
/YuJang Tswei/Primary Examiner, Art Unit 2614
Prosecution Insights