DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 12/23/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
The information disclosure statement (IDS) submitted on 06/22/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Preliminary Amendment
Preliminary amendments filed 12/23/2024 have been acknowledged
Claims 1-4 7-11 have been amended.
Claims 12-20 are new.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 13-17 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claims 13-17 recite “wherein the processing step comprises”. In their respective dependencies (claims 2-6) there is no “processing step” only a “determining step” and a modification by the calculator in claim 3. It is unclear if the processing steps, determination steps and claim 3’s modifying step are in tandem, series, simultaneous etc.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 7-14, 18, 19 are rejected under 35 U.S.C. 103 as being unpatentable over Tuan et al (Tuan hereinafter US 9064279 B1) in view of Merkle et al (Merkle hereinafter US 11373059 B2)
As per claim 1
Tuan, teaches A method for determining parameters relative to the coloration of a body zone of an individual (Figure 6, Paragraph (10) “he system receives the actual skin tone of the customer based on a skin tone color set to determine a skin tone identifier for the customer” Paragraph (35) “The system includes a cosmetic product selection tool, which uses skin tone to determine suitable products for customers.” Paragraph (12) “using a scanning device to determine skin shades” Paragraph (16) “determine a skin tone identifier corresponding to a skin tone of a customer and the determined skin tone identifier is based on a skin tone color space, where the skin tone color space includes skin tone identifiers and each skin tone identifier corresponds to a skin tone found in a population sample” Paragraph (72) “ As described above, this can be done by using a color scanner to scan various points on the person's face or other area of the body where they want to apply a cosmetic product. Then the scanner displays a skin tone identifier. “) the method comprising the steps of: a. selecting a sample for an individual amongst a set of samples (Figure 4, Figure 5 Figure 16, Figure 18, Paragraph (148) “ FIG. 18 shows a screen 1801 to select a skin tone color shade of the system. The system displays at least a portion of a skin tone color set of the system and allows a user to select one of the skin tone identifiers. In screen 1801, an array of thirty skin tone color identifiers (which can be referred to as a skin tone identifier chart”) each sample of the set being a sample representative of a body zone and having a reference colour distinct from the other samples (Figure 16, Figure 18, Figure 15, Paragraph (72) “As described above, this can be done by using a color scanner to scan various points on the person's face or other area of the body where they want to apply a cosmetic product.” Paragraph 128 “The first two areas 1503 and 1505 generally correspond to areas of the customer's face, while a third area 1507 corresponds to areas below the customer's lips.” Paragraph (129) “In a specific implementation, area 1503 is a region of the face above the customer's eyebrows. Area 1505 is a region of the face between the eyebrows and the chin. Area 1507 is a region of the face or body below the chin. For example, area 1503 may include the forehead. Area 1505 may include the cheekbone or cheek. Area 1507 may include the chin, neck, or upper torso.” Paragraph (148) “an array of thirty skin tone color identifiers (which can be referred to as a skin tone identifier chart) are displayed with their associated skin tone color shades. This is a portion of the available 110 skin tone identifiers. The user selects a skin tone color by selected pointing to the appropriate skin tone color in the array.”) each sample having an identifier visible on the sample (Figure 18) the identifier of each sample being associated with a colorimetric data representative of the reference colour of the sample (Figure 18 , Paragraph (123) “…the processor receives information of scans from the measurement optics (e.g., two, three, four, or more scans) and processes (or maps, averages, blends, or mixes) the information to determine a single color value. This color value maps to a single skin tone identifier in the skin tone color space. This information can be displayed7) “ on the display 1405 of the scanning device. The skin tone identifier displayed value in display 1405 can be entered (e.g., manually entered) into tablet device by a user.) so-called sample reference data, the sample reference data of each identifier being memorised in a database (Figure 3, Paragraph (35) “The system can be based off a skin tone color set that indexes skin tone colors such as Pantone's Skintone Library, although other skin tone color sets can be used by the system” Paragraph (69) “The skin tone color set can be referred to as a unified, uniform, or reference skin tone color set against which products are mapped. A database of products across different manufacturers 411 is mapped to the reference skin tone color set. Paragraph (71) “Then, for each evaluated product, the skin tone identifier or identifiers (in the reference color set) is added into a products database with skin tone 413.” Paragraph (97) “the customer enters in their skin tone identifier (e.g., four digit alphanumeric code) 701. The system queries the products database with skin tones for the products associated with the determined skin tone identifier 703. As discussed above, the product database can be stored on the computing device or accessible over a network…”)
Tuan does not teach acquiring, by a sensor an image, so-called initial image imaging both the selected sample (10) and a corresponding body zone of the individual. Processing the initial image (IM), by a calculator, so as to obtain: I. the sample reference data (DrefE) of the sample imaged on the initial image (IM) according to the identifier of said sample and the database accessible by the calculator colorimetric data representative of the actual colour of the sample (10) on the initial image (IM), so-called sample actual data (Dr E), the sample reference data (DrefE) and the sample actual data (DrE) forming parameters relative to the coloration of the body zone of the individual.
Merkle teaches acquiring, by a sensor an image, so-called initial image imaging both the selected sample (10) and a corresponding body zone of the individual. (Paragraph (12) “reference images are obtained of a user with and without makeup applied. The images include, or are correlated with, a reference target such as a color checker.” Paragraph (39) “in one example, the color checker contains 14 unique skin tone colors, which will be helpful for calibrating the skin complexions” Paragraph (42) “ the color checker is also imaged together with each of the subjects in order to create a standard reference for each of the images.” processing the initial image (IM), by a calculator (Figure 1, Figure 2 Figure3, Paragraph (42) “ In some embodiments, the color checker is also imaged together with each of the subjects in order to create a standard reference for each of the images.” Paragraph (46) “a two-stage process utilizes gray balancing and polynomial regression to rectify the colors of interest, based on images that capture the color checker in the same frame as each of the subjects.” Paragraph (47) “to be able to calibrate the colors, the values of the color patches in the original image and their corresponding ground truth (i.e., measured) values are obtained. This may be done for some or all of the available color patches on the color checker. However, in one embodiment, only a subset of color patches is utilized based on the visual relationship of the patches to common skin colors “ Paragraph (30) “data may transfer back and forth between the color space converter 104 and the color set calculator 106, or their respective modular implementations, in order to perform a combination of conversion and calculations to accomplish the image processing for specific color sets.” Furthermore, the processing/computing system performs the image and color processing as well as processes the resulting color information and is effectively a calculator. This includes the modeling engine and the prediction engine, the computing system of figure 1 as a whole and its modules. ) a colorimetric data representative of the actual colour of the sample (10) on the initial image (IM), so-called sample actual data (Dr E) so-called sample actual data (Dr E) the sample reference data (DrefE) and the sample actual data (DrE) forming parameters relative to the coloration of the body zone of the individual. (Figure 1, Paragraph (30) “Although the color set calculator 106 is shown downstream from the color space converter 104, there is no limitation on physical layout or the order of operations or functions that can be performed by these component modules (or any other components) within the prediction model system 100. Additionally, in some embodiments, data may transfer back and forth between the color space converter 104 and the color set calculator 106, or their respective modular implementations, “ This shows modularity of the computing systems specific processing paths, logic and components. Paragraph (37) “The prediction is based upon the color coordinates of the skin and of the foundation shade that are retrieved from images, which may be taken by the user. To ensure color accuracy, these images are calibrated with the help of a color checker…” Paragraph (37) “This allows the RGB values of the color checker patches to be mapped to reference values ” Paragraph (47) “ the values of the color patches in the original image and their corresponding ground truth (i.e., measured) values are obtained.” Paragraph (48) “the RGB values of the identified patches in the image are extracted through an automated process This may be done by locating the spatial centroid of each patch and then computing the average RGB values over the center region of the patch.” The RGB values from the image are the colorimetric data. The so called sample actual data is the RGB value of the color checker patch extracted from the image and the sample reference data is the corresponding ground truth reference value of the CIE XYZ value that is stored. The ground truth is the reference data because Merkle states “To be able to calibrate the colors, the values of the color patches in the original image and their corresponding ground truth (i.e., measured) values are obtained. …the reference CIE XYZ values of the target color patches can be measured with a spectrophotometer under a standardized illuminant (e.g., D50 illuminant), and these values may be stored as a separate measured color set.” This gives the ground truth and claimed “sample reference data” the same role. The ground truth exists separately from how the checker happens to look like in the subject image. Merkle uses the pairing of the image RGB value and ground truth CIE XYZ values to create the calibration of color used to get an accurate estimation of the subjects skin and output the various color sets..
In regards to the limitation so as to obtain: i. the sample reference data (DrefE) of the sample imaged on the initial image (IM) according to the identifier of said sample and the database (11) accessible by the calculator, Tuan teaches correlating skin tone identifiers with stored reference color information. Tuan also teaches computing device/”calculator” having access to a database based on a skin tone identifier to obtain associated information. Merkle teaches imagining a reference color sample with the subject and obtaining its corresponding ground truth color value. The combined teachings provide obtaining stored reference color data corresponding to the identified sample imaged with the subject. The combined teachings provide a calculator accessing the database to obtain stored reference color data corresponding to the identified sample imaged with the subject. Paragraph (30) of Merkle insists modularity between the calculator and other components that a person of ordinary skill in the art would have found it obvious enough for the calculator to query the database. The calculator is a movable/modular processer. Furthermore, as explained above, the term “calculator” under broadest reasonable interpretation can mean anything that calculates mathematical and or logical paths. For all intents and purposes any processing computing system is a calculator.
Accordingly, a person of ordinary skill in the art, at the time this invention was effectively filed would have found it obvious to modify Tuan’s methodology with Merkle’s concept of color reference calculation and calibration. A person of ordinary skill in the art would make this modification because Merkle teaches using a color checker imaged with the subject to provide a reference for accurately determining color from the captured image. Incorporation of this technique with Tuan would predictably allow Tuan’s known reference skin tone information to be correlated to how the reference color actually appears under the particular image conditions. This improves the reliability of the skin color determination. The combination yields the advantage of improved accuracy and consistency of Tuan’s skin tone determination through compensating for variations introduced by the camera and illumination conditions resulting in more reliable skin color information for cosmetic product selection.
As per claim 2
Tuan and Merkle teach all the claim limitations previously rejected in claim 1’s 103 rejection. See claim 1’s 103 rejection.
Merkle teaches wherein the method comprises a step of determining, by the calculator (14), the influence of the environment on the colorimetric rendering of the body zone of the individual by comparison of the sample reference data (DrefE) with the sample actual data (DrE). (Paragraph (11) “In other embodiments, lighting conditions may vary, and prediction models account for actual or approximated differences in such conditions.” Paragraph (12) “The color checker may be included in the original images or obtained separately under the same illumination conditions. Relying on the XYZ values (according to the International Commission on Illumination (CIE)) of the reference values on patches of the color checker, a mapping is created from a camera-dependent RGB space to a standard CIE XYZ space.” Paragraph (37) “The prediction is based upon the color coordinates of the skin and of the foundation shade that are retrieved from images, which may be taken by the user. To ensure color accuracy, these images are calibrated with the help of a color checker.” Paragraph (42) “In some embodiments, the color checker is also imaged together with each of the subjects in order to create a standard reference for each of the images.” Paragraph (47) “To be able to calibrate the colors, the values of the color patches in the original image and their corresponding ground truth (i.e., measured) values are obtained…Next, the RGB values of the identified patches in the image are extracted through an automated process. This may be done by locating the spatial centroid of each patch and then computing the average RGB values over the center region of the patch. In other embodiments, other models for RGB value extraction may be used. …“Paragraph (56) “he color difference between the calibrated CIE L*a*b* coordinates and the ground truth measured values of each of the patches is computed…which indicates that color correction is quite effective for these test images.” The RGB value of the color checker patches is obtained and their corresponding ground truth (reference) color values are used together to elucidate the relationship between these values for image color calibration. The difference between the captured RGB appearance and known reference value reflects the effects introduced by the image capturing. The image capture device influences the environment through illumination.
As per claim 3
Tuan and Merkle teach all claim limitations previously rejected in claim 1’s 103 rejection. See claim 1’s 103 rejection.
Merkle/Tuan teaches modifying, by the calculator (14), the initial image (IM) according to the sample actual data (DrE) so that the colour of the body zone of the individual on the initial image (IM) is replaced by the actual colour of the sample on the initial image (IM), (Merkle: Figure 4, Paragraph (66) “In addition to makeup applications, in other embodiments aspects of the processing described herein can be used to synthesize skin tones which are impacted by cosmetic and/or medical procedures… synthesize the coloring effects of a chemical peel procedure to evaluate redness resulting from the procedure… synthesize coloring changes before and after treatments for dark circles under the user's eyes…a user may be able to synthesize coloring changes resulting from treatment or removal of skin features such as moles or skin cancer. In another example, a user may be able to synthesize skin coloring and conditions during or after treatment for acne…In each of these examples, embodiments described herein can create a more realistic representation of skin coloring and conditions than the simplistic addition of color layers with a level of transparency on top of a user's image. The use of analyzed, calibrated, and balanced color processing help to visualize color changes with increased accuracy, regardless of whether the changes are related to skin tone brightness, correction, or averaging” Claim 10: “wherein the modeling engine is further configured to process the output color set to display a synthesized visual representation of the output color set applied to skin pixels of a subject user. Claim 11: “display the synthesized visual representation of the output color set to replace skin pixels of the skin color set within a visual framework inclusive of the subject user.” Merkle’s computing system (previously mapped to the calculator) performs the image processing. )
In regards to the limitation of “according to the sample actual data” Merkle’s claim 1 mapping corresponds the sample actual data to the color data obtained from the sample color patch. Therefore Merkle teaches computer modification of the subject image whereby the subject’s skin pixels are replaced with the selected color to produce the virtual synthetic image.
Tuan in view of Merkle teaches virtually applying the selected cosmetic reference color to the individuals body zone by modifying the subjects image according to the corresponding color data and displaying the resulting image render.
As per claim 7
Tuan and Merkle teach all claim limitations previously rejected in claim 1’s 103 rejection. See claim 1’s 103 rejection.
Merkle teaches comprises highlighting the sample (10) on the initial image (IM) (Paragraph (48) “he RGB values of the identified patches in the image are extracted through an automated process. This may be done by locating the spatial centroid of each patch and then computing the average RGB values over the center region of the patch” Paragraph (52) “e patches centered around the centroid of Set 1 are identified based on pixels having a Euclidean distance of less than 80 to the centroid of Set 1” Merkle shows here that they identify and locates color checker patches within the image. Under broadest reasonable interpretation this corresponds to highlighting in the image. ) and determining the sample actual data (DrE) over the portion of the initial image (IM) featuring the sample (10). (Paragraph (47) “the values of the color patches in the original image and their corresponding ground truth (i.e., measured) values are obtained. This may be done for some or all of the available color patches on the color checker. Paragraph (48) “ the RGB values of the identified patches in the image are extracted through an automated process This may be done by locating the spatial centroid of each patch and then computing the average RGB values over the center region of the patch” This shows that Merkle determines sample actual data from the pixels within the region containing the identified color checker rather than the whole image.)
As per claim 8
Tuan and Merkle teach all claim limitations previously rejected in claim 7’s 103 rejection. See claim 7’s 103 rejection.
Merkle teaches wherein highlighting is carried out by segmentation ((Paragraph (52) “To form Set 1, the mean RGB values of the pixels within the skin segmentation mask are computed. This is referred to as the centroid of Set 1. Then, the patches centered around the centroid of Set 1 are identified based on pixels having a Euclidean distance of less than 80 to the centroid of Set 1) or is carried out by extraction of the zone around the identifier (10A) of the sample (10) on the initial image (IM) after identification of the identifier (10A) on the initial image” (Paragraph (48) “ the RGB values of the identified patches in the image are extracted through an automated process This may be done by locating the spatial centroid of each patch and then computing the average RGB values over the center region of the patch”)
In regards carried out by extraction of the zone around the identifier (10A) of the sample (10) on the initial image (IM) after identification of the identifier (10A) on the initial image. Tuan supplies the identifier and Merkle supplies the localization and extraction of the image region associated with the identified color patch and computing the average RGB values over the center region of the patch. The combination allows the modified methodology to identify the sample by its identifier and extract the image zone surrounding the identified sample for determining its color data.
As per claim 9
Tuan and Merkle teach all claim limitations previously rejected in claim 1’s 103 rejection. See claim 1’s 103 rejection.
Tuan teaches wherein the samples (10) are selected from the list including: photographs, post-its, fabrics, figurines, miniature products and real or fake samples of parts of the human body, ( Figure 9, Paragraph (64) “Referring to the back end in FIG. 4, the system determines and creates a skin tone color set which the products will be mapped to. This color set is determined based on taking a sample population of people 401 that represent the range of skin tones. For example, the colors can be developed to present the best color match when evaluated under D65 (Daylight 6500K) lighting. This population sample can include people of different ethnicities and geographical areas. This population sample is analyzed to gather skin tone colors 403.”)
As per claim 10
Tuan and Merkle teach all claim limitations previously rejected in claim 1’s 103 rejection. See claim 1’s 103 rejection.
Tuan teaches wherein the identifier is selected from among: a number, a series of alphanumeric characters, a barcode, a radio identification marker, a near-field communication marker and a marker allowing a visual identification. (Figure 9, Figure 16 Figure 17, Figure 18, Paragraph (127) “the scanner can display a bar code or QR code on a display of the scanner representative of a skin tone identifier. “)
As per claim 11
Claim 11 is the device claim that parallels claim 1’s method claim. Therefore claim 11 will be rejected under the same premise.
As per claim 12
Tuan and Merkle teach all claim limitations previously rejected in claim 2s 103 rejection. See claim 2’s 103 rejection.
Merkle/Tuan teaches modifying, by the calculator (14), the initial image (IM) according to the sample actual data (DrE) so that the colour of the body zone of the individual on the initial image (IM) is replaced by the actual colour of the sample on the initial image (IM), (Merkle: Figure 4, Paragraph (66) “In addition to makeup applications, in other embodiments aspects of the processing described herein can be used to synthesize skin tones which are impacted by cosmetic and/or medical procedures… synthesize the coloring effects of a chemical peel procedure to evaluate redness resulting from the procedure… synthesize coloring changes before and after treatments for dark circles under the user's eyes…a user may be able to synthesize coloring changes resulting from treatment or removal of skin features such as moles or skin cancer. In another example, a user may be able to synthesize skin coloring and conditions during or after treatment for acne…In each of these examples, embodiments described herein can create a more realistic representation of skin coloring and conditions than the simplistic addition of color layers with a level of transparency on top of a user's image. The use of analyzed, calibrated, and balanced color processing help to visualize color changes with increased accuracy, regardless of whether the changes are related to skin tone brightness, correction, or averaging” Claim 10: “wherein the modeling engine is further configured to process the output color set to display a synthesized visual representation of the output color set applied to skin pixels of a subject user. Claim 11: “display the synthesized visual representation of the output color set to replace skin pixels of the skin color set within a visual framework inclusive of the subject user.” Merkle’s computing system (previously mapped to the calculator) performs the image processing. )
In regard to the limitation of “according to the sample actual data” Merkle’s claim 1 mapping corresponds the sample actual data to the color data obtained from the sample color patch. Therefore, Merkle teaches computer modification of the subject image whereby the subject’s skin pixels are replaced with the selected color to produce the virtual synthetic image.
Tuan in view of Merkle teaches virtually applying the selected cosmetic reference color to the individuals body zone by modifying the subjects image according to the corresponding color data and displaying the resulting image render.
As per claim 13
Tuan and Merkle teach all claim limitations previously rejected in claim 2’s 103 rejection. See claim 2’s 103 rejection.
Merkle teaches comprises highlighting the sample (10) on the initial image (IM) (Paragraph (48) “he RGB values of the identified patches in the image are extracted through an automated process. This may be done by locating the spatial centroid of each patch and then computing the average RGB values over the center region of the patch” Paragraph (52) “e patches centered around the centroid of Set 1 are identified based on pixels having a Euclidean distance of less than 80 to the centroid of Set 1” Merkle shows here that they identify and locates color checker patches within the image. Under broadest reasonable interpretation this corresponds to highlighting in the image. ) and determining the sample actual data (DrE) over the portion of the initial image (IM) featuring the sample (10). (Paragraph (47) “the values of the color patches in the original image and their corresponding ground truth (i.e., measured) values are obtained. This may be done for some or all of the available color patches on the color checker. Paragraph (48) “ the RGB values of the identified patches in the image are extracted through an automated process This may be done by locating the spatial centroid of each patch and then computing the average RGB values over the center region of the patch” This shows that Merkle determines sample actual data from the pixels within the region containing the identified color checker rather than the whole image.)
As per claim 14
Tuan and Merkle teach all claim limitations previously rejected in claim 3’s 103 rejection. See claim 3’s 103 rejection.
Merkle teaches wherein the process step comprises highlighting the sample (10) on the initial image (IM) (Paragraph (48) “he RGB values of the identified patches in the image are extracted through an automated process. This may be done by locating the spatial centroid of each patch and then computing the average RGB values over the center region of the patch” Paragraph (52) “e patches centered around the centroid of Set 1 are identified based on pixels having a Euclidean distance of less than 80 to the centroid of Set 1” Merkle shows here that they identify and locates color checker patches within the image. Under broadest reasonable interpretation this corresponds to highlighting in the image. ) and determining the sample actual data (DrE) over the portion of the initial image (IM) featuring the sample (10). (Paragraph (47) “the values of the color patches in the original image and their corresponding ground truth (i.e., measured) values are obtained. This may be done for some or all of the available color patches on the color checker. Paragraph (48) “ the RGB values of the identified patches in the image are extracted through an automated process This may be done by locating the spatial centroid of each patch and then computing the average RGB values over the center region of the patch” This shows that Merkle determines sample actual data from the pixels within the region containing the identified color checker rather than the whole image.)
As per claim 18
Tuan and Merkle teach all claim limitations previously rejected in claim 2’s 103 rejection. See claim 2’s 103 rejection.
Tuan teaches wherein the samples (10) are selected from the list including: photographs, post-its, fabrics, figurines, miniature products and real or fake samples of parts of the human body, ( Figure 9, Paragraph (64) “Referring to the back end in FIG. 4, the system determines and creates a skin tone color set which the products will be mapped to. This color set is determined based on taking a sample population of people 401 that represent the range of skin tones. For example, the colors can be developed to present the best color match when evaluated under D65 (Daylight 6500K) lighting. This population sample can include people of different ethnicities and geographical areas. This population sample is analyzed to gather skin tone colors 403.”)
As per claim 19
Tuan and Merkle teach all claim limitations previously rejected in claim 3’s 103 rejection. See claim 3’s 103 rejection.
Tuan teaches wherein the samples (10) are selected from the list including: photographs, post-its, fabrics, figurines, miniature products and real or fake samples of parts of the human body, ( Figure 9, Paragraph (64) “Referring to the back end in FIG. 4, the system determines and creates a skin tone color set which the products will be mapped to. This color set is determined based on taking a sample population of people 401 that represent the range of skin tones. For example, the colors can be developed to present the best color match when evaluated under D65 (Daylight 6500K) lighting. This population sample can include people of different ethnicities and geographical areas. This population sample is analyzed to gather skin tone colors 403.”)
Claims 4-6, 15, 16, 17, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Tuan et al (Tuan hereinafter US 9064279 B1) in view of Merkle et al (Merkle hereinafter US 11373059 B2) in further view of Lee et al (Lee hereinafter US 20200175729 A1)
As per claim 4
Tuan and Merkle teach all claim limitations previously rejected in claim 3’s 103 rejection. See claim 3’s 103 rejection.
Merkle teaches wherein the method comprises a step of determining, by the calculator (14), a coloration for the body zone of the individual according to the initial image (Paragraph (22) “the camera 102 may capture images of skin color subjects 112, or people without makeup applied to their skin. The data from skin color subjects 112 can be used, in some form, by the prediction model generator 110 as input to generate the prediction model” Paragraph (27) “For skin color subjects 112, the image processing is focused on identifying and characterizing primary skin colors. Similarly, for target color subjects 116, the image processing focuses on identifying and characterizing applied makeup colors on skin.” , the sample reference data (DrefE) and the sample actual data (Dr E) (Paragraph (37) “This allows the RGB values of the color checker patches to be mapped to reference values “ Paragraph (47) “To be able to calibrate the colors, the values of the color patches in the original image and their corresponding ground truth (i.e., measured) values are obtained” Paragraph (49) “Using the image RGB and reference CIE XYZ value pairs” )
Tuan nor Merkle teach this is done “upon reception of a command validating the rendering image (IMR).”
Lee teaches upon reception of a command validating the rendering image (IMR). (Paragraph [0044] “The user may select an original image and a desired color through the touch screen to change a hair color in the original image, confirm the selected original image, and confirm a transformed, virtual dyeing image.” Figure 3 shows that Lee teaches that the virtual image is created changing the hair color of the users original image to the user’s selected desired color.
In a combined teaching Tuan and Merkle in view of Lee teach determining a cosmetic coloration using the previously obtained reference and actual color data and providing a virtual rendering that the user can confirm through the user interface
Accordingly a person of ordinary skill in the art would have found it obvious to further modify the Tuan/Merkle methodology with Lee’s concept of allowing the user control over rendering image validation. This allows the individual to confirm that the virtual rendered image is acceptable before proceeding with further downstream processes. Lee provides an interactive virtual hair dyeing system in which a user can choose desired color and gets a virtual render. They then can control the validation of the transformed image through an interface and or terminal. The modification obviously brings the advantage of providing the user control over the cosmetic coloration pipeline by allowing the individual to review and validate their choices through the rendered image before further downstream processes. Ultimately the user, a consumer, is the ultimate decider of what skin tone and coloration works for them, empowering the user to choose and alter their desired appearance for multiple scenarios and applications.
As per claim 5
Tuan, Merkle and Lee teach all claim limitations previously rejected in claim 4’s 103 rejection. See claim 4’s 103 rejection.
Merkle teaches wherein the step of determining a coloration comprises determining a colorimetric data representative of the actual colour of the body zone of the individual on the initial image called individual actual data (DrI) (Figure 1, Paragraph (44) “the skin pixels in the images are detected and isolated from the non-skin pixels. An algorithm for skin detection may be implemented.” Paragraph (45) “In one example, a fast and efficient RGB-H-CbCr model is adopted. The skin detection model may utilize one or more criteria for identifying a skin pixel. In one embodiment, under uniform illumination conditions, skin pixels may be filtered using the following criteria. First, the RGB values fall within a specific range of absolute and relative values. Second, the red and blue chrominance values satisfy relative value conditions.” Paragraph (46) “After the skin pixels are detected, image calibration converts the device-dependent RGB values into the CIE XYZ values. In one embodiment, a two-stage process utilizes gray balancing and polynomial regression to rectify the colors of interest, “ The RGB and color coordinates derived from the skin pixels corresponds to claimed “individual actual data” ) the coloration of the body zone of the individual being determined according to the individual actual data (DrI), the sample reference data (DrefE) and the sample actual data (Dr E). (Figure 1, Figure 2 Paragraph (59) “, a prediction model is developed that can predict the CIE L*a*b* coordinates of the skin-with-foundation color (i.e., the output color set) based on inputs using data from the CIE L*a*b* coordinates of a skin-with-no-foundation color (i.e., the skin color set) and a foundation color (i.e., the makeup color set).” The skin color data corresponds to the individual actual data. The color checker image values and their measured CIE reference values which are stored are the sample actual and sample reference data previously mapped in claim 1’s 103 rejection are used to calibrate the images from which the coloration values are determined. )
As per claim 6
Tuan, Merkle and Lee teach all claim limitations previously rejected in claim 5’s 103 rejection. See claim 5’s 103 rejection.
Merkle teaches wherein the step of determining a coloration further comprises:a. determining a colorimetric discrepancy between the sample reference data (DrefE) and the sample actual data (Dr E), (Paragraph (55) “n order to evaluate the accuracy of the calibration results, the difference can be measured between the calibrated XYZ values obtained from the transformation matrices and the corresponding reference XYZ values of the color patches obtained using a spectrophotometer” Paragraph (56) “The color difference between the calibrated CIE L*a*b* coordinates and the ground truth measured values of each of the patches is computed. Experimental results measuring 35 patches found that 33 patches have an acceptable difference (e.g., less than 3),” determining a colorimetric data representative of a reference colour for the body area of the individual, so-called individual reference data (Drefi), according to the individual actual data (DrI) and the discrepancy (Paragraph (46) “After the skin pixels are detected, image calibration converts the device-dependent RGB values into the CIE XYZ values” Paragraph (54) “he transformation matrices can be applied to the entire image. For example, the image pixels are classified into three sets based on the three precomputed centroids, and then the linearized RGB values are converted to CIE XYZ using the corresponding matrices. In other embodiments, another number of sets may be used.” Paragraph (60) “ three different transformation matrices are computed separately and then applied to the corresponding pixels in the image. The calibration accuracy is measured by the color difference ΔE between the calibrated value in CIE L*a*b* space and the measured reference value.” Merkle shows that the calibration relationship/error established from the actual and reference color checker values and applying the resulting transformation to the image. A applying that correction to the skin pixel data produces calibrated skin color data corresponding to the claimed individual reference data.) the coloration of the body zone of the individual being determined according to the individual reference data (Drefi) and the sample reference data (DrefE). (Paragraph (37) “The prediction is based upon the color coordinates of the skin and of the foundation shade that are retrieved from images, which may be taken by the user. To ensure color accuracy, these images are calibrated with the help of a color checker” Paragraph (38) “A prediction model can be developed that takes the color coordinates of the skin and the foundation as inputs and then outputs those of a modeled color to anticipate the combined foundation and skin application “ Paragraph (59) “Using the experimental results obtained, a prediction model is developed that can predict the CIE L*a*b* coordinates of the skin-with-foundation color (i.e., the output color set) based on inputs using data from the CIE L*a*b* coordinates of a skin-with-no-foundation color (i.e., the skin color set) and a foundation color (i.e., the makeup color set)” This shows that Merkle determines the resulting cosmetic coloration using the calibrated reference skin color corresponding to the individual reference data together with the reference cosmetic color information and the sample reference data.)
As per claim 15
Tuan, Merkle and Lee teach all claim limitations previously rejected in claim 4’s 103 rejection. See claim 4’s 103 rejection.
Merkle teaches wherein the process step comprises highlighting the sample (10) on the initial image (IM) (Paragraph (48) “he RGB values of the identified patches in the image are extracted through an automated process. This may be done by locating the spatial centroid of each patch and then computing the average RGB values over the center region of the patch” Paragraph (52) “e patches centered around the centroid of Set 1 are identified based on pixels having a Euclidean distance of less than 80 to the centroid of Set 1” Merkle shows here that they identify and locates color checker patches within the image. Under broadest reasonable interpretation this corresponds to highlighting in the image. ) and determining the sample actual data (DrE) over the portion of the initial image (IM) featuring the sample (10). (Paragraph (47) “the values of the color patches in the original image and their corresponding ground truth (i.e., measured) values are obtained. This may be done for some or all of the available color patches on the color checker. Paragraph (48) “ the RGB values of the identified patches in the image are extracted through an automated process This may be done by locating the spatial centroid of each patch and then computing the average RGB values over the center region of the patch” This shows that Merkle determines sample actual data from the pixels within the region containing the identified color checker rather than the whole image.)
As per claim 16
Tuan, Merkle and Lee teach all claim limitations previously rejected in claim 5’s 103 rejection. See claim 5’s 103 rejection.
Merkle teaches wherein the process step comprises highlighting the sample (10) on the initial image (IM) (Paragraph (48) “he RGB values of the identified patches in the image are extracted through an automated process. This may be done by locating the spatial centroid of each patch and then computing the average RGB values over the center region of the patch” Paragraph (52) “e patches centered around the centroid of Set 1 are identified based on pixels having a Euclidean distance of less than 80 to the centroid of Set 1” Merkle shows here that they identify and locates color checker patches within the image. Under broadest reasonable interpretation this corresponds to highlighting in the image. ) and determining the sample actual data (DrE) over the portion of the initial image (IM) featuring the sample (10). (Paragraph (47) “the values of the color patches in the original image and their corresponding ground truth (i.e., measured) values are obtained. This may be done for some or all of the available color patches on the color checker. Paragraph (48) “ the RGB values of the identified patches in the image are extracted through an automated process This may be done by locating the spatial centroid of each patch and then computing the average RGB values over the center region of the patch” This shows that Merkle determines sample actual data from the pixels within the region containing the identified color checker rather than the whole image.)
As per claim 17
Tuan, Merkle and Lee teach all claim limitations previously rejected in claim 6’s 103 rejection. See claim 6’s 103 rejection.
Merkle teaches wherein the process step comprises highlighting the sample (10) on the initial image (IM) (Paragraph (48) “he RGB values of the identified patches in the image are extracted through an automated process. This may be done by locating the spatial centroid of each patch and then computing the average RGB values over the center region of the patch” Paragraph (52) “e patches centered around the centroid of Set 1 are identified based on pixels having a Euclidean distance of less than 80 to the centroid of Set 1” Merkle shows here that they identify and locates color checker patches within the image. Under broadest reasonable interpretation this corresponds to highlighting in the image. ) and determining the sample actual data (DrE) over the portion of the initial image (IM) featuring the sample (10). (Paragraph (47) “the values of the color patches in the original image and their corresponding ground truth (i.e., measured) values are obtained. This may be done for some or all of the available color patches on the color checker. Paragraph (48) “ the RGB values of the identified patches in the image are extracted through an automated process This may be done by locating the spatial centroid of each patch and then computing the average RGB values over the center region of the patch” This shows that Merkle determines sample actual data from the pixels within the region containing the identified color checker rather than the whole image.)
As per claim 20
Tuan, Merkle and Lee teach all claim limitations previously rejected in claim 4’s 103 rejection. See claim 4’s 103 rejection.
Tuan teaches wherein the samples (10) are selected from the list including: photographs, post-its, fabrics, figurines, miniature products and real or fake samples of parts of the human body, ( Figure 9, Paragraph (64) “Referring to the back end in FIG. 4, the system determines and creates a skin tone color set which the products will be mapped to. This color set is determined based on taking a sample population of people 401 that represent the range of skin tones. For example, the colors can be developed to present the best color match when evaluated under D65 (Daylight 6500K) lighting. This population sample can include people of different ethnicities and geographical areas. This population sample is analyzed to gather skin tone colors 403.”)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHANE WRENSFORD CODRINGTON whose telephone number is (571)272-8130. The examiner can normally be reached 8:00am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHANE WRENSFORD CODRINGTON/Examiner, Art Unit 2667 /MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights