DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Preliminary Amendment
As per the preliminary amendments submitted by the applicant/s, claims 9 – 15 have been cancelled. Claims 1 – 8 and 16 – 25 have been examined accordingly.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 8, 16 – 17 and 20 – 23 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The limitations, under their broadest reasonable interpretation, cover mental process (concept performed in a human mind, including as observation, evaluation, judgment, opinion, organizing human activity and mathematical concepts and calculations). The claim(s) recite(s) a machine-readable storage medium configured to detect presence or absence of a mask. This judicial exception is not integrated into a practical application because the steps do not add meaningful limitations to be considered specifically applied to a particular technological problem to be solved .The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the steps of the claimed invention can be done mentally and no additional features in the claims would preclude them from being performed as such except for the generic computer elements at high level of generality (i.e., processor, memory).
According to the USPTO guidelines, a claim is directed to non-statutory subject matter if:
STEP 1: the claim does not fall within one of the four statutory categories of invention (process, machine, manufacture or composition of matter), or
STEP 2: the claim recites a judicial exception, e.g. an abstract idea, without reciting additional elements that amount to significantly more than the judicial exception, as determined using the following analysis:
STEP 2A (PRONG 1): Does the claim recite an abstract idea, law of nature, or natural phenomenon?
STEP 2A (PRONG 2): Does the claim recite additional elements that integrate the judicial exception into a practical application?
STEP 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
Using the two-step inquiry, it is clear that claims 1 and 10 are directed to an abstract idea as shown below:
STEP 1: Do the claims fall within one of the statutory categories? YES. Claim 1 is directed to an apparatus and claim 16 is directed to a non-transitory machine readable medium, i.e. a system.
STEP 2A (PRONG 1): Is the claim directed to a law of nature, a natural phenomenon or an abstract idea? YES, the claims are directed toward a mental process (i.e. abstract idea).
With regard to STEP 2A (PRONG 1), the guidelines provide three groupings of subject matter that are considered abstract ideas:
Mathematical concepts – mathematical relationships, mathematical formulas or equations, mathematical calculations;
Certain methods of organizing human activity – fundamental economic principles or practices (including hedging, insurance, mitigating risk); commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations); managing personal behavior or relationships or interactions between people (including social activities, teaching, and following rules or instructions); and
Mental processes – concepts that are practicably performed in the human mind (including an observation, evaluation, judgment, opinion).
The apparatus and system of claim 1 and 16 comprise a mental process that can be practicably performed in the human mind (or generic computers or components configured to perform the method) and, therefore, an abstract idea.
Regarding Claim 1: the claim recites:
determine a characteristic of an upper area of a face of a person in an image (mental process including observation and evaluation, and can be done mentally in the human mind; determining the characteristic of an upper area of a face of a person in an image can be done mentally as the characteristic is not specified…);
determine a characteristic of a lower area of the face of the person (mental process including observation and evaluation, and can be done mentally in the human mind; determining the characteristic of a lower area of the face of the person in an image can be done mentally as the characteristic is not specified...);
calculate a distance between the characteristic of the upper area and the characteristic of the lower area person (mathematical concepts, mathematical relationships, mathematical formulas or equations, mathematical calculations; calculating a distance between the characteristic of the upper area and the characteristic of the lower area person);
compare the distance to a threshold (mental process including observation and evaluation, and can be done mentally in the human mind; comparing …);
identify a presence of a face mask when the distance is greater than the threshold (mental process including observation and evaluation, and can be done mentally in the human mind; identifying presence of a mask…).
Regarding Claim 16: the claim recites:
map a characteristic of an upper area of a face of a person in an image to a color plot including a face skin tone cone (mental process including observation and evaluation, and can be done mentally in the human mind; map a characteristic of an upper area of a face of a person in an image to a color plot including a face skin tone cone);
map a characteristic of a lower area of the face to the color plot (mental process including observation and evaluation, and can be done mentally in the human mind; map a characteristic of a lower area of the face of the person in an image to the color plot);
identify a presence or an absence of a face mask based on the respective positions of the characteristic of the upper area and the characteristic of the lower area relative to the face skin tone cone (mental process including observation and evaluation, and can be done mentally in the human mind; identify presence or absence of a face mask…)
These limitations, as drafted, is a simple process that, under their broadest reasonable interpretation, covers performance of the limitations in the mind or by a human. The Examiner notes that under MPEP 2106.04(a)(2)(III), the courts consider a mental process (thinking) that “can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). As the Federal Circuit explained, "methods which can be performed mentally, or which are the equivalent of human mental work, are unpatentable abstract ideas the ‘basic tools of scientific and technological work’ that are open to all.’" 654 F.3d at 1371, 99 USPQ2d at 1694 (citing Gottschalk v. Benson, 409 U.S. 63, 175 USPQ 673 (1972)). See also Mayo Collaborative Servs. v. Prometheus Labs. Inc., 566 U.S. 66, 71, 101 USPQ2d 1961, 1965 ("‘[M]ental processes[] and abstract intellectual concepts are not patentable, as they are the basic tools of scientific and technological work’" (quoting Benson, 409 U.S. at 67, 175 USPQ at 675)); Parker v. Flook, 437 U.S. 584, 589, 198 USPQ 193, 197 (1978) (same). As such, a person could mentally analyze an image and map the color (characteristic) of the upper and lower areas of the face of a person in the image with a face skin tone cone and identify the presence or absence of a mask. The mere nominal recitation that the various steps are being executed by a device/in a device (e.g. processing unit) does not take the limitations out of the mental process grouping. Thus, the claims recite a mental process.
STEP 2A (PRONG 2): Does the claim recite additional elements that integrate the judicial exception into a practical application? NO, the claims do not recite additional elements that integrate the judicial exception into a practical application.
With regard to STEP 2A (prong 2), whether the claim recites additional elements that integrate the judicial exception into a practical application, the guidelines provide the following exemplary considerations that are indicative that an additional element (or combination of elements) may have integrated the judicial exception into a practical application:
an additional element reflects an improvement in the functioning of a computer, or an improvement to other technology or technical field;
an additional element that applies or uses a judicial exception to affect a particular treatment or prophylaxis for a disease or medical condition;
an additional element implements a judicial exception with, or uses a judicial exception in conjunction with, a particular machine or manufacture that is integral to the claim;
an additional element effects a transformation or reduction of a particular article to a different state or thing; and
an additional element applies or uses the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception.
While the guidelines further state that the exemplary considerations are not an exhaustive list and that there may be other examples of integrating the exception into a practical application, the guidelines also list examples in which a judicial exception has not been integrated into a practical application:
an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea;
an additional element adds insignificant extra-solution activity to the judicial exception; and
an additional element does no more than generally link the use of a judicial exception to a particular technological environment or field of use.
Claims 1 and 16 does/do not recite any of the exemplary considerations that are indicative of an abstract idea having been integrated into a practical application.
Claim 1 recites the further limitations of:
at least one memory; machine readable instructions; and processor circuitry to at least one of instantiate or execute the machine readable instructions to: (instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea).
Claim 16 recite(s) the further limitations of:
A non-transitory machine readable storage medium comprising instructions that, when executed, cause programmable circuitry to at least (instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea).
These limitations are recited at a high level of generality (i.e. as a general action or change being taken based on the results of the acquiring step) and amounts to mere post solution actions, which is a form of insignificant extra-solution activity. Further, the claims are claimed generically and are operating in their ordinary capacity such that they do not use the judicial exception in a manner that imposes a meaningful limit on the judicial exception. Accordingly, even in combination, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
STEP 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? NO, the claims do not recite additional elements that amount to significantly more than the judicial exception.
With regard to STEP 2B, whether the claims recite additional elements that provide significantly more than the recited judicial exception, the guidelines specify that the pre-guideline procedure is still in effect. Specifically, that examiners should continue to consider whether an additional element or combination of elements:
adds a specific limitation or combination of limitations that are not well-understood, routine, conventional activity in the field, which is indicative that an inventive concept may be present; or
simply appends well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception, which is indicative that an inventive concept may not be present.
Claims 1 and 16 does not recite any additional elements that are not well-understood, routine or conventional. The use of a computer to “obtaining, determining, and calculating, etc., as claimed in Claim(s) 1 and 16 is a routine, well-understood and conventional process that is performed by computers.
Thus, since Claim(s) 1 and 16 is/are: (a) directed toward an abstract idea, (b) do not recite additional elements that integrate the judicial exception into a practical application, and (c) do not recite additional elements that amount to significantly more than the judicial exception, it is clear that Claim(s) 1 and 16 is/are not eligible subject matter under 35 U.S.C 101.
Regarding Claim 8: the additional limitations do not integrate the mental process into practical application or add significantly more to the mental process. The limitation(s): identify a boundary line on the face of the person; and divide the face into the upper area and the lower area based on the boundary line (mental process including observation and evaluation, and can be done mentally in the human mind; identifying and dividing…). Since the claims are directed toward an abstract idea (mental process and data manipulation) using conventional tools in a generic way, without integration into a practical application or an inventive concept, they are ineligible under 35 U.S.C. 101.
Regarding Claims 17, 20 and 21: the additional limitations do not integrate the mental process into practical application or add significantly more to the mental process. The limitations of claim 17: determine a position of a boundary between the upper and lower areas is a mental process including observation and evaluation, and can be done mentally in the human mind. The limitations of claims 20 and 21: identifying the presence or absence of the face mask based in the characteristics (skin tone) can be a mental process including observation and evaluation, and can be done mentally in the human mind. Since the claims are directed toward an abstract idea (mental process and data manipulation) using conventional tools in a generic way, without integration into a practical application or an inventive concept, they are ineligible under 35 U.S.C. 101.
Regarding Claims 22 and 23: the additional limitations do not integrate the mental process into practical application or add significantly more to the mental process. The limitations of claims 22 and 23 recites calculating a distance between the characteristic of the upper area and the characteristic of the lower area (mathematical concepts, mathematical relationships, mathematical formulas or equations, mathematical calculations; calculating a distance between the characteristic of the upper area and the characteristic of the lower area person); compare the distance to a threshold (mental process including observation and evaluation, and can be done mentally in the human mind; comparing …); verify the presence or absence of the face mask (mental process including observation and evaluation, and can be done mentally in the human mind; verify …); verify the presence of the face mask when the distance is greater than the threshold (mental process including observation and evaluation, and can be done mentally in the human mind; verify …). Since the claims are directed toward an abstract idea (mental process and data manipulation) using conventional tools in a generic way, without integration into a practical application or an inventive concept, they are ineligible under 35 U.S.C. 101.
Regarding Claims 2 – 7, 18 – 19 and 24 – 25: these claims recite additional limitations that integrate the mental process into practical application or add significantly more to the mental process. Therefore, these claims are eligible under 35 U.S.C. 101 and are not rejected.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Yang et al. (See Machine Translation for CN 109101923 A; hereafter referred to as Yang) in view of Yu, Yi-jie (See Machine Translation for CN 112434562 A; hereafter referred to as Yu).
Regarding Claim 1, Yang teaches:
An apparatus comprising:
at least one memory (Yang, Fig.7, page 7, para 11, “The apparatus includes a processor 701, a memory 702”);
machine readable instructions (Yang, Fig. 7, page 7, para 12, “When the processor 701 executes the program instructions in the memory 702”); and
processor circuitry to at least one of instantiate or execute the machine readable instructions (Yang, Fig.7, page 7, para 12, “The processor 702 is coupled to the processor 701 via a bus 703. The memory 703 is used to store program instructions. When the processor 701 executes the program instructions in the memory 702”) to:
determine a characteristic of an upper area of a face of a person in an image (Yang, Page 2, S2, “S2, by performing threshold segmentation on the target face image, obtaining a binary image about the target face”; Yang, page 2, “S6, dividing Fb into upper half face image Fb1 and lower half face image Fb2”);
determine a characteristic of a lower area of the face of the person (Yang, page 2, S3, calculating a connected domain of the lower half of the binary image, and counting a width and a height of the circumscribed rectangular frame of the maximum connected domain”; Yang, page 2, “S6, dividing Fb into upper half face image Fb1 and lower half face image Fb2”);
While Yang teaches determining if a person is wearing a mask based on the number of pixels in the upper half face image and lower half face image, Yang does not explicitly teach:
calculate a distance between the characteristic of the upper area and the characteristic of the lower area;
compare the distance to a threshold; and
identify a presence of a face mask when the distance is greater than the threshold.
In the same field of endeavor, Yu teaches:
calculate a distance between the characteristic of the upper area and the characteristic of the lower area (Yu, page 3, last para, page 4, first para, “the case that the detection boundary is located between the third boundary and the second boundary, and the distance between the detection boundary and the third boundary is greater than a first preset distance, it is determined that the mask is only Covering the mouth area and the chin area, wherein the first preset distance is obtained according to the distance between the third boundary and the second boundary and a first preset ratio”);
compare the distance to a threshold (Yu, page 9, last para, “Upbound is the detection boundary of the mask, Bound1 is the first boundary, Bound2 is the second boundary, Bound3 is the third boundary, DistanceUp3 is the distance between the detection boundary and the third boundary, and Distance23 is the second”); and
identify a presence of a face mask when the distance is greater than the threshold (Yu, page 9, last para, page 10 – first para, “The distance between the boundary and the third boundary, DistanceUp1 is the distance between the detection boundary and the first boundary, and Distances12 is the distance between the first boundary and the second boundary. When each boundary meets the constraints of Formula 1, it is determined that the mask only covers the chin area, and when each boundary meets the constraints of Formula 2, it is determined that the mask only covers the mouth area and the chin area, and each boundary meets the constraints of Formula 3 Under the condition, it is determined that the mask covers the nose area, mouth area and chin area.”).
Yang and Yu are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Yang with the method of determining the presence or absence of a mask based in the distance as taught by Yu to make the invention that calculates a distance between the characteristic of the upper area and the characteristic of the lower are; compare the distance to a threshold; and identify a presence of a face mask when the distance is greater than the threshold; doing so can improve the detection accuracy of the mask wearing state while reducing detection cost (Yu, Summary); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 8, Yang in view of Yu teaches the apparatus of claim 1, wherein the processor circuitry is to:
identify a boundary line on the face of the person (Yu, Fig. 9, page 12, para 3, “The obtaining module 92 is configured to obtain the region of interest in the face image according to the key points and obtain the detection mark of the mask, where the region of interest includes at least one region of facial features and/or facial contours”; Yu, Fig. 5, page 8, para 3, “Therefore, the area of interest in this embodiment is set as the nose area, the mouth area and the chin area, and the boundary of the area of interest can be extracted by key points. FIG. 5 is a schematic diagram of the boundary line of the region of interest”); and
divide the face into the upper area and the lower area based on the boundary line (Yu, page 3, para 7 – 11, “Judging the mask wearing state of the pedestrian according to the positional relationship between the boundary line of the region of interest and the detection mark of the mask; …Determining the center line of the face image according to the key points of the nose area”).
The reasons for combining Yang and Yu are similar to that stated in the rejection of claim 1.
Claims 2 – 5 are rejected under 35 U.S.C. 103 as being unpatentable over Yang et al. (See Machine Translation for CN 109101923 A; hereafter referred to as Yang) in view of Yu, Yi-jie (See Machine Translation for CN 112434562 A; hereafter referred to as Yu) further in view of Xie et al. (See Machine Translation for CN 112489142 A; hereafter referred to as Xie).
Regarding Claim 2, Yang in view of Yu teaches the apparatus of claim 1, but do not explicitly teach:
wherein the characteristic is at least one of a hue, a saturation, or a value.
In the same field of endeavor, Xie teaches:
wherein the characteristic is at least one of a hue, a saturation, or a value (Xie, page 9, para 3, “The color category can be based on H (hue), S (saturation) Degree), V (brightness) value range is determined…. the HSV reference color is used to determine the color category of the target pixel. It should be noted that if the color recognition model is used to predict in the subsequent steps The color type or calculated color value then determines the color category.”).
Yang, Yu and Xie are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Yang in view of Yu with the method of using hue, a saturation, or a value as taught by Xie to make the invention that uses characteristics of hue, a saturation, or a value; doing so can improve the accuracy and integrity of color recognition of image regions (Xie, Summary); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 3, Yang in view of Yu further in view of Xie teaches the apparatus of claim 2, wherein the processor circuitry is to:
convert RGB color space of the image into HSV color space (Xie, page 12, S58, “calculating the RGB value of the second image block, and then converting the obtained RGB value into an HSV color space value, the color category of the second image block is determined according to the HSV reference color category, and the HSV reference color”; Xie, page 14, last para); and
determine the characteristic of the upper area and the characteristic of the lower area based on the HSV color space (Xie, page 12, S58, “the color category of the second image block is determined according to the HSV reference color category, and the HSV reference color…The color value of the sub-image block is used to obtain the color category of the sub-image block, and the sub-image block is classified according to the color category of the sub-image block”).
The reasons for combining Yang, Yu and Xie are similar to that stated in the rejection of claim 2.
Regarding Claim 4, Yang in view of Yu teaches the apparatus of claim 1, but does not explicitly teach:
wherein the characteristic of the upper area is at least one of an average of a hue across the upper area, an average of a saturation across the upper area, or an average of a value across the upper area, and wherein the characteristic of the lower area is at least one of an average of a hue across the lower area, an average of a saturation across the lower area, or an average of a value across the lower area.
In the same field of endeavor, Xie teaches:
wherein the characteristic of the upper area is at least one of an average of a hue across the upper area, an average of a saturation across the upper area, or an average of a value across the upper area, and wherein the characteristic of the lower area is at least one of an average of a hue across the lower area, an average of a saturation across the lower area, or an average of a value across the lower area (Xie, page 10, S45, “S45: Calculate the average value of the color values of the first image block to obtain an average color value, and create the missing number of image blocks by setting the color values of the pixel points to the image creation method of the average color value As a complement of sub-image blocks”; Xie, page 9, para 3, “The color category can be based on H (hue), S (saturation) Degree), V (brightness) value range is determined”).
Yang, Yu and Xie are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Yang in view of Yu with the method of using characteristic of the upper area and lower area of the face image as taught by Xie to make the invention that uses the characteristic of the upper area as at least one of an average of a hue across the upper area, an average of a saturation across the upper area, or an average of a value across the upper area, and the characteristic of the lower area as at least one of an average of a hue across the lower area, an average of a saturation across the lower area, or an average of a value across the lower area; doing so can improve the accuracy and integrity of color recognition of image regions (Xie, Summary); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 5, Yang in view of Yu teaches the apparatus of claim 1, but does not explicitly teach:
wherein the characteristic of the upper area is based on an average of a hue across the upper area, an average of a saturation across the upper area, and an average of a value across the upper area, and wherein the characteristic of the lower area is based on an average of a hue across the lower area, an average of a saturation across the lower area, and an average of a value across the lower area.
In the same field of endeavor, Xie teaches:
wherein the characteristic of the upper area is based on an average of a hue across the upper area, an average of a saturation across the upper area, and an average of a value across the upper area, and wherein the characteristic of the lower area is based on an average of a hue across the lower area, an average of a saturation across the lower area, and an average of a value across the lower area (Xie, page 10, S45, “S45: Calculate the average value of the color values of the first image block to obtain an average color value, and create the missing number of image blocks by setting the color values of the pixel points to the image creation method of the average color value As a complement of sub-image blocks”; Xie, page 9, para 3, “The color category can be based on H (hue), S (saturation) Degree), V (brightness) value range is determined”).
Yang, Yu and Xie are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Yang in view of Yu with the method of using characteristic of the upper area and lower area of the face image as taught by Xie to make the invention that uses characteristic of the upper area based on an average of a hue across the upper area, an average of a saturation across the upper area, and an average of a value across the upper area, and wherein the characteristic of the lower area is based on an average of a hue across the lower area, an average of a saturation across the lower area, and an average of a value across the lower area; doing so can improve the accuracy and integrity of color recognition of image regions (Xie, Summary); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Claims 6 – 7 are rejected under 35 U.S.C. 103 as being unpatentable over Yang et al. (See Machine Translation for CN 109101923 A; hereafter referred to as Yang) in view of Yu, Yi-jie (See Machine Translation for CN 112434562 A; hereafter referred to as Yu) further in view of Sato (US 20160219227 A1; hereafter referred to as Sato).
Regarding Claim 6, Yang in view of Yu teaches the apparatus of claim 1, but does not explicitly teach:
wherein the processor circuitry is to process the upper area with automatic exposure and skip the lower area when the face mask is identified.
In the same field of endeavor, Sato teaches:
wherein the processor circuitry is to process the upper area with automatic exposure and skip the lower area when the face mask is identified (Sato, the program storage section 7 has stored thereon a program for causing the control section 2 to perform AE (Auto Exposure) control, AF (Auto Focus) control, AWB (Auto White Balance) control, and the like, and various data”).
Yang, Yu and Sato are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Yang in view of Yu with the method of using automatic exposure as taught by Sato to make the invention that process the upper area with automatic exposure; doing so the face of a person is detected from an original image and processing strength is controlled based on the size of the face so as not to perform the processing uniformly regardless of the contents of the image. (Sato, [0005]); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 7, Yang in view of Yu teaches the apparatus of claim 1, but does not explicitly teach:
wherein the processor circuitry is to process the upper area with automatic white balance and skip the lower area when the face mask is identified.
In the same field of endeavor, Sato teaches:
wherein the processor circuitry is to process the upper area with automatic white balance and skip the lower area when the face mask is identified (Sato, the program storage section 7 has stored thereon a program for causing the control section 2 to perform AE (Auto Exposure) control, AF (Auto Focus) control, AWB (Auto White Balance) control, and the like, and various data”).
Yang, Yu and Sato are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Yang in view of Yu with the method of using white balance as taught by Sato to make the invention that process the upper area with automatic white balance; doing so the face of a person is detected from an original image and processing strength is controlled based on the size of the face so as not to perform the processing uniformly regardless of the contents of the image. (Sato, [0005]); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Claims 16, 18 – 21, and 24 – 25 are rejected under 35 U.S.C. 103 as being unpatentable over Yang et al. (See Machine Translation for CN 109101923 A; hereafter referred to as Yang) in view of Xie et al. (See Machine Translation for CN 112489142 A; hereafter referred to as Xie).
Regarding Claim 16, Yang teaches:
A non-transitory machine readable storage medium comprising instructions that, when executed, cause programmable circuitry (Yang, page 3, S52, “a detecting device for a person wearing a mask, comprising a memory and a processor, the memory storing at least one program, the at least one program being executed by the processor to implement the person”) to at least:
map a characteristic of an upper area of a face of a person in an image to a color plot including a face skin tone cone (Yang, page 4, step 102, “the skin color portion and the non-skin color portion of the target face image are threshold-divided according to the pixel value of the pixel point”; Jiang, page 5, “step 205, it is determined as a skin color pixel point”);
map a characteristic of a lower area of the face to the color plot (Yang, page 3, “S52: Statistic the value of the Cr and Cb channel pixels corresponding to all skin color pixels in the lower half of the binary image in step S2, and convert into a two-dimensional histogram”); and
While Yang teaches identifying the presence or absence of a face mask relating to skin tone (Yang, page 4, last para – page 5, “by performing back projection processing on the target face image, it is possible to more accurately characterize the difference between the skin color portion and the non-skin color portion blocked by the mask…In step 106, the Fb is divided into an upper half face image Fb1 and a lower half face image Fb2, and the number of pixels n1 and n2 in which the pixel point values in the Fb1 and Fb2 exceed the preset threshold are respectively counted, and it is determined whether n1 and n2 are satisfied. The preset condition, if satisfied, gives the detection result of the mask 107 not being worn, and vice versa”; Yang page 6, para 1, “The position pixel is assigned a value of 0, otherwise it is assigned a value of 255 to obtain a binarized image about the face. The skin color portion shows black (0), and the non-skin color portion displays white (255).), Yang does not explicitly teach:
positions of the characteristic of the upper area and the characteristic of the lower area relative to the face skin tone cone.
In the same field of endeavor, Xie teaches:
positions of the characteristic of the upper area and the characteristic of the lower area relative to the face skin tone cone (Xie, page 9, para 3, “in order to obtain a more accurate main color category of each of the sub-image blocks, based on the above-mentioned segmentation of the segmented region image to obtain the sub-image blocks, the sub-image blocks are further extracted Several target pixels at preset positions”; Xie, page 10, para 3, “the segmented area is firstly divided into blocks according to a preset size to obtain each sub-image block, and then the target pixel point in the preset position in the sub-image block is extracted and the color value is calculated to determine the color category of the target pixel, and finally the color category of the corresponding sub-image block is determined according to the target pixel and its color category”; Xia, page 7, para 6, Xie recites that the image containing human body is used as target region where in areas such as masks are identified).
Yang, and Xie are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Yang with the method of using position characteristics as taught by Xiao to make the invention that uses the characteristic of the upper area and the characteristic of the lower area relative to the face skin tone cone to identify the presence or absence of mask; doing so the target image region (face) is detected from an original image and the accuracy and integrity of color recognition of the image regions are improved (Xie, Summary); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 18, Yang in view of Xie teaches storage medium of claim 16, wherein the instructions cause the programmable circuitry to:
convert RGB color space of pixels in the image into HSV color space (Xie, page 12, S58, “calculating the RGB value of the second image block, and then converting the obtained RGB value into an HSV color space value, the color category of the second image block is determined according to the HSV reference color category, and the HSV reference color”; Xie, page 14, last para, “The calculation unit is configured to calculate the RGB value of the second image block, convert the RGB value of the second image block into a corresponding HSV color space value, and determine the color corresponding to the HSV color space value according to the HSV reference color Category to determine the color category of the second image block”); and
determine the characteristic of the upper area and the characteristic of the lower area based on the HSV color space (Xie, page 12, S58, “the color category of the second image block is determined according to the HSV reference color category, and the HSV reference color…The color value of the sub-image block is used to obtain the color category of the sub-image block, and the sub-image block is classified according to the color category of the sub-image block”).
The reasons for combining Yang and Xie are similar to that stated in the rejection of claim 16. In addition, this same reasoning is pertinent and applicable to the rejection of claim 19 below.
Regarding Claim 19, Yang in view of Xie teaches storage medium of claim 16, wherein the instructions cause the programmable circuitry to:
determine the characteristic of the upper area by aggregating an average hue, an average saturation, and an average value of a plurality of pixels in the upper area (Xie, page 10, S45, “S45: Calculate the average value of the color values of the first image block to obtain an average color value, and create the missing number of image blocks by setting the color values of the pixel points to the image creation method of the average color value As a complement of sub-image blocks, and finally splicing the created two complementary sub-image blocks with the corresponding first image block to obtain a complete mosaic and complementary image block, that is, the main color used to reflect the first image block The second image block of the category”; Xie, page 9, para 3, “The color category can be based on H (hue), S (saturation) Degree), V (brightness) value range is determined”); and
determine the characteristic of the lower area by aggregating an average hue, an average saturation, and an average value of a plurality of pixels in the lower area (Xie, page 10, S45, “S45: Calculate the average value of the color values of the first image block to obtain an average color value, and create the missing number of image blocks by setting the color values of the pixel points to the image creation method of the average color value As a complement of sub-image blocks, and finally splicing the created two complementary sub-image blocks with the corresponding first image block to obtain a complete mosaic and complementary image block, that is, the main color used to reflect the first image block The second image block of the category”; Xie, page 9, para 3, “The color category can be based on H (hue), S (saturation) Degree), V (brightness) value range is determined”).
Regarding Claim 20, Yang in view of Xie teaches storage medium of claim 16, wherein the instructions cause the programmable circuitry to identify the presence of the face mask when the characteristic of the upper area is within the face skin tone cone and the characteristic of the lower area is not within the skin tone cone (Yang, page 4, last para - page 5, para 1, “and combined with the binary image in step S2. A back projection face image Fb is obtained. In this step 104, by performing back projection processing on the target face image, it is possible to more accurately characterize the difference between the skin color portion and the non-skin color portion blocked by the mask. the Fb is divided into an upper half face image Fb1 and a lower half face image Fb2, and the number of pixels n1 and n2 in which the pixel point values in the Fb1 and Fb2 exceed the preset threshold are respectively counted, and it is determined whether n1 and n2 are satisfied. The preset condition, if satisfied, gives the detection result of the mask 107 not being worn, and vice versa, the detection result of wearing the mask 108.”; Yang, page 4, para 7, “the portion of the face that is blocked by the mask and the portion of the skin that is not blocked are greatly different in image characteristics. In a possible practical operation, the skin color portion and the non-skin color portion of the target face image are threshold-divided according to the pixel value of the pixel point”).
Regarding Claim 21, Yang in view of Xie teaches storage medium of claim 16, wherein the instructions cause the programmable circuitry to identify the absence of the face mask when the characteristic of the upper area is not within the face skin tone cone (Yang, page 5, para 1, “In step 106, the Fb is divided into an upper half face image Fb1 and a lower half face image Fb2, and the number of pixels n1 and n2 in which the pixel point values in the Fb1 and Fb2 exceed the preset threshold are respectively counted, and it is determined whether n1 and n2 are satisfied. The preset condition, if satisfied, gives the detection result of the mask 107 not being worn, and vice versa, the detection result of wearing the mask 108.”).
Regarding Claim 24, Yang teaches:
A method of processing images based on detection of a face mask (Yang, page 3, S52, “a method for detecting a person wearing a mask as described in the first aspect”), the method comprising:
identifying, by executing instructions with a processor, an upper area of a face and a lower area of a face (Yang, Page 2, S2, “S2, by performing threshold segmentation on the target face image, obtaining a binary image about the target face”; Yang, page 2, “S6, dividing Fb into upper half face image Fb1 and lower half face image Fb2”);
mapping, by executing instructions with the processor, the Pu and the PI to a plot including a face skin tone cone (Yang, page 3, “S52: Statistic the value of the Cr and Cb channel pixels corresponding to all skin color pixels in the lower half of the binary image in step S2, and convert into a two-dimensional histogram”; Yang, page 4, step 102, “the skin color portion and the non-skin color portion of the target face image are threshold-divided according to the pixel value of the pixel point”; Yang, page 5, “step 205, it is determined as a skin color pixel point”); and
identifying, by executing instructions with the processor, a presence of an absence of the face mask based on a respective position of the Pu on the map and the PI on the map relative to the face skin tone cone (Yang, page 4, last para – page 5, first para, “by performing back projection processing on the target face image, it is possible to more accurately characterize the difference between the skin color portion and the non-skin color portion blocked by the mask…In step 106, the Fb is divided into an upper half face image Fb1 and a lower half face image Fb2, and the number of pixels n1 and n2 in which the pixel point values in the Fb1 and Fb2 exceed the preset threshold are respectively counted, and it is determined whether n1 and n2 are satisfied. The preset condition, if satisfied, gives the detection result of the mask 107 not being worn, and vice versa”).
However, Yang does not explicitly teach:
calculating, by executing instructions with the processor:
an average hue (Hu), an average saturation (Su), and an average value (Vu) for the upper area;
an average hue (HI), an average saturation (SI), and an average value (VI) for the lower area;
an HSV combination for the upper area: Pu (Hu, Su, Vu); and
an HSV combination for the lower area: PI (HI, SI, VI)
In the same field of endeavor, Xie teaches:
an average hue (Hu), an average saturation (Su), and an average value (Vu) for the upper area (Xie, page 11, para 1, “The average value of the color values of the first black image block to obtain the average color value, and then by setting the color values of the pixels to the average color value of the image creation method, two image blocks are created as complementary sub-images Block”; Xie, page 9, para 3, “the HSV reference color is used to determine the color category of the target pixel”);
an average hue (HI), an average saturation (SI), and an average value (VI) for the lower area (Xie, page 12, S58, “the color category of the second image block is determined according to the HSV reference color category, and the HSV reference color The category is consistent with the HSV reference color category in the foregoing embodiment. It should be noted that, in this embodiment, the color category of the second image block is consistent with the color category of the sub-image block corresponding to the second image block”);
an HSV combination for the upper area: Pu (Hu, Su, Vu); and an HSV combination for the lower area: PI (HI, SI, VI) (Xie, page 6, last para, page 7, first para, “When the segmented area image corresponds to multiple second image blocks and the color category of the second image block is not a single color category, it means that the segmented area image is not a solid color, and the segmented area image has multiple colors Category, the color category sequence of the segmented region image can be determined according to the color categories of the multiple second image blocks, and the combination of the color categories of each second image block is the color of the segmented region image Category sequence”; Xie, page 9, para 3, “The color category can be based on H (hue), S (saturation) Degree), V (brightness) value range is determined”; Xie, page 10, last para);
Yang, and Xie are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Yang with the method of calculating an average hue (Hu), an average saturation (Su), and an average value (Vu) as taught by Xiao to make the invention that calculated an average hue (Hu), an average saturation (Su), and an average value (Vu) for the upper area and lower area and calculates HSV combination for the upper area and lower area; doing so the target image region (face) is detected from an original image and the accuracy and integrity of color recognition of the image regions are improved. (Xie, Summary); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 25, Yang in view of Xie teaches method of claim 24, further including (Examiner Note: since the claim requires (1) or (2), the Examiner is mapping (1) to meet the claim limitation mapping):
(1) identifying, by executing instructions with the processor, the presence of the face mask when the Pu is within the face skin tone cone and the PI is not within the skin tone cone (Yang, page 4, last para - page 5, para 1, “and combined with the binary image in step S2. A back projection face image Fb is obtained. In this step 104, by performing back projection processing on the target face image, it is possible to more accurately characterize the difference between the skin color portion and the non-skin color portion blocked by the mask. the Fb is divided into an upper half face image Fb1 and a lower half face image Fb2, and the number of pixels n1 and n2 in which the pixel point values in the Fb1 and Fb2 exceed the preset threshold are respectively counted, and it is determined whether n1 and n2 are satisfied. The preset condition, if satisfied, gives the detection result of the mask 107 not being worn, and vice versa, the detection result of wearing the mask 108.”; Yang, page 4, para 7, “the portion of the face that is blocked by the mask and the portion of the skin that is not blocked are greatly different in image characteristics. In a possible practical operation, the skin color portion and the non-skin color portion of the target face image are threshold-divided according to the pixel value of the pixel point”); or
(2) identifying, by executing instructions with the processor, the presence of the face mask when (a) the Pu is within the face skin tone cone (b) the PI is within the face skin tone cone and (c) a calculated distance between one or more of the Hu, Su, Vu and one or more respective HI, SI, VI is greater than a threshold distance.
Claims 17, and 22 – 23 are rejected under 35 U.S.C. 103 as being unpatentable over Yang et al. (See Machine Translation for CN 109101923 A; hereafter referred to as Yang) in view of Xie et al. (See Machine Translation for CN 112489142 A; hereafter referred to as Xie) further in view of Yu, Yi-jie (See Machine Translation for CN 112434562 A; hereafter referred to as Yu).
Regarding Claim 17, Yang in view of Xie teaches the storage medium of claim 16, but does not explicitly teach:
wherein the instructions cause the programmable circuitry to determine a position of a boundary between the upper area and the lower area.
In the same field of endeavor, Yu teaches:
wherein the instructions cause the programmable circuitry to determine a position of a boundary between the upper area and the lower area (Yu, page 3, para 7 – 11, “Judging the mask wearing state of the pedestrian according to the positional relationship between the boundary line of the region of interest and the detection mark of the mask;…Determining the center line of the face image according to the key points of the nose area”).
Yang, Xie and Yu are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Yang in view of Xie with the method of determining a position of a boundary between the upper area and the lower area as taught by Xie to make the invention that determines a position of a boundary between the upper area and the lower area; doing so the detection accuracy of the mask wearing state can be improved while the detection cost is reduced (Yu, Summary); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 22, Yang in view of Xie teaches the storage medium of claim 16, but does not explicitly teach:
calculate a distance between the characteristic of the upper area and the characteristic of the lower area; and
verify the presence or the absence of the face mask based on the distance.
In the same field of endeavor, Yu teaches:
calculate a distance between the characteristic of the upper area and the characteristic of the lower area (Yu, page 3, last para, page 4, first para, “the case that the detection boundary is located between the third boundary and the second boundary, and the distance between the detection boundary and the third boundary is greater than a first preset distance, it is determined that the mask is only Covering the mouth area and the chin area, wherein the first preset distance is obtained according to the distance between the third boundary and the second boundary and a first preset ratio”); and
verify the presence or the absence of the face mask based on the distance (Yu, page 9, last para – page 10, first para, “The distance between the boundary and the third boundary, DistanceUp1 is the distance between the detection boundary and the first boundary, and Distances12 is the distance between the first boundary and the second boundary. When each boundary meets the constraints of Formula 1, it is determined that the mask only covers the chin area, and when each boundary meets the constraints of Formula 2, it is determined that the mask only covers the mouth area and the chin area, and each boundary meets the constraints of Formula 3 Under the condition, it is determined that the mask covers the nose area, mouth area and chin area.”).
Yang, Xie and Yu are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Yang in view of Xie with the method of calculating a distance between the characteristic of the upper area and the characteristic of the lower area as taught by Xie to make the invention that calculates a distance between the characteristic of the upper area and the characteristic of the lower area; and verifies the presence or the absence of the face mask based on the distance; doing so the detection accuracy of the mask wearing state can be improved while the detection cost is reduced (Yu, Summary); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 23, Yang in view of Xie further in view of Yu teaches the storage medium of claim 22, wherein the instructions cause the programmable circuitry to:
compare the distance to a threshold distance (Yu, page 9, last para, “Upbound is the detection boundary of the mask, Bound1 is the first boundary, Bound2 is the second boundary, Bound3 is the third boundary, DistanceUp3 is the distance between the detection boundary and the third boundary, and Distance23 is the second”); and
verify the presence of the face mask when the distance is greater than the threshold (Yu, page 9, last para, page 10 – first para, “The distance between the boundary and the third boundary, DistanceUp1 is the distance between the detection boundary and the first boundary, and Distances12 is the distance between the first boundary and the second boundary. When each boundary meets the constraints of Formula 1, it is determined that the mask only covers the chin area, and when each boundary meets the constraints of Formula 2, it is determined that the mask only covers the mouth area and the chin area, and each boundary meets the constraints of Formula 3 Under the condition, it is determined that the mask covers the nose area, mouth area and chin area.”).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20190014884 A1 Systems And Methods For Virtual Facial Makeup Removal And Simulation, Fast Facial Detection And Landmark Tracking, Reduction In Input Video Lag And Shaking, And A Method For Recommending Makeup : method for virtually removing facial makeup, the method comprising providing a facial image of a user with makeups being applied thereto, locating facial landmarks from the facial image of the user in one or more regions, decomposing some regions into first channels which are fed to histogram matching to obtain a first image without makeup in that region and transferring other regions into color channels which are fed into histogram matching under different lighting conditions to obtain a second image without makeup in that region, and combining the images to form a resultant image with makeups removed in the facial regions. The disclosure also provides systems and methods for virtually generating output effects on an input image having a face, for creating dynamic texturing to a lip region of a facial image, for a virtual eye makeup add-on that may include multiple layers, a makeup recommendation system based on a trained neural network model
US 20210142042 A1 Skin Tone Assisted Digital Image Color Matching: skin tone assisted digital image color matching, a device implements a color editing system, which includes a facial detection module to detect faces in an input image and in a reference image, and includes a skin tone model to determine a skin tone value reflective of a skin tone of each of the faces. A color matching module can be implemented to group the faces into one or more face groups based on the skin tone value of each of the faces, match a face group pair as an input image face group paired with a reference image face group, and generate a modified image from the input image based on color features of the reference image
US 20140341442 A1 IMAGE MASKS FOR FACE-RELATED SELECTION AND PROCESSING IN IMAGES: a method includes identifying one or more face regions of an image, the face regions including pixels that depict at least a portion of one or more faces of persons. The face regions are identified based on identifying facial landmarks of the faces. The method determines an associated face mask for each of the faces based on the face regions, where each face mask indicates which pixels in the image depict the corresponding face. Face pixels can be selected for processing by applying the face masks, and image pixels outside the faces can be selected by inversely applying the face masks.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VAISALI RAO KOPPOLU whose telephone number is (571)270-0273. The examiner can normally be reached Monday - Friday 8:30 - 5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format.
For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
VAISALI RAO. KOPPOLU
Examiner
Art Unit 2664
/VAISALI RAO KOPPOLU/Examiner of Art Unit 2664
Prosecution Insights