DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
PNG
media_image1.png
813
675
media_image1.png
Greyscale
PNG
media_image2.png
699
708
media_image2.png
Greyscale
Groupings of Abstract Ideas
A.) Mathematical Concepts
mathematical relationships
mathematical formulas or equations
mathematical calculations
B.) Mental Processes
concepts performed in the human mind (including an observation, evaluation, judgment, opinion)
C.) Certain Methods Of Organizing Human Activity
fundamental economic principles or practices (including hedging, insurance, mitigating risk)
commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations)
managing personal behavior or relationships or interactions between people (including social activities, teaching, and following rules or instructions)
NOTE: The recitation of generic computer components in a claim does not necessarily preclude that claim from reciting an abstract idea.
Claims 1-10 and 12-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more. Claims 1-10 and 12-20 are directed to the abstract idea of mental processes (Step 2A prong 1). Independent claims 1, 9, and 15 do not include additional elements that are sufficient to amount to significantly more than the judicial exception or that integrate the judicial exception into a practical application (Step 2A prong 2). It is noted that dependent claims 2-8, 10, 12-14, and 16-20 do not claim any additional elements beyond the abstract idea indicating the claimed method can be read on by a person performing the same steps, and therefore contain nothing more, significant or otherwise.
In claim 1 the limitations recite:
A method comprising:
-obtaining an image including text and a region overlapping the text, wherein the text comprises a first color (Interpreted as reciting a mental process as the limitation states the concept of acquiring an image that includes text and an area which overlaps steps that can be performed via a person with a sheet of paper drawing an image, writing text, and drawing a border over the text in a given color thus this step can be thought of without exhaustive effort by the human mind.);
-selecting a second color that contrasts with the first color and (Interpreted as reciting a mental process as the limitation states the concept color selection which contrasts the original color used which can be simply selecting a different colored writing tool than the one previously used, thus this step can be thought of without exhaustive effort by the human mind.);
-generating a modified image including the text and a modified region using a machine learning model that takes the image and the second color as input (Interpreted as reciting a mental process as the limitation states outputting of a modified image using a model that takes the image and second color as input. Herein the machine learning model is essentially used as a black box as it takes in inputs and provides an output without any specific details on an algorithm or steps of how the machine learning model obtains the output regarding the modified image, and therefore this can simply be interpreted as the user making a change to the color of the originally drawn image with a secondary colored utensil, thus this step can be thought of without exhaustive effort by the human mind.).
The claim meets the requirements of Step 1 as the claim is directed toward a method (i.e. process) for image editing. As shown above the claim recites nothing more than the process of generating a modified image (Step 2A Prong 1) and thus the claim has been interpreted as being directed toward the abstract idea of a mental process. Subsequently with regards to the (Step 2A Prong 2) analysis, it has been determined that the claim does not recite any additional elements that integrate the judicial exception into a practical application. The additional limitations fail to meet the standards of what is considered a practical application with respect to the list below:
Improvements to the functioning of a computer, or to any other technology or technical field - see MPEP 2106.05(a)
Applying the judicial exception with, or by use of, a particular machine - see MPEP 2106.05(b)
Effecting a transformation or reduction of a particular article to a different state or thing - see MPEP 2106.05(c)
Applying or using the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception - see MPEP 2106.05(e).
In regards to dependent claim 2, the limitations further recite segmentation steps to identify objects overlapping the text for providing the secondary color to create a first modified region in relation to claim 1, however this fails to meet the standards of improving the functioning of a computer, applying the exception to a particular machine, effecting some sort of transformation of an article, or integrating the judicial exception into a practical application as these limitations do not amount to “significantly more” than the abstract idea as previously described as these functions can all be performed without exhaustive effort of the human mind.
In regards to dependent claim 3, the limitations further recite adding noise to the overlapping region regarding the text to create the modified image in relation to claim 1, however this fails to meet the standards of improving the functioning of a computer, applying the exception to a particular machine, effecting some sort of transformation of an article, or integrating the judicial exception into a practical application as these limitations do not amount to “significantly more” than the abstract idea as previously described as these functions can all be performed without exhaustive effort of the human mind.
In regards to dependent claim 4, the limitations further recite generating a mask to indicate the area that overlaps the text via the addition of noise in relation to claim 3, however this fails to meet the standards of improving the functioning of a computer, applying the exception to a particular machine, effecting some sort of transformation of an article, or integrating the judicial exception into a practical application as these limitations do not amount to “significantly more” than the abstract idea as previously described as these functions can all be performed without exhaustive effort of the human mind.
In regards to dependent claim 5, the limitations further recite that the noise is partially colored noise relating to the second color in relation to claim 3, however this fails to meet the standards of improving the functioning of a computer, applying the exception to a particular machine, effecting some sort of transformation of an article, or integrating the judicial exception into a practical application as these limitations do not amount to “significantly more” than the abstract idea as previously described as these functions can all be performed without exhaustive effort of the human mind.
In regards to dependent claim 6, the limitations further recite generating a combined image based on the original image and modified image in relation to claim 1, however this fails to meet the standards of improving the functioning of a computer, applying the exception to a particular machine, effecting some sort of transformation of an article, or integrating the judicial exception into a practical application as these limitations do not amount to “significantly more” than the abstract idea as previously described as these functions can all be performed without exhaustive effort of the human mind.
In regards to dependent claim 7, the limitations further recite generating a composite image based on adding text to the modified image in relation to claim 1, however this fails to meet the standards of improving the functioning of a computer, applying the exception to a particular machine, effecting some sort of transformation of an article, or integrating the judicial exception into a practical application as these limitations do not amount to “significantly more” than the abstract idea as previously described as these functions can all be performed without exhaustive effort of the human mind.
In regards to dependent claim 8, the limitations further recite generating a color palette based on the region overlapping the text to select the secondary color in relation to claim 1, however this fails to meet the standards of improving the functioning of a computer, applying the exception to a particular machine, effecting some sort of transformation of an article, or integrating the judicial exception into a practical application as these limitations do not amount to “significantly more” than the abstract idea as previously described as these functions can all be performed without exhaustive effort of the human mind.
In regards to independent claim 9, this claim recites features similar in scope to that of claim 1, and therefore is also rejected as being directed toward the abstract idea of reciting a mental process with respect to image editing.
In regards to dependent claim 10, the limitations further recite segmentation steps to identify objects overlapping the text for providing the secondary color to create a first modified region in relation to claim 9, however this fails to meet the standards of improving the functioning of a computer, applying the exception to a particular machine, effecting some sort of transformation of an article, or integrating the judicial exception into a practical application as these limitations do not amount to “significantly more” than the abstract idea as previously described as these functions can all be performed without exhaustive effort of the human mind.
In regards to dependent claim 12, the limitations further recite adding noise to the overlapping region regarding the text to create the modified image in relation to claim 9, however this fails to meet the standards of improving the functioning of a computer, applying the exception to a particular machine, effecting some sort of transformation of an article, or integrating the judicial exception into a practical application as these limitations do not amount to “significantly more” than the abstract idea as previously described as these functions can all be performed without exhaustive effort of the human mind.
In regards to dependent claim 13, the limitations further recite generating a combined image based on the original image and modified image in relation to claim 9, however this fails to meet the standards of improving the functioning of a computer, applying the exception to a particular machine, effecting some sort of transformation of an article, or integrating the judicial exception into a practical application as these limitations do not amount to “significantly more” than the abstract idea as previously described as these functions can all be performed without exhaustive effort of the human mind.
In regards to dependent claim 14, the limitations further recite generating a composite image based on adding text to the modified image in relation to claim 9, however this fails to meet the standards of improving the functioning of a computer, applying the exception to a particular machine, effecting some sort of transformation of an article, or integrating the judicial exception into a practical application as these limitations do not amount to “significantly more” than the abstract idea as previously described as these functions can all be performed without exhaustive effort of the human mind.
In regards to independent claim 15, this claim recites features similar in scope to that of claims 1 and 9 aside from the additional generic computer components regarding the processor and memory, and therefore is also rejected as being directed toward the abstract idea of reciting a mental process with respect to image editing.
In regards to dependent claim 16, the limitations further recite segmentation steps to identify objects in relation to claim 15, however this fails to meet the standards of improving the functioning of a computer, applying the exception to a particular machine, effecting some sort of transformation of an article, or integrating the judicial exception into a practical application as these limitations do not amount to “significantly more” than the abstract idea as previously described as these functions can all be performed without exhaustive effort of the human mind.
In regards to dependent claim 17, the limitations further recite adding noise to the image regarding the text region in relation to claim 15, however this fails to meet the standards of improving the functioning of a computer, applying the exception to a particular machine, effecting some sort of transformation of an article, or integrating the judicial exception into a practical application as these limitations do not amount to “significantly more” than the abstract idea as previously described as these functions can all be performed without exhaustive effort of the human mind.
In regards to dependent claim 18, the limitations further recite extracting superpixels from the region regarding the text in relation to claim 15, however this fails to meet the standards of improving the functioning of a computer, applying the exception to a particular machine, effecting some sort of transformation of an article, or integrating the judicial exception into a practical application as these limitations do not amount to “significantly more” than the abstract idea as previously described as these functions can all be performed without exhaustive effort of the human mind.
In regards to dependent claim 19, the limitations further recite generating a combined image based on the original image and a background image in relation to claim 15, however this fails to meet the standards of improving the functioning of a computer, applying the exception to a particular machine, effecting some sort of transformation of an article, or integrating the judicial exception into a practical application as these limitations do not amount to “significantly more” than the abstract idea as previously described as these functions can all be performed without exhaustive effort of the human mind.
In regards to dependent claim 20, the limitations further recite defining the machine learning model as a generative diffusion model in relation to claim 15, however this fails to meet the standards of improving the functioning of a computer, applying the exception to a particular machine, effecting some sort of transformation of an article, or integrating the judicial exception into a practical application as these limitations do not amount to “significantly more” than the abstract idea as previously described as these functions can all be performed without exhaustive effort of the human mind.
The Examiner suggests inclusion of language that shows an algorithm or steps for how the machine learning model goes about obtaining the generated modified image from the inputs regarding the image having the text and the selected second color, which would then satisfy a requirement of tying the abstract idea to a practical application (see the rejection of claim 1 for the standards of meeting the practical application requirements).
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
Use of the word “means” (or “step for”) in a claim with functional language creates a rebuttable presumption that the claim element is to be treated in accordance with 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph). The presumption that 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph) is invoked is rebutted when the function is recited with sufficient structure, material, or acts within the claim itself to entirely perform the recited function.
Absence of the word “means” (or “step for”) in a claim creates a rebuttable presumption that the claim element is not to be treated in accordance with 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph). The presumption that 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph) is not invoked is rebutted when the claim element recites function but fails to recite sufficiently definite structure, material or acts to perform that function.
Claim elements in this application that use the word “means” (or “step for”) are presumed to invoke 35 U.S.C. 112(f) except as otherwise indicated in an Office action. Similarly, claim elements that do not use the word “means” (or “step for”) are presumed not to invoke 35 U.S.C. 112(f) except as otherwise indicated in an Office action.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier.
Such claim limitation(s) is/are:
Segmentation component in claim 16
Noise component in claim 17
Superpixel component in claim 18
Combination component in claim 19
**The components listed above has been interpreted as tied to the structure of a processor as disclosed in the originally filed specification at least at paragraph [0047]**
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 10-11 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph for lack of antecedent basis.
In regards to dependent claim 10, the limitation recites “the background image” in line 6, in which no previous instance of “a background image” has been provided, and thus there is insufficient antecedent basis for this limitation in the claim.
In regards to dependent claim 11, the limitation recites “the presence” in lines 3-5, in which no previous instance of “a presence” has been provided, and thus there is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-3, 5, 6, 9-13, and 15-19 are rejected under 35 U.S.C. 102(a)(1)/(a)(2) as being anticipated by Lund (US 2019/0114743 A1, hereinafter referenced “Lund”).
In regards to claim 1. Lund discloses a method (Lund, Abstract) comprising:
-obtaining an image including text and a region overlapping the text, wherein the text comprises a first color (Lund, para [0040] and [0042]; Reference at [0040] discloses API 310 allows images to be input to the server so that the server can recognize and extract information from the images. The images that are input through API 310 are provided to extraction service 320, which performs the recognition of relevant portions of the images and extraction of text or other information (images, signatures, etc.) from the relevant portions of the images (i.e. obtaining an image including text and a region overlapping the text). Para [0042] discloses the pre-processing includes linearizing and centering the image, converting color in the image to gray scale and normalizing the grayscale to a range from −0.5 to 0.5 (i.e. thus text of image starts with a default or first color));
-selecting a second color that contrasts with the first color (Lund, Fig. 4 and para [0049]; Reference discloses the images are processed to increase the contrast between text pixels and non-text pixels before OCR processing is performed. The processing to increase the text contrast is performed after the convolutional neural network identifies which pixels form text components of the image, and which pixels form non-text (background) components of the image (i.e. selecting a second color that contrasts with the first color));
-and generating a modified image including the text and a modified region using a machine learning model that takes the image and the second color as input (Lund, para [0049]; Reference at [0049] discloses the processing to increase the text contrast is performed after the convolutional neural network (i.e. machine learning model) identifies which pixels form text components of the image, and which pixels form non-text (background) components of the image….Then, the intensity of each pixel may be adjusted by a variable amount that depends on the associated probability and the original intensity of the pixel. In one embodiment, the intensity-adjusted image may be processed to identify bounding boxes for text prior to performing OCR on the portions of the image within the bounding boxes. The intensity-adjusted image interpreted as the modified image including text and a modified region regarding the background as the contrast of intensity is interpreted as the use of a secondary color as input),
-wherein the modified region overlaps the text and includes the second color (Lund, para [0049]; Reference discloses the processing to increase the text contrast is performed after the convolutional neural network identifies which pixels form text components of the image, and which pixels form non-text (background) components of the image (i.e. image having overlap regarding text and background components)….Then, the intensity of each pixel may be adjusted by a variable amount that depends on the associated probability and the original intensity of the pixel. In one embodiment, the intensity-adjusted image may be processed to identify bounding boxes for text prior to performing OCR on the portions of the image within the bounding boxes. The intensity-adjusted image interpreted as the modified image including text and second color regarding contrast of intensity)).
In regards to claim 2. Lund discloses the method of claim 1.
Lund further discloses
-further comprising: segmenting the region to identify one or more objects overlapping the text; and applying the second color to the one or more objects to obtain a first modified region, wherein the modified image is generated based on the first modified region (Lund, para [0054]; Reference discloses this may be achieved, for example, by identifying both text and non-text elements in the image, and then increasing the contrast between the text and non-text elements (i.e. segmenting the region to identify one or more objects overlapping the text)…Based on the values in this heat map, the intensities (lightness and darkness) of the pixels are adjusted to increase the contrast of the text against the background. OCR can then be performed on the increased-contrast document image (i.e. applying the second color to the one or more objects to obtain a first modified region, wherein the modified image is generated based on the first modified region) with increased efficiency, and data can be extracted from the recognized characters.).
In regards to claim 3. Lund discloses the method of claim 1.
Lund further discloses
-further comprising: adding noise to the region overlapping the text to obtain a noisy image, wherein the modified image is generated based on the noisy image (Lund, para [0058] and [0059]; Reference at [0058] discloses the scaled probability map is then used to apply intensity corrections to the original input image, producing a modified, intensity-corrected image (635). In one embodiment, the degree to which the pixels are lightened or darkened is dependent upon the degree of certainty with which the pixels are determined to be text or non-text (i.e. adding noise to overlapping region). Para [0059] discloses there are known methods for making conversions between these systems. In this case, the advantage of the HSL system is that only one of the values—lightness—needs to be modified to change the intensity of the pixel. The modified intensity-corrected image interpreted as generated based on the noisy image regarding the corrections applied to the input image having text and non-text overlapping regions).
In regards to claim 5. Lund discloses the method of claim 3.
Lund further discloses
-wherein: at least a portion of the noise comprises colored noise corresponding to the second color image (Lund, para [0058] and [0059]; Reference at [0058] discloses the scaled probability map is then used to apply intensity corrections to the original input image, producing a modified, intensity-corrected image (635). In one embodiment, the degree to which the pixels are lightened or darkened is dependent upon the degree of certainty with which the pixels are determined to be text or non-text (i.e. adding noise to overlapping region). Para [0059] discloses in one embodiment, the process of adjusting the intensities of the pixels involves a conversion of the pixel from an RGB representation to an HSL representation…There are known methods for making conversions between these systems. In this case, the advantage of the HSL system is that only one of the values—lightness—needs to be modified to change the intensity of the pixel (i.e. colors outside of black or white for the modified intensity image). The modified intensity-corrected image interpreted as generated based on the noisy image)).
In regards to claim 6. Lund discloses the method of claim 1.
Lund further discloses
-further comprising: combining the image and the modified image to obtain a combined image (Lund, para [0058]; Reference discloses the scaled probability map is then used to apply intensity corrections to the original input image, producing a modified, intensity-corrected image (635) (i.e. interpreted as containing image having background and text thus being a combined image)).
In regards to claim 9. Lund discloses a non-transitory computer readable medium storing code, the code comprising instructions executable by a processor (Lund, para [0010]) to:
-obtain an image including text and a region overlapping the text, wherein the text comprises a first color (Lund, para [0040] and [0042]; Reference at [0040] discloses API 310 allows images to be input to the server so that the server can recognize and extract information from the images. The images that are input through API 310 are provided to extraction service 320, which performs the recognition of relevant portions of the images and extraction of text or other information (images, signatures, etc.) from the relevant portions of the images (i.e. obtaining an image including text and a region overlapping the text). Para [0042] discloses the pre-processing includes linearizing and centering the image, converting color in the image to gray scale and normalizing the grayscale to a range from −0.5 to 0.5 (i.e. thus text of image starts with a default or first color));
-select a second color from the region overlapping the text, wherein the second color contrasts with the first color (Lund, Fig. 4 and para [0049]; Reference discloses the images are processed to increase the contrast between text pixels and non-text pixels before OCR processing is performed. The processing to increase the text contrast is performed after the convolutional neural network identifies which pixels form text components of the image, and which pixels form non-text (background) components of the image (i.e. selecting a second color that contrasts with the first color));
-and generate a modified image including the text and a modified region using a machine learning model that takes the image and the second color as input (Lund, para [0049]; Reference at [0049] discloses the processing to increase the text contrast is performed after the convolutional neural network (i.e. machine learning model) identifies which pixels form text components of the image, and which pixels form non-text (background) components of the image….Then, the intensity of each pixel may be adjusted by a variable amount that depends on the associated probability and the original intensity of the pixel. In one embodiment, the intensity-adjusted image may be processed to identify bounding boxes for text prior to performing OCR on the portions of the image within the bounding boxes. The intensity-adjusted image interpreted as the modified image including text and a modified region regarding the background as the contrast of intensity is interpreted as the use of a secondary color as input),
-wherein the modified region overlaps the text and includes the second color (Lund, para [0049]; Reference discloses the processing to increase the text contrast is performed after the convolutional neural network identifies which pixels form text components of the image, and which pixels form non-text (background) components of the image (i.e. image having overlap regarding text and background components)….Then, the intensity of each pixel may be adjusted by a variable amount that depends on the associated probability and the original intensity of the pixel. In one embodiment, the intensity-adjusted image may be processed to identify bounding boxes for text prior to performing OCR on the portions of the image within the bounding boxes. The intensity-adjusted image interpreted as the modified image including text and second color regarding contrast of intensity)).
In regards to claim 10. Lund discloses the non-transitory computer readable medium of claim 9.
Lund further discloses
-wherein the code further comprises instructions executable by the processor to: segment the image to identify one or more objects in the region overlapping the text; and apply the second color to the one or more objects to obtain a first modified region, wherein the background image is generated based on the first modified region (Lund, para [0054]; Reference discloses this may be achieved, for example, by identifying both text and non-text elements in the image, and then increasing the contrast between the text and non-text elements (i.e. segmenting the region to identify one or more objects overlapping the text)…Based on the values in this heat map, the intensities (lightness and darkness) of the pixels are adjusted to increase the contrast of the text against the background. OCR can then be performed on the increased-contrast document image (i.e. applying the second color to the one or more objects to obtain a first modified region, wherein the background image is generated based on the first modified region) with increased efficiency, and data can be extracted from the recognized characters.).
In regards to claim 11. Lund discloses the non-transitory computer readable medium of claim 10.
Lund further discloses
-wherein the code further comprises instructions executable by the processor to: compute a probability score for the one or more objects indicating a likelihood of the presence of the one or more objects (Lund, para [0047]; Reference discloses although the boundaries are clearly delineated in FIG. 5B for purposes of simplicity, it should be noted that the probabilities need not be binary representations. In other words, each pixel may have a probability that is in a range from 0 (0%) to 1 (100%) (i.e. probability score for text or object presence). This is illustrated in FIG. 5C. FIG. 5C depicts, for a single row of the pixels on the image, the probability that each of these pixels is within the boundaries of a text character in the document image);
-determine a low probability for the presence of the one or more objects based on the probability score; and extract a plurality of superpixels from the region overlapping the text based on the determination, wherein the first modified region includes the plurality of superpixels (Lund, para [0047] and [0049]; Reference at [0047] discloses this is illustrated in FIG. 5C. FIG. 5C depicts, for a single row of the pixels on the image (i.e. superpixels), the probability that each of these pixels is within the boundaries of a text character in the document image. Thus, the pixels in groups 540, 542 and 544 have higher probabilities of being text pixels, while the pixels in groups 550, 552, 554 and 556 have lower probabilities of being text pixels (i.e., higher probabilities of being non-text pixels) (i.e. determining high and low probability for presence of objects based on percentage or score)). Para [0049] discloses the processing to increase the text contrast is performed after the convolutional neural network identifies which pixels form text components of the image, and which pixels form non-text (background) components of the image (i.e. extracted superpixels)….Then, the intensity of each pixel may be adjusted by a variable amount that depends on the associated probability and the original intensity of the pixel. In one embodiment, the intensity-adjusted image may be processed to identify bounding boxes for text prior to performing OCR on the portions of the image within the bounding boxes (i.e. wherein the first modified region includes the plurality of superpixels)).
In regards to claim 12. Lund discloses the non-transitory computer readable medium of claim 9.
Lund further discloses
-wherein the code further comprises instructions executable by the processor to: add noise to the image in the region overlapping the text to obtain a noisy image, wherein the modified image is generated based on the noisy image (Lund, para [0058] and [0059]; Reference at [0058] discloses the scaled probability map is then used to apply intensity corrections to the original input image, producing a modified, intensity-corrected image (635). In one embodiment, the degree to which the pixels are lightened or darkened is dependent upon the degree of certainty with which the pixels are determined to be text or non-text (i.e. adding noise to overlapping region). Para [0059] discloses there are known methods for making conversions between these systems. In this case, the advantage of the HSL system is that only one of the values—lightness—needs to be modified to change the intensity of the pixel. The modified intensity-corrected image interpreted as generated based on the noisy image regarding the corrections applied to the input image having text and non-text overlapping regions).
In regards to claim 13. Lund discloses the non-transitory computer readable medium of claim 9.
Lund further discloses
-wherein the code further comprises instructions executable by the processor to: combine the image and the modified image to obtain a combined image (Lund, para [0058]; Reference discloses the scaled probability map is then used to apply intensity corrections to the original input image, producing a modified, intensity-corrected image (635) (i.e. interpreted as containing image having background and text thus being a combined image)).
In regards to claim 15. Lund discloses an apparatus for image editing (Lund, Abstract), comprising:
-a processor (Lund, Fig. 1; Reference discloses Processor 122);
-a memory including instructions executable by the processor to perform operations (Lund, para [0028]; Reference discloses server 120 utilizes a processor 122 that receives program instructions from a memory 126 which implement a convolutional neural network 124) including:
-obtaining an image including text and a region overlapping the text, wherein the text comprises a first color (Lund, para [0040] and [0042]; Reference at [0040] discloses API 310 allows images to be input to the server so that the server can recognize and extract information from the images. The images that are input through API 310 are provided to extraction service 320, which performs the recognition of relevant portions of the images and extraction of text or other information (images, signatures, etc.) from the relevant portions of the images (i.e. obtaining an image including text and a region overlapping the text). Para [0042] discloses the pre-processing includes linearizing and centering the image, converting color in the image to gray scale and normalizing the grayscale to a range from −0.5 to 0.5 (i.e. thus text of image starts with a default or first color));
-selecting a second color that contrasts with the first color (Lund, Fig. 4 and para [0049]; Reference discloses the images are processed to increase the contrast between text pixels and non-text pixels before OCR processing is performed. The processing to increase the text contrast is performed after the convolutional neural network identifies which pixels form text components of the image, and which pixels form non-text (background) components of the image (i.e. selecting a second color that contrasts with the first color));
-and generate a background image for the text based on the second color using a machine learning model (Lund, para [0049]; Reference at [0049] discloses the processing to increase the text contrast is performed after the convolutional neural network (i.e. machine learning model) identifies which pixels form text components of the image, and which pixels form non-text (background) components of the image….Then, the intensity of each pixel may be adjusted by a variable amount that depends on the associated probability and the original intensity of the pixel. In one embodiment, the intensity-adjusted image may be processed to identify bounding boxes for text prior to performing OCR on the portions of the image within the bounding boxes. The intensity-adjusted image interpreted as the background image including text and a background as the contrast of intensity is interpreted as the use of a secondary color),
-wherein the background image includes the second color in a region corresponding to the text (Lund, para [0049]; Reference at [0049] discloses the processing to increase the text contrast is performed after the convolutional neural network identifies which pixels form text components of the image, and which pixels form non-text (background) components of the image ….Then, the intensity of each pixel may be adjusted by a variable amount that depends on the associated probability and the original intensity of the pixel. In one embodiment, the intensity-adjusted image may be processed to identify bounding boxes for text prior to performing OCR on the portions of the image within the bounding boxes. The intensity-adjusted image interpreted as the background image including text and second color regarding contrast of intensity)).
In regards to claim 16. Lund discloses the apparatus of claim 15.
Lund further discloses
-further comprising: a segmentation component configured to segment the image to identify one or more objects (Lund, para [0038]; Reference discloses extraction service 320 (i.e. segmentation component) may be configured as described herein to modify received images as described herein to adjust their pixel intensities, increasing the contrast between text and non-text pixels…Extraction service 320 uses a convolutional neural network 322 to identify regions within input images where the desired types of information (e.g., text, images, signatures, etc.) are found (i.e. segmenting the region to identify one or more objects)).
In regards to claim 17. Lund discloses the apparatus of claim 15.
Lund further discloses
-further comprising: a noise component (Lund, Fig. 1; Computational neural network 124) configured to add noise to the image in the region corresponding to the text (Lund, para [0058] and [0059]; Reference at [0058] discloses the scaled probability map is then used to apply intensity corrections to the original input image, producing a modified, intensity-corrected image (635). In one embodiment, the degree to which the pixels are lightened or darkened is dependent upon the degree of certainty with which the pixels are determined to be text or non-text (i.e. adding noise to overlapping region). Para [0059] discloses there are known methods for making conversions between these systems. In this case, the advantage of the HSL system is that only one of the values—lightness—needs to be modified to change the intensity of the pixel. The modified intensity-corrected image interpreted as generated based on the noisy image regarding the corrections applied to the input image having text and non-text overlapping regions).
In regards to claim 18. Lund discloses the apparatus of claim 15, further comprising:
Lund further discloses
-a superpixel component (Lund, Fig. 1; Computational neural network 124) configured to extract a plurality of superpixels from the region corresponding to the text (Lund, para [0047] and [0049]; Reference at [0047] discloses this is illustrated in FIG. 5C. FIG. 5C depicts, for a single row of the pixels on the image, the probability that each of these pixels is within the boundaries of a text character in the document image. Para [0049] discloses the processing to increase the text contrast is performed after the convolutional neural network identifies which pixels form text components of the image, and which pixels form non-text (background) components of the image (i.e. extracted superpixels from the region corresponding to text)).
In regards to claim 19. Lund discloses the apparatus of claim 15.
Lund further discloses
-further comprising: a combination component (Lund, Fig. 1; Computational neural network 124) configured to combine the image and the background image to obtain a combined image (Lund, para [0058]; Reference discloses the scaled probability map is then used to apply intensity corrections to the original input image, producing a modified, intensity-corrected image (635) (i.e. interpreted as containing image having background and text thus being a combined image)).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 4, 7, 8, and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Lund (US 2019/0114743 A1) in view of Sousa (US 2021/0217215 A1, hereinafter referenced “Sousa”).
In regards to claim 4. Lund discloses the method of claim 3.
Lund does not explicitly disclose but Sousa teach
-further comprising: generating a mask indicating the region overlapping the text, wherein the noise is added to the image b