Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Claims 1-20 were previously pending. Claims 1, 8, 15 have been amended. No new claims have been added and no claims have been canceled. Thus claims 1-20 are currently pending including independent claims 1, 8, 15.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-4, 8-11, 15-18 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Mironica (US 20220392025 A1).
Regarding claim 1, Mironica discloses a computer-implemented method comprising (Fig. 1, [0036] computer system environment for implementing image restoration):
providing, for display on a client device, a digitalized image digitalized from a physical record (Fig. 11, [0127] the system provides an image restoration interface for display on the client device, wherein the user can select image restoration elements to be executed; Fig. 2, [0047] the digital image is a scan or other digitized version of an old, historic photograph that has been degraded; Fig. 1, [0039]-[0040] the image is also displayed on the user interface for the user);
determining, by utilizing an image enhancement engine (Fig. 11, [0127]-[0128] the interface includes selectable checkboxes for each restoration element and the system detects user interaction with the interface to select elements):
a first portion of the digitalized image to restore by classifying the first portion as a first type of image component and using a restoration pipeline of the image enhancement engine on the first portion according to the first type of image component (Fig. 12-13, [0135] the local defect manager detects, extracts, or identifies local defects such as scratches in the image, including repairing the local defects utilizing an inpainting model to replace the local defect pixels with pixels that reduce the appearance of local defects; Fig. 14, [0145] generating a segmentation mask to indicate locations of one or more scratches (i.e. indicating a portion which is to be restored via a segmentation mask); and
a second portion of the digitalized image to colorize by classifying the second portion as a second type of image component and using a colorization pipeline of the image enhancement engine on the second portion according to the second type of image component (Fig. 13, [0136] a global imperfection manager is included in the image restoration system and detects or identifies global imperfections with a digital image and utilizes a global correction neural network to identify and correct global imperfections; Fig. 15, [0152]-[0153] improving the global imperfection may involve improving image color; [0025]-[0026] global imperfections in the form of faded color or saturation);
restoring the first portion of the digitalized image utilizing the restoration pipeline by applying a circularity measure rectifying a scratch depicted in the first portion of the digitalized image (Fig. 14, [0146]-[0147] restoring local defects via inpainting pixels; [0145] the local defect may be a scratch indicated by portion of a segmentation mask; Fig. 3, [0063] the inpainting model involves pixels from the digital image that blend with the surroundings of the local defect to perform inpainting and reduce the appearance of the scratch; Fig. 11, [0129] the local defect scratch reduction option on the user interface includes a slider bar, based on user interaction with the slider the image restoration system modifies an area of the digital image to analyze and repair for local defects; [0129] the image restoration system 102 increases or decreases a measure or degree of modification applied to a digital image e.g. to modify an area of the digital image to analyze and repair, see Fig. 11 element 1108); and
colorizing the second portion of the digitalized image utilizing the colorization pipeline to modify colors of the second portion of the digitalized image according to a colorfulness metric ([0025]-[0026] global imperfections in the form of faded color or saturation; Fig. 15, [0152] modifying the feature vector by using a global correction neural network according to learned parameters to reduce appearance of the one or more global imperfections such as improving image color according to learned parameters; [0153] generating from the modified feature vector a modified digital image depicting reductions in the one or more global imperfections; see also Fig. 11 "colorize" selectable option on the restoration user interface)).
Regarding claim 2, Mironica discloses the computer-implemented method of claim 1 as applied above. Mironica further discloses wherein restoring the first portion of the digitalized image comprises: detecting a disruptive image portion that would disrupt one or more of colorization or restoration; and removing the disruptive image portion from the digitalized image (Fig. 3, [0061] identify locations of local defects in the image using a neural network; [0064] utilize inpainting model to fill in the local defects and generate a modified image with defects removed).
Regarding claim 3, Mironica discloses the computer-implemented method of claim 2 as applied above. Mironica further discloses wherein the disruptive image portion comprises an image portion depicting one or more of image noise, a scratch, a fold, or a tear ([0047] example local defects include dust, scratches, tears, folds, and creases).
Regarding claim 4, Mironica discloses the computer-implemented method of claim 1 as applied above. Mironica further discloses wherein restoring the first portion of the digitalized image comprises utilizing a neural network trained to retain disappearing artifacts depicted in the digitalized image (Fig. 3, [0060]-[0063] a defect detection neural network detects defects which are then repaired using an inpainting model, the neural network processes pixels of the digital image to extract features and generates a segmentation mask, which is then used in the inpainting (i.e. the neural network is trained to retain the defect artifacts which are then filled in by the repair model)).
Regarding claim 8, Mironica discloses everything claimed as applied above (see rejection of claim 1) further including a system comprising: at least one processor; and a non-transitory computer-readable medium storing instructions that, when executed by the at least one processor (Fig. 1, [0036] computer system environment for implementing image restoration; Fig. 17, [0172] the computing device can comprise a processor and a memory; [0174] the memory is a non-volatile memory and is coupled to the processor for storing programs executed by the processor).
Regarding claim 9, Mironica discloses the system of claim 8 as applied above. Mironica further discloses everything claimed as applied above (see rejection of claim 2).
Regarding claim 10, Mironica discloses the system of claim 9 as applied above. Mironica further discloses everything claimed as applied above (see rejection of claim 3).
Regarding claim 11, Mironica discloses the system of claim 8 as applied above. Mironica further discloses everything claimed as applied above (see rejection of claim 4).
Regarding claim 15, Mironica discloses everything claimed as applied above (see rejection of claim 1) further including a non-transitory computer-readable medium storing instructions that, when executed by at least one processor (Fig. 1, [0036] computer system environment for implementing image restoration; a non-volatile memory and is coupled to the processor for storing programs executed by the processor).
Regarding claim 16, Mironica discloses the non-transitory computer-readable medium of claim 15 as applied above. Mironica further discloses everything claimed as applied above (see rejection of claim 2).
Regarding claim 17, Mironica discloses the non-transitory computer-readable medium of claim 16 as applied above. Mironica further discloses everything claimed as applied above (see rejection of claim 3).
Regarding claim 18, Mironica discloses the non-transitory computer-readable medium of claim 15 as applied above. Mironica further discloses everything claimed as applied above (see rejection of claim 4).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 5-6, 12-13, 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mironica (US 20220392025 A1) in view of Peters (US 20130154934 A1).
Regarding claim 5, Mironica discloses the computer-implemented method of claim 1 as applied above. Mironica further discloses generating an enhanced image by utilizing the image enhancement engine to perform restoration and colorization (Fig. 14, [0146]-[0147] restoring local defects via inpainting pixels; [0145] the local defect may be a scratch indicated by portion of a segmentation mask; Fig. 15, [0152]-[0153] the global correction neural network may improve image color according to learned parameters (see also Fig. 11 "colorize" selectable option on the restoration user interface)),
providing, for display on the client device, the enhanced image and a notification indicating the restoration and colorization performed ([0040] the server transmits data to the client device to cause the client device to display or present a modified digital image with reduced local defects and global defects).
Mironica fails to disclose detecting an additional digitalized image stored within a genealogy tree of a user account associated with the client device; performing the image editing on the additional digitalized image; and providing, for display on the client device, the enhanced additional digitalized image stored within the genealogy tree.
Peters, in a related system from the same field of endeavor of image processing for creating and displaying family trees (Abstract), discloses detecting an additional digitalized image stored within a genealogy tree of a user account associated with the client device (Fig. 1, [0101] tree generation module checks tree data store 48 or database 42 to determine whether a tree has already been constructed for the user and retrieves associated portraits on the display 16 of the user device; [0059] the images being digital photographs uploaded from a digital camera);
performing image editing on the additional digitalized image (Fig. 5, [0060] retrieve picture from store and present to the cropping module to crop the portrait for display ); and providing, for display on the client device, the enhanced additional digitalized image stored within the genealogy tree (Fig. 1, [0060] the portion of the picture so extracted has predetermined proportions and constitutes a portrait picture for purposes of display in a family tree on the client device).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to combine Peters with Mironica and detect an additional image, perform image editing on the additional image, and provide the enhanced image for display within a genealogy tree, as disclosed by Peters, as part of a computer-implemented method for restoring portions of a digitalized image, as disclosed by Mironica, for the purpose of enhancing the user of experience of generating and viewing a genealogical family tree based on portrait images (see Peters: [0008]-[0012]).
Regarding claim 6, Mironica in view of Peters discloses the computer-implemented method of claim 5 as applied above. Mironica fails to disclose providing, for display on the client device, an option to save the enhanced image to the genealogy tree of the user account; and in response to a selection of the option, saving the enhanced image to the genealogy tree of the user account.
Peters, in a related system from the same field of endeavor of image processing for creating and displaying family trees (Abstract), discloses providing, for display on the client device, an option to save the enhanced image to the genealogy tree of the user account ([0092] a user is able to edit and save a life story consisting of at least one portrait; [0095] the option to edit and save is provided as text displayed at the top); and in response to a selection of the option, saving the enhanced image to the genealogy tree of the user account ([0094] a constructed life story may be retained in a storage unit 60 of the electronic device; [0119] members with access to a family tree can view data including life story information).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to combine Peters with Mironica and provide an option to save the enhancing image to the tree and save the image to the tree, as disclosed by Peters, as part of a computer-implemented method for restoring portions of a digitalized image, as disclosed by Mironica, for the purpose of enhancing the user of experience of generating and viewing a genealogical family tree based on portrait images (see Peters: [0008]-[0012]).
Regarding claim 12, Mironica discloses the system of claim 8 as applied above. Mironica in view of Peters discloses everything claimed as applied above (see rejection of claim 5).
Regarding claim 13, Mironica in view of Peters discloses the system of claim 12 as applied above. Mironica in view of Peters discloses everything claimed as applied above (see rejection of claim 6).
Regarding claim 19, Mironica discloses the non-transitory computer-readable medium of claim 15 as applied above. Mironica in view of Peters discloses everything claimed as applied above (see rejection of claim 5).
Regarding claim 20, Mironica in view of Peters discloses the non-transitory computer-readable medium of claim 19 as applied above. Mironica in view of Peters discloses everything claimed as applied above (see rejection of claim 6).
Claim(s) 7, 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mironica (US 20220392025 A1) in view of Lee (US 20210133932 A1).
Regarding claim 7, Mironica discloses the computer-implemented method of claim 1 as applied above. Mironica fails to disclose generating a grayscale image portion by converting the second portion of the digitalized image to grayscale; and colorizing the grayscale image portion using color channel weights to prevent a tie-dye effect.
Lee, in a related system from the same field of endeavor of restoring images including colorization (Abstract), discloses generating a grayscale image portion by converting the second portion of the digitalized image to grayscale (Fig. 13, [0290] an input image is enhanced using a DNN via colorization through a grayscale conversion process; Fig. 11, [0283]-[0284] an input image is converted into grayscale); and colorizing the grayscale image portion using color channel weights to prevent a tie-dye effect (Fig. 11, 13, [0286] colorize the input image that has been converted into grayscale by using a DNN (see also [0292]); [0069]-[0070] wherein the DNN includes weights; [0290] and wherein the colorization is based on color channels).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to combine Lee with Mironica and generate a grayscale image portion by converting to grayscale and colorize the grayscale portion using weights, as disclosed by Lee, as part of a computer-implemented method for restoring portions of a digitalized image, as disclosed by Mironica, for the purpose of improved color restoration of images (see Lee: [0007]-[0011]).
Regarding claim 14, Mironica discloses the system of claim 8 as applied above. Mironica in view of Lee discloses everything claimed as applied above (see rejection of claim 7).
Response to Arguments
Applicant's arguments filed 07/21/2026 have been fully considered but they are not persuasive.
Applicant asserts on page 12 that “the office fails to establish the Mironica discloses classifying portions of an image as a type of image component and using a pipeline according to the classified image component as recited in the amended claims". Examiner disagrees. As stated above, Mironica discloses identifying portions of the image as local defects and global imperfections (i.e. first and second portions containing first and second components, respectively) in at least [0135]-[0136]. Thus the rejection of amended claim 1 is maintained as applied above.
Applicant further asserts on page 14 that "the office action fails to establish that Mironica discloses colorizing the second portion of a digitalized image using a colorization pipeline as recited in the amended claims". Examiner disagrees. As stated above, Mironica discloses identifying global imperfections (i.e. imperfections of a second portion of the image wherein the second portion is the entire image or most of the image) such as faded color or saturation and improving the color imperfection according to learned parameters of a global correction neural network (colorization pipeline). Thus, the rejection of amended claim 1 is maintained as stated above.
Applicant further asserts on page 14 that "the office action fails to establish that Mironica discloses applying a circularity measure rectifying a scratch depicted in the first portion of the digitalized image". Examiner disagrees. The circularity measure is not specifically defined by the claim, nor does the specification provide a specific definition of the term or of how it is applied to rectify a scratch in the digitalized image. Thus, the examiner interprets "applying a circularity measure rectifying a scratch" broadly to mean utilizing a quantitative measure as part of a scratch repair process in an image. As stated above, Mironica discloses in [0129] to increase or decrease a measure or degree of modification applied to the digital image, for example to modify an area of the image such as a scratch. Thus, the rejection of amended claim 1 is maintained as stated above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Benzarti (Benzarti, Faouzi, and Hamid Amiri. "Repairing and Inpainting Damaged Images using Diffusion Tensor." arXiv preprint arXiv:1305.2221 (2013).) discloses repairing imperfections of digital images or videos such as removing scratches of digitalized photographs, based on a tensor which is a circle of a given radius.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CAROLINE DEPALMA whose telephone number is (571)270-0769. The examiner can normally be reached Mon-Thurs 9:00am-4pm Eastern Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CAROLINE E. DEPALMA/Examiner, Art Unit 2675
/SJ Park/Primary Examiner, Art Unit 2675
Prosecution Insights