DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This Office Action is in response to the Applicants’ communication filed on October 25, 2024. Claims 13 and 16-17 have been canceled. In virtue of this communication, claims 1-12, 14-15, 18-23 are currently presented in the instant application.
Drawings
The drawings submitted on October 25, 2024. These drawings are reviewed and accepted by the examiner.
Information Disclosure Statement
The information Disclosure Statement (IDS) Forms PTO-1449, filed on January 22, 2025 and February 04, 2025 in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosed therein was considered by the examiner.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copies have been filed on May 08, 2023.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 14-15, 18-19 and 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over YONETSUJI (US 20180342084 A1) in view of Sloat (US 20230195433 A1).
Regarding claim 1. YONETSUJI discloses an image processing method (YONETSUJI, see FIG. 3,), comprising:
acquiring a line drawing and initial color prompt information from a user (YONETSUJI, see at least par. [0041] Next, the coloring processing flow in the automatic line drawing coloring apparatus 10 of the present example is described. FIG. 3 is a flowchart showing a coloring processing flow in the automatic line drawing coloring apparatus 10 of the present example. The coloring processing in the automatic line drawing coloring apparatus 10 of the present example starts first by acquiring the line drawing data (step S01). For example, the acquisition is performed by a user selecting the line drawing data of the coloring processing target. At this time, the hint information for coloring the line drawing data may also be acquired together along with the acquisition of the line drawing data. The reduction processing is performed on the acquired line drawing data so as to be the predetermined reduced size (step S02). At this time, the line drawing data with the original size is also kept separately.);
coloring the line drawing based on the initial color prompt information to generate an initial colored image for the line drawing (YONETSUJI, see at least par. [0016] A graphical user interface method according to the present disclosure is a graphical user interface method for an automatic line drawing coloring tool provided to a client terminal connected via a communication network from a server apparatus storing the automatic line drawing coloring method and provides a graphical user interface to a display of the client terminal by causing the server apparatus to realize a line drawing data input form display function of displaying, on a display screen, a form region for a user, who operates the client terminal, to input line drawing data, a line drawing image display function of displaying, in a line drawing image display region provided on the display screen, line drawing indicated by the line drawing data inputted, and a colored image display function of displaying, in a colored image display region provided on the display screen, a colored image indicated by colored image data obtained by performing coloring processing on the line drawing data with the automatic line drawing coloring method.);
acquiring color modification information associated with the initial colored image from the user (YONETSUJI, see at least [0013] Moreover, the automatic line drawing coloring method according to the present disclosure causes the computer to realize a hint information acquisition function of acquiring hint information on coloring the line drawing data with at least one color and realize a function of performing the coloring processing with the reduced line image data and the hint information as inputs in the first coloring processing function.);
coloring the line drawing based on the target color prompt information to generate a target colored image for the line drawing (YONETSUJI, see at least par. [0020], a first coloring processing unit which performs coloring processing on the reduced line drawing data based on any one first learned model of the plurality of first learned models; and a second coloring processing unit which performs coloring processing on original line drawing data with the colored reduced data obtained by performing the coloring processing on the reduced line drawing data by the first coloring processing unit and the original line drawing data as inputs based on any one second learned model of the plurality of second learned models..).
YONETSUJI does not disclose generating target color prompt information based on the initial color prompt information, the color modification information and the initial colored image. However,
Sloat discloses:
generating target color prompt information based on the initial color prompt information, the color modification information and the initial colored image (Sloat, see at least par. [0039] In addition to displaying the drop-down list box 120 to select a first color operation, as shown in FIG. 2A, the GUI 200 may also automatically generate and display a secondary list box 230 for selecting a second change operation that is to be applied to the target color. In the example depicted in FIG. 2A, the secondary list box 230 prompts the user to select a relative luminance operation. Option 242 applies a maximum relative luminance value to the target color in deriving the desired color given the selected saturation and RGB ratio of the target color. Option 244 applies the minimum relative luminance value given the selected saturation and the RGB ratio of the target color. Option 246 holds the relative luminance value to that of the target color. Option 248 prompts the user to enter a relative luminance value that is to be applied to the target color to derive the desired color. Option 250 prompts the user to enter a secondary hexadecimal color value for a color. The relative luminance for the derived color is selected to match the color represented by the entered secondary hexadecimal value. Option 252 prompts the user for a contrast ratio separation from that of the target color and sets the contrast ratio for the derived color to that of a color that is separated by the specified contrast ratio separation from the target color. The contrast ratio specifies the relationship between the relative luminance of two different colors. Option 254 prompts the user to enter a hexadecimal color value for a secondary color, prompts the user for a contrast ratio separation from that of the secondary color and sets the contrast ratio for the derived color to that of a color that is separated by the specified contrast ratio separation from the secondary color.);
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system and method of YONETSUJI, with generating target color prompt information based on the initial color prompt information, the color modification information and the initial colored image, as provided by Sloat. The modification provides an improved system and method for coloring a line draw, thereby to allow the software designer to enter information regarding the reference color and the desired changes in color characteristics. The user interface may display the reference color and that resulting desired color so that the software designer may get a visual appreciation for the difference in the colors. (Sloat, see par. [0030]).
Regarding claim 2. YONETSUJI in view of Sloat discloses the method according to claim 1 (as rejected claim 1 above), and YONETSUJI in view of Sloat further discloses wherein the initial color prompt information indicates one or more initial regions in the line drawing and respective initial colors to color the one or more regions specified by the user (YONETSUFI, see at least par. [0049] Moreover, as shown in FIG. 4B, in the screen displaying the line drawing image and the colored image, hint information input tools for designating a portion which should be colored with a selected color in the line drawing data displayed in the line drawing image display region are displayed. In the example shown in FIG. 4B, the hint information input tools are “return to previous work”, “progress to next work”, “select a pen for inputting the hint information”, “delete the inputted hint information (eraser)”, and “select the color for the coloring”, but are not limited to these. For example, a color used for the coloring is selected by a mouse operation, and a portion which should be colored with the selected color in the line drawing image in the line drawing image display region is actually colored by a technique such as adding a dot, adding a line segment, or filling a region by a pointer, thereby providing the hint information. Then, when a coloring execution button displayed on the same screen is clicked by a mouse operation or the like, the coloring processing is executed in a state in which the hint information is included, and the colored image reflecting the hint information is displayed in the colored image display region.).
Regarding claim 3. YONETSUJI in view of Sloat discloses the method according to claim 2 (as rejected above), and YONETSUJI in view of Sloat further discloses wherein the coloring the line drawing based on the initial color prompt information to generate the initial colored image for the line drawing comprises:
inputting the one or more initial regions and the respective initial colors and the line drawing into a first model to acquire the initial colored image for the line drawing, wherein colors of the one or more initial regions in the initial colored image are consistent with the initial colors (YONETSUJI , see at least par. [0051] The colored image data obtained by the coloring processing is transmitted to the client terminal. At the client terminal, the line drawing image indicated by the line drawing data is displayed in the line drawing image display region provided on the display screen, and the colored image indicated by the colored image data is displayed in the colored image display region (step S23). Moreover, the hint information input tools are displayed on the display screen, and the input of the hint information by the hint information input tools is accepted (step S24). The user who wishes to re-color by providing the hint of the coloring provides the hint information on the coloring to the line drawing image displayed in the line drawing image display region. Then, when the re-coloring is instructed by clicking the coloring execution button (step S25-Y), the hint information and the line drawing data are transmitted to the server apparatus, and the re-coloring processing is executed in a state in which the hint information is attached (step S22). The colored image data which is given the hint information and obtained by the re-coloring processing is transmitted to the client terminal, and the colored image indicated by the colored image data given the hint information is displayed in the colored image display region (step S23). In this way, at the stage where the colored image data desired by the user is obtained, no further re-coloring is performed (step S25-N), the process of the automatic line drawing coloring tools ends.).
Regarding claim 14. YONETSUJI discloses an electronic device (YONETSUJI, see FIG. 1 and par. [0028-0029]) performs same steps of claim 1. Therefore, claim 14 is further rejected based on the same rationale of claim 1 set forth above and incorporated herein.
Regarding claim 15. YONETSUJI discloses a non-transitory computer readable storage medium storing a computer program executed by a processor (YONETSUJI, see FIG. 1 and pars. [0028-0029]) for implementing an image processing method of claim 1. Therefore, claim 15 is further rejected based on the same rationale of claim 1 set forth above and incorporated herein.
Regarding claim 18. The electronic device according to claim 18 performs the same step of claim 2. Therefore, claim 18 is further rejected based on the same rationale as claim 2 set forth above and incorporated herein.
Regarding claim 19. The electronic device according to claim 19, performs the same step of claim 3. Therefore, claim 19 is further rejected based on the same rationale as claim 3 set forth above and incorporated herein.
Regarding claim 21. The non-transitory computer readable storage medium according to claim 21 performs the same step of claim 2. Therefore, claim 18 is further rejected based on the same rationale as claim 2 set forth above and incorporated herein.
Regarding claim 22. The non-transitory computer readable storage medium according to claim 22 performs the same step of claim 3. Therefore, claim 22 is further rejected based on the same rationale as claim 3 set forth above and incorporated herein.
Claims 4, 6-8 are rejected under 35 U.S.C. 103 as being unpatentable over YONETSUJI (US 20180342084 A1) in view of Sloat (US 20230195433 A1), as applied claim 2 above and further in view of Musha et al. (US 20030046267 A1).
Regarding claim 4. YONETSUJI in view of Sloat discloses the method according to claim 2 (as rejected above), and YONETSUJI in view of Sloat does not disclose wherein the acquiring the color modification information associated with the initial colored image comprises:
acquiring the color modification information associated with one or more color block regions of the plurality of color block regions (YONETSUJI, see at least par. [0052] As described above, the GUI is provided to the display screen of the display of the client terminal from the server apparatus to provide the automatic line drawing coloring tools to the user by the GUI, and the line drawing image display region and the colored image display region are provided in the same display screen as functions of the GUI. Thus, the user can view the original line drawing image and the colored image side by side so that the atmosphere of the creation or visual impression that varies before and after the coloring can be compared directly. Moreover, the hint information for designating the portion to be colored with the selected color in the line drawing image indicated by the line drawing data displayed in the line drawing image display region can be inputted, and the re-coloring processing can be executed in a state in which the hint information is attached. Thus, the user can execute automatic coloring on the line drawing image by freely giving the coloring hint. Note that this addition of the hint information may not designate that the place is painted with the designated color, but may cause the learned models to execute the coloring in a state in which the hint information is included. Thus, it can be appreciated that the coloring may not be performed with the designated color. The learnings are performed by including the hint information in the learning processes of the first learned model and the second learned model used by the automatic line drawing coloring method. Thus, it can be appreciated that how the designated hint information is adopted is determined by the sample data and the tendency of the hint information used for the learnings. This is different compared to functions of the coloring processing in conventional image editing software or the like which performs coloring with a designated color.).
YONETSUJI in view of Sloat does not disclose color blocking the initial colored image in response to a color modification request to generate an initial color block image for the initial colored image, wherein the color block image includes a plurality of color block regions. However,
Musha discloses:
color blocking the initial colored image in response to a color modification request to generate an initial color block image for the initial colored image, wherein the color block image includes a plurality of color block regions (Musha, see at least par. [0027] FIG. 8 is an illustrative drawing for explaining a mechanism of creating an image-and-area-attribute database composed of an image database and an area-attribute information database. First, images to be retrieved are prepared, extracted one by one, and manipulated as described below. An image 801 is divided into a plurality of areas 802. For brevity's sake, the image 801 is divided into areas arrayed in three rows and three columns. Among the areas, one area 803 is extracted and processed in the form of a color image. A plurality of color attributes C1, C2, C5, and C7 is extracted and combined with an index (2, 1) specifying the area, whereby area-attribute information 805 is produced. Moreover, a plurality of shape attributes L0, L2, L4, and L5 is extracted from an area 804 that has undergone gray scaling, whereby area-attribute information 805 is produced. Moreover, area-attribute information 806 is produced using a keyword as attribute information. As the area-attribute information, a ratio at which an attribute occupies an area, a frequency by which an area-attribute appears in all images in an image-and-area-attribute database to be retrieved, or a value proportional to the rareness of the area-attribute information to all the information items recorded in an area-attribute information database (an amount of information) can be adopted. The thus produced area-attribute information is recorded in the image-and-area-attribute database 807 in association with an image ID.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system and method of YONETSUJI, with color blocking the initial colored image in response to a color modification request to generate an initial color block image for the initial colored image, wherein the color block image includes a plurality of color block regions, as provided by Musha. The modification provides an improved system and method for coloring a line draw, thereby to allowing a user to create a retrieval key while looking at the results of retrieval. Specifically, a user's action of depicting a vision is closely associated with image retrieval. Images are selected with each stroke made in order to paint a color or draw a line on a canvas at a step of creating a retrieval key. The resultant images are presented to the user. (Musha, see par. [0006]).
Regarding claim 6. YONETSUJI in view of Sloat and further in view of Musha discloses the method according to claim 4 (as rejected above), and YONETSUJI in view of Sloat and further in view of Musha further discloses wherein the generating the target color prompt information based on the initial color prompt information, the color modification information and the initial colored image comprises:
acquiring a target color block image from the initial color block image based on the color modification information associated with the one or more color block regions of the plurality of color block regions, wherein the target color block image includes a plurality of color blocks (Musha, see at least par. [0020] FIG. 2 is an illustrative drawing concerned with a method of producing area-attribute information when a color is selected from the palette 106 and pained on the canvas 103. A user selects a color 201 from the palette 106, and makes a stroke 204 on the canvas 103. The canvas 103 is divided into areas. An index specifying an area, such as, (1, 2) is assigned to each area. Assume that it is detected using the GUI that the color 201 is painted over areas (0, 0) , (0, 1) , (0, 2) , and (0, 3) in that order using a mouse cursor 205. Area-attribute information items 206 of C0(0, 0), C0(0, 1), C0(0, 2), and CO(0, 3) are produced with the progress of the painting. Images having the area-attribute information items 206 are retrieved from an area-attribute information database, and provided. To be more specific, when the stroke 204 made by the user passes through the area (0,0), the area-attribute information C0(0, 0) is produced, and images having the area-attribute information are retrieved. Thereafter, when the stroke passes through the area (0, 1), the area-attribute information C0(0, 1) is produced. Images having the area-attribute information C0 (0, 1) are selected from the previously retrieved images. Likewise, images having the area-attribute information items C0(0, 2) and C0(0, 3) are selected. While the user makes a stroke, images are selected step by step. If a keyword can be designated as attribute information, the keyword is substituted for the aforesaid color. A keyword is assigned to each area, whereby image retrieval is carried out. Moreover, when an eraser 203 is selected from the palette 106, the mouse cursor is passed through an area to which attribute information has already been assigned. Consequently, the area-attribute information is deleted. Eventually, a range of retrieval can be expanded. A drawing 202 on the palette 106 will be described in conjunction with FIG. 3.);
generating the target color prompt information based on the target color block image (Musha, see at least par. [0027] FIG. 8 is an illustrative drawing for explaining a mechanism of creating an image-and-area-attribute database composed of an image database and an area-attribute information database. First, images to be retrieved are prepared, extracted one by one, and manipulated as described below. An image 801 is divided into a plurality of areas 802. For brevity's sake, the image 801 is divided into areas arrayed in three rows and three columns. Among the areas, one area 803 is extracted and processed in the form of a color image. A plurality of color attributes C1, C2, C5, and C7 is extracted and combined with an index (2, 1) specifying the area, whereby area-attribute information 805 is produced. Moreover, a plurality of shape attributes L0, L2, L4, and L5 is extracted from an area 804 that has undergone gray scaling, whereby area-attribute information 805 is produced.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system and method of YONETSUJI, with wherein the generating the target color prompt information based on the initial color prompt information, the color modification information and the initial colored image comprises: acquiring a target color block image from the initial color block image based on the color modification information associated with the one or more color block regions of the plurality of color block regions, wherein the target color block image includes a plurality of color blocks; and generating the target color prompt information based on the target color block image as provided by Musha. The modification provides an improved system and method for coloring a line draw, thereby to allowing a user to create a retrieval key while looking at the results of retrieval. Specifically, a user's action of depicting a vision is closely associated with image retrieval. Images are selected with each stroke made in order to paint a color or draw a line on a canvas at a step of creating a retrieval key. The resultant images are presented to the user. (Musha, see par. [0006]).
Regarding claim 7. YONETSUJI in view of Sloat and further in view of Musha discloses the method according to claim 6 (as rejected above), and YONETSUJI in view of Sloat and further in view of Musha further discloses wherein the target color prompt information indicates one or more target regions in the line drawing and respective target colors to color the one or more regions specified by the user (YONETSUJI, see at least par. [0020] Still further, the GUI is provided to the display screen of the display of the client terminal from the server apparatus to provide the automatic line drawing coloring tool to the user by the GUI, and the line drawing image display region and the colored image display region are provided in the same display screen as functions of the GUI. Thus, the user can view the original line drawing data and the colored image parallel so that the atmosphere of the creation that changes before and after the coloring can be compared directly. In addition, the hint information for designating the portion that should be colored with the selected color in the line drawing data displayed in the line drawing image display region can be inputted, and the re-coloring processing can be executed in a state in which the hint information is attached. Thus, the user can execute automatic coloring on the line drawing data by freely giving the coloring hint.).
Regarding claim 8. YONETSUJI in view of Sloat and further in view of Musha discloses the method according to claim 7 (as rejected above), and YONETSUJI in view of Sloat and further in view of Musha further discloses, wherein the coloring the line drawing based on the target color prompt information to generate the target colored image for the line drawing comprises:
inputting the one or more target regions and the respective target colors and the line drawing into the first model to acquire the target colored image for the line drawing, wherein colors of the one or more target regions in the target colored image are consistent with the target colors (YONETSUJI, see at least par. [0030] The line drawing data acquisition unit 11 has a function of acquiring coloring target line drawing data. The coloring target line drawing in the present disclosure is not particularly limited, but it is desirable to incorporate the target line drawing into sample data in the learning process of a learning model described later to be learned in advance. There are various line drawings with the different thicknesses of the lines and the different types of touch, and the types of line drawing that can be colored increase by learning based on various line drawing data.).
Allowable Subject Matter
Claims 5, 9-12, 20 and 23 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claim 5. YONETSUJI in view of Sloat and further in view of Musha discloses the method of claim 5. However, the limitations:
“wherein the color blocking the initial colored image in response to the color modification request to generate the initial color block image for the initial colored image comprises:
inputting the initial colored image into a second model to determine the plurality of color block regions and respective region boundaries of the initial colored image;
acquiring color mean values of pixels within respective color block regions from the initial colored image based on the region boundaries;
filling respective regions of the plurality of color block regions using respective color mean values to acquire the initial color block image.”, taken as a whole, render the claims patentably distinct over the prior art.
Regarding claims 11-12 are object because of their dependencies.
Regarding claim 9. YONETSUJI in view of Sloat discloses the method according to claim 3. However, the limitations:
wherein the first model is a coloring prompt model which is trained and obtained by the following steps:
acquiring a first sample line drawing corresponding to a first sample image;
acquiring initial sample color prompt information from the first sample image;
coloring the first sample line drawing based on the initial sample color prompt information according to the first model to be trained to generate an initial sample colored image for the first sample line drawing;
generating a first objective loss function according to the initial sample colored image and the first sample image; and
training parameters for the first model according to the initial sample colored image and the first sample image and based on backward propagation of the first objective loss function to generate the coloring prompt model.”, taken as a whole, render the claims patentably distinct over the prior art.
Regarding claim 10 is object because of its dependency.
Regarding claim 20. YONETSUJI in view of Sloat discloses the method according to claim 3 (as rejected above). However, the limitations:
“wherein the first model is a coloring prompt model which is trained and obtained by the following steps:
acquiring a first sample line drawing corresponding to a first sample image;
acquiring initial sample color prompt information from the first sample image;
coloring the first sample line drawing based on the initial sample color prompt information according to the first model to be trained to generate an initial sample colored image for the first sample line drawing;
generating a first objective loss function according to the initial sample colored image and the first sample image; and
training parameters for the first model according to the initial sample colored image and the first sample image and based on backward propagation of the first objective loss function to generate the coloring prompt model.”, taken as a whole, render the claims patentably distinct over the prior art.
Regarding claim 23. YONETSUJI in view of Sloat discloses the non-transitory computer readable storage medium according to claim 22 (as rejected above). However, the limitations:
“wherein the first model is a coloring prompt model which is trained and obtained by the following steps:
acquiring a first sample line drawing corresponding to a first sample image;
acquiring initial sample color prompt information from the first sample image;
coloring the first sample line drawing based on the initial sample color prompt information according to the first model to be trained to generate an initial sample colored image for the first sample line drawing;
generating a first objective loss function according to the initial sample colored image and the first sample image; and
training parameters for the first model according to the initial sample colored image and the first sample image and based on backward propagation of the first objective loss function to generate the coloring prompt model.”, taken as a whole, render the claims patentably distinct over the prior art.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KIM THANH THI TRAN whose telephone number is (571)270-1408. The examiner can normally be reached Monday-Friday 8:00am-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ALICIA HARRINGTON can be reached at 5712722330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KIM THANH T TRAN/Examiner, Art Unit 2615
/JAMES A THOMPSON/Primary Examiner, Art Unit 2615
Prosecution Insights