Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-8, 10-21 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 1, the limitation “channel attention mechanism” renders the claim indefinite because it is unclear and confusing what such mechanism it refers to. The claimed “channel attention mechanism” does not appear to be a well-known technological term with a standard definition, and the applicant’s disclosure does not provide any explicit definition either. In the field of image analysis, the limitation “channel” can possibly cover a broad spectrum of elements, such as: red/green/blue channels, luminance/chrominance channels, hue/saturation/intensity channels, frequency channels, etc., and consequently, it would have been unclear to one of ordinary skill in the art what the claimed “channel” of the “channel attention mechanism” refers to. Please amend the claim for clarification.
Independent claims 10, 11 recite a similar element “channel attention mechanism” and are thus similarly rejected. Dependent claims 2-8, 12-21 are similarly rejected.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-2, 4 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Liu (Liu, Wei, et al. "Xmp-font: Self-supervised cross-modality pre-training for few-shot font generation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.).
Regarding claim 1, Liu discloses a character processing method ([3.1 Overall pipeline] an encoder inputs a glyph image and determines style and content of the characters), comprising:
acquiring a first image comprising a to-be-processed character ([3.1 Overall pipeline] the encoder takes two modalities as input: the glyph image of a specific style and a sequence of stroke labels representing the corresponding character structure of the glyph);
training a target stroke order determination model by combining with a spatial attention mechanism and a channel attention mechanism (Fig. 2, [2.3 Cross-modality pre-training] in cross-modality pre-training, we build a transformer model consists of three encoders: a glyph processing module, a stroke processing encoder, and a cross-modality module…to endow our model with the capability of connecting a glyph image and its related stroke labels; see also [3.3 Pre-training strategy]); and
inputting the first image into the target stroke order determination model trained in advance to obtain a target stroke order corresponding to the to-be-processed character (Fig. 2, Fig. 3, [3.4. Downstream task of few-shot font generation] we pre-train an LSTM-based stroke predictor, which is able to predict the stroke labels sequentially in the correct order given a glyph image as input, then we use the predictor to compute the stroke loss).
Regarding claim 2, Liu discloses the method according to claim 1 as applied above. Liu further discloses acquiring at least one first training sample; wherein the first training sample comprises a sample character image and a theoretical character stroke order to which the sample character image corresponds ([3.1 Overall pipeline] as shown in Fig. 2m the encoder takes two modalities as input: the glyph image of specific style and a sequence of stroke labels representing the corresponding character structure of the glyph);
inputting, for the at least one first training sample, the sample character image of a current first training sample into a to-be-trained stroke order determination model to obtain a predicted stroke order (Fig. 2, Fig. 3, [3.4. Downstream task of few-shot font generation] we pre-train an LSTM-based stroke predictor, which is able to predict the stroke labels sequentially in the correct order given a glyph image as input, then we use the predictor to compute the stroke loss);
determining a loss value based on the predicted stroke order and the theoretical character stroke order in the current first training sample, and correcting a model parameter for the to-be-trained stroke order determination model based on the loss value (Fig. 3, [3.4. Downstream task of few-shot font generation] we pre-train an LSTM-based stroke predictor, which is able to predict the stroke labels sequentially in the correct order given a glyph image as input, then we use the predictor to compute the stroke loss); and
obtaining the target stroke order determination model by taking a convergence of a loss function in the to-be-trained stroke order determination model as a training target (Fig. 3, [3.4. Downstream task of few-shot font generation] compute the feature differences between the generated and ground-truth glyph then add the losses with equal weights as in Equation 2).
Regarding claim 4, Liu discloses the method according to claim 1 as applied above. Liu further discloses obtaining a target style feature fusion model by training a to-be-trained style feature fusion model, with the target stroke order determination model used as a loss model of the to-be-trained style feature fusion model; wherein the target style feature fusion model is configured to fuse at least two font styles ([3.4 Downstream task of few-shot font generation] one the encoder is pre-trained, we freeze the parameters and use it for the font generation task…the font generation model is trained with three losses; [3.1 Overall pipeline] we follow the style-content disentanglement paradigm and synthesize novel fonts by combining the style features of the reference glyphs and the content embeddings of the source glyphs).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 10-12, 14, 19, 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Liu (Liu, Wei, et al. "Xmp-font: Self-supervised cross-modality pre-training for few-shot font generation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.).
Regarding claim 10, Liu discloses a character processing method ([3.1 Overall pipeline] an encoder inputs a glyph image and determines style and content of the characters), comprising:
acquiring a first image comprising a to-be-processed character ([3.1 Overall pipeline] the encoder takes two modalities as input: the glyph image of a specific style and a sequence of stroke labels representing the corresponding character structure of the glyph);
training a target stroke order determination model by combining with a spatial attention mechanism and a channel attention mechanism (Fig. 2, [2.3 Cross-modality pre-training] in cross-modality pre-training, we build a transformer model consists of three encoders: a glyph processing module, a stroke processing encoder, and a cross-modality module…to endow our model with the capability of connecting a glyph image and its related stroke labels; see also [3.3 Pre-training strategy]); and
inputting the first image into the target stroke order determination model trained in advance to obtain a target stroke order corresponding to the to-be-processed character (Fig. 2, Fig. 3, [3.4. Downstream task of few-shot font generation] we pre-train an LSTM-based stroke predictor, which is able to predict the stroke labels sequentially in the correct order given a glyph image as input, then we use the predictor to compute the stroke loss).
Liu fails to disclose an electronic device, comprising: at least one processor; and a storage device configured to store at least one program, wherein the at least one program, when executed by the at least one processor, is configured to cause the at least one processor to implement a method.
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to combine Liu with generic computer components including an electronic device, comprising: at least one processor; and a storage device configured to store at least one program, wherein the at least one program, when executed by the at least one processor, is configured to cause the at least one processor to implement a method for the purpose of effectively implementing the character processing method of Liu in an application setting such as generating font libraries with a reduced labor requirement (See Liu: [Abstract]).
Regarding claim 11, Liu discloses a character processing method ([3.1 Overall pipeline] an encoder inputs a glyph image and determines style and content of the characters), comprising:
acquiring a first image comprising a to-be-processed character ([3.1 Overall pipeline] the encoder takes two modalities as input: the glyph image of a specific style and a sequence of stroke labels representing the corresponding character structure of the glyph);
training a target stroke order determination model by combining with a spatial attention mechanism and a channel attention mechanism (Fig. 2, [2.3 Cross-modality pre-training] in cross-modality pre-training, we build a transformer model consists of three encoders: a glyph processing module, a stroke processing encoder, and a cross-modality module…to endow our model with the capability of connecting a glyph image and its related stroke labels; see also [3.3 Pre-training strategy]); and
inputting the first image into the target stroke order determination model trained in advance to obtain a target stroke order corresponding to the to-be-processed character (Fig. 2, Fig. 3, [3.4. Downstream task of few-shot font generation] we pre-train an LSTM-based stroke predictor, which is able to predict the stroke labels sequentially in the correct order given a glyph image as input, then we use the predictor to compute the stroke loss).
Liu fails to disclose a non-transient storage medium comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a computer processor, are configured to implement a method.
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to combine Liu with generic computer components including a non-transient storage medium comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a computer processor, are configured to implement a method for the purpose of effectively implementing the character processing method of Liu in an application setting such as generating font libraries with a reduced labor requirement (See Liu: [Abstract]).
Allowable Subject Matter
Claims 3, 5-8, 13, 15-18, 20 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim 3, Liu discloses the method according to claim 2 as applied above. Liu further discloses wherein the inputting the sample character image of a current first training sample into a to-be-trained stroke order determination model to obtain a predicted stroke order, comprises: inputting the sample character image into a convolutional layer to obtain a first to-be-processed feature ([3.2 Cross-modality encoder, Input embedding module] the input glyph image of size 256×256×3 is mapped to a feature map of size 8×8×512 with a 5-layer convolutional encoder).
Liu fails to disclose performing a feature extraction on the first to-be-processed feature by the channel attention mechanism and the spatial attention mechanism to obtain a second to-be-processed feature; inputting the second to-be-processed feature into a recurrent neural network unit to obtain a feature sequence corresponding to a position and an order of each stroke; and processing the feature sequence based on a classifier to obtain the predicted stroke order.
Similar reasoning applies to claims 13, 20 which are directed to similar subject matter to claim 3.
Regarding claim 5, Liu discloses the method according to claim 4 as applied above. Liu fails to disclose wherein the obtaining a target style feature fusion model by training a to-be-trained style feature fusion model comprises:
determining at least one second training sample; wherein the second training sample comprises a to-be-trained character image and a reference character image;
inputting, for the at least one second training sample, the to-be-trained character image and the reference character image of a current second training sample into the to-be-trained style feature fusion model to obtain an actually output character image corresponding to the to-be-trained character image;
performing stroke loss processing on the actually output character image and the to-be-trained character image based on the target stroke order determination model to obtain a first loss value;
determining, for the actually output character image and the to-be-trained character image, a reconstruction loss based on a reconstruction loss function;
determining, for the actually output character image and a fused character image, a style loss value based on a style encoding loss function; wherein the fused character image is determined based on a font style of the to-be-trained character image and a font style of the reference character image;
correcting a model parameter in the to-be-trained style feature fusion model based on the first loss value, the reconstruction loss, and the style loss value; and
obtaining the target style feature fusion model by training the to-be-trained style feature fusion model, with a convergence of a loss function in the to-be-trained style feature fusion model used as a training target.
Similar reasoning applies to claim 6 which is dependent on claim 5. Similar reasoning applies to claims 15-16 which are directed to similar subject matter as claims 5-6.
Regarding claim 7, Liu discloses the method according to claim 4 as applied above. Liu further discloses receiving a target reference style character image and a target style conversion character image ([3.1 Overall pipeline] combining the style features of the reference glyphs and the content embeddings of the source glyphs).
Liu fails to disclose outputting at least one display character image based on a character content and a conversion character style of the target style conversion character image, as well as a reference character style of the target reference style character image to determine a target display character image based on a triggering operation.
Similar reasoning applies to claim 8 which is dependent on claim 5. Similar reasoning applies to claims 17-18 which are directed to similar subject matter as claims 7-8.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Wang (CN 113627260 A) discloses a method for recognizing the stroke order of handwritten characters, particularly Chinese characters, including feature extraction and obtaining loss functions.
Song (CN 111899292 A) discloses a character recognition method including an attention mechanism which includes a spatial attention mechanism and a channel attention mechanism.
Liu, Peng (US 20090041354 A1) discloses a method of generating handwritten characters using a trained model based on a received character, including mimicking real stroke order.
Liu, Xin (X. Liu, B. Hu, Q. Chen, X. Wu and J. You, "Stroke Sequence-Dependent Deep Convolutional Neural Network for Online Handwritten Chinese Character Recognition," in IEEE Transactions on Neural Networks and Learning Systems, vol. 31, no. 11, pp. 4637-4648, Nov. 2020, doi: 10.1109/TNNLS.2019.2956965.) discloses online recognition of handwritten Chinese characters using CNNs and generating a sequence-dependent representation to be combined with extracted features.
Shi (C. -Z. Shi, S. Gao, M. -T. Liu, C. -Z. Qi, C. -H. Wang and B. -H. Xiao, "Stroke Detector and Structure Based Models for Character Recognition: A Comparative Study," in IEEE Transactions on Image Processing, vol. 24, no. 12, pp. 4952-4964, Dec. 2015, doi: 10.1109/TIP.2015.2473105.) discloses character recognition using a stroke-detection based method including training a model to detect sub-strokes of specific characters.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CAROLINE DEPALMA whose telephone number is (571)270-0769. The examiner can normally be reached Mon-Thurs 9:00am-4pm Eastern Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Moyer can be reached at 571-272-9523. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CAROLINE E. DEPALMA/Examiner, Art Unit 2675
/SJ Park/Primary Examiner, Art Unit 2675
Prosecution Insights