Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 3, 12, 13 and 15 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by He et al. ("Sketch Recognition with Deep Visual-Sequential Fusion Model).
Regarding claim 1, He teaches a device for recognizing a sketch, the device comprising a memory configured to store at least one instruction and a processor configured to execute the at least one instruction stored in the memory (section 3.6),
wherein the processor generates a plurality of frames from sketch data about a sketch image created by a user (section 3.1, a sketch is treated as a sequence of stroke images),
extracts features from each of the plurality of frames (section 3.2 and Fig. 2, extracts deep features from each of the selected stroke images),
trains a deep learning model configured to classify the sketch image into a class, on the basis of the extracted features (abstract and section 3.4, Deep Visual-Sequential Fusion model), and
performs sketch recognition using the trained deep learning model (section 4, training).
Regarding claim 3, He generates the plurality of frames by detecting stroke images at time points determined in accordance with a preset criterion (section 3.1 and Fig. 3; section 4.2).
Regarding claim 12, see He, sections 3.4 and 4.
Regarding claim 13, see the rejection of claim 1.
Regarding claim 15, see the rejection of claim 3.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over He in view of the Google (Quick, Draw! Dataset, attached is a printout of the googlecreativelab/quickdraw-dataset, publicly available since 2017).
Regarding claim 2, He teaches sketch data comprising stroke data for each of the strokes constituting the sketch image (He, section 3.1, the drawing process is the accumulation of strokes).
Google teaches that the sketch data further includes point data about each of the points constituting the strokes, and that the point data includes position coordinates and generation times of the points (raw drawing format, each stroke is stored with pixel coordinates and time in milliseconds).
It would have been obvious prior to the effective filing date of the invention to one of ordinary skill in the art to store the sketch data of He in the stroke point and time format taught by Google in order to preserve the per-point coordinate and timing information needed to reconstruct the stroke-completeness.
Regarding claim 14, see the rejection of claim 2.
Claims 4 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over He in view of Kang (“Fluid3DGuides: A Technique for Structured 3D Drawing in VR”).
He teaches sampling the plurality of frames at uniform intervals over the drawing, the drawing being an inherently time-ordered accumulation of the user's strokes (He, section 3.1 and Fig. 3),
Kang teaches generating a plurality of frames by sampling at equal time intervals, calculated by dividing the total time by a preset value (see section 2, prototype).
It would have been obvious prior to the effective filing date of the invention to one of ordinary skill in the art to include in He the ability to use equal time intervals for sampling as taught by Kang. The reason is to uniformly sample.
Regarding claim 16, see the rejection of claim 4.
Claims 5-6 and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over He in view of Bhunia et al. ("Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrieval").
He teaches generating the plurality of frames by selecting stroke images at intervals (section 3.1 and Fig. 3; section 4.2),
Bhunia teaches determining the sampling interval by dividing the total count by a preset value (section 3.2, "we rasterize the sketch a total of T times, i.e., at steps of interval floor(N/T)").
It would have been obvious prior to the effective filing date of the invention to one of ordinary skill in the art to determine He's stroke-completeness sampling interval by calculating a value that divides the total number of strokes by a preset value, as taught by Bhunia. The reason is to uniformly sample the drawing.
Regarding claim 6, see Bhunia section 3 and 3.2, "a sketch is represented as S …."; section 3.2. we rasterize the sketch)
Regarding claim 17, see the rejection of claim 5.
Regarding claim 18, see the rejection of claim 7.
Claims 7, 8, 19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over He in view of Zheng ("Sketch-Specific Data Augmentation for Freehand Sketch Recognition").
Regarding claim 7, He teaches training the model on the stroke-image frames in the rejection of claim 1 but does not expressly teach augmenting the frames to generate additional training frames.
Zheng teaches generating new sketches by augmentation and training the recognition network on the augmented data together with the original data (Zheng, Abstract and section 1, a Bezier pivot based deformation (BPD) strategy enriches the training data by generating a substantial amount of new freehand sketches).
It would have been obvious prior to the effective filing date of the invention to one of ordinary skill in the art to augment the stroke image frames of He by the stroke deformation taught by Zheng in order to enlarge and diversify the training set and thereby improve recognition accuracy given the limited amount of sketch training data.
Regarding claims 8, Zheng teaches that the augmentation is performed by deforming the geometry of the strokes (Zheng, section 3.1, Bezier deformation of the strokes; see also section 2.2, sketch deformation).
Regarding claim 20, see the rejection of claim 8.
Claim 9-11 are rejected under 35 U.S.C. 103 as being unpatentable over He in view of Girdhar ("Anticipative Video Transformer") and further in view of Ng ("Beyond Short Snippets: Deep Networks for Video Classification").
Regarding claim 9, He teaches a deep learning model that processes the plurality of stroke-image frames and includes recurrent (R-LSTM) sequential networks and a fusion layer (He, sections 3.3-3.4), but does not expressly teach that the model includes a transformer model producing per-frame class possibility data and a separate ensemble model producing the sketch-image class possibility data from the per-frame class possibility data.
Girdhar teaches a transformer model that receives extracted per-frame features in sections 3-4, transformer decoder producing per-frame predicted features and per-frame class distributions.
It would have been obvious prior to the effective filing date of the invention to one of ordinary skill in the art to implement the frame model of He as a transformer producing per-frame class possibilities as taught by Girdhar. The reason is allowing the system to classify each frame using a transformer.
Ng teaches an ensemble of recurrent models that produces a class prediction for each frame and combines the per-frame predictions into a single prediction for the whole sequence (section 3.2, a softmax classifier makes a prediction at every frame and the frame level predictions are combined into a single video-level prediction).
It would have been obvious prior to the effective filing date of the invention to one of ordinary skill in the art to include in He and Girdhar the ability to combine those per-frame class possibilities with an ensemble recurrent models as taught by Ng. The reason is to learn the relationships among the frames and aggregate fir classification.
Regarding claim 10, Girdhar teaches that the transformer model receives the extracted frame features and performs a plurality of masked multi head self-attention processes across a plurality of stacked transformer decoder layers to learn relationships among the frames (Girdhar, section 3, transformer decoder with multiple layers of masked multihead attention.
Regarding claim 11, see Ng section 3.2 and Fig. 4.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: Harikumnar (20230419551), teaches generating sketches using a neural network.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HADI AKHAVANNIK whose telephone number is (571)272-8622. The examiner can normally be reached 9 AM - 5 PM Monday to Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HADI AKHAVANNIK/ Primary Examiner, Art Unit 2676
Prosecution Insights