Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-11 are presented for examination.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim(s) 6-11 is/are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
Regarding claim(s) 6-10, this/these claim(s) recites a “device” for training a model. There is no structural component associated with the device within the claim limitations or the disclosure, and as such the device can be directed to software per se. Therefore the claimed invention is directed to a judicial exception (i..e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more. Therefore claims 6-10 is directed to non-statutory subject matter.
Claim 11 is directed towards a “program”. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because a “program” is not directed to either a process, machine, manufacture or composition of matter.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 4-6, 9-11, is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Haidar (US 20220335303 A1).
Regarding claim 1, Haidar teaches an AI training method involving inputting training data to a training-target AI model including layers to generate a new trained AI model, the method comprising (Haidar [128-130] method performed using a device executing software stored in memory, to train an AI model with layers):
inputting the training data to an existing trained AI model to acquire a trained-layer feature value output from a trained layer in the existing trained AI model (Haidar [73, 92] teacher (existing trained AI model) and student models (training-target AI model) may be trained or untrained),
merging the trained-layer feature value with a training-target layer feature value output from a training-target layer in the training-target AI model to generate a merged feature value (Haidar [63, 73-76, 102-105] teacher and student layer feature outputs (intermediate representation(s) may be converted to vectors with layer-to-layer correspondence, layer attention-related information determined (intermediate representation loss, merged)), and
inputting the merged feature value to a training-target layer subsequent to the training-target layer to generate the new trained AI model (Haidar [85, 89, 112, 114, 115, 117, 121, 125, 126] intermediate representation loss and layer output(s) used in forward propagation (subsequent layer) to train student layers).
Regarding claim 4, Haidar teach(es) the invention as claimed in claim 1 above.
Haidar further teaches terminating generation of the merged feature value when a training progress index has reached a prescribed progress index threshold value (Haidar [85, 123] training epochs are terminated (and therefore generation of merged feature) based on termination condition (training progress index), which could be reaching predetermined number of training epochs, falling below a gradient threshold or satisfying some other convergence threshold (progress index threshold value)).
Regarding claim 5, Haidar teach(es) the invention as claimed in claim 1 above.
Haidar further teaches wherein the training-target AI model and the existing trained AI model have a layer structure with a same number of layers, each pair of layers corresponding to each other between the training-target AI model and the existing trained AI model having a same configuration)), and inputting the merged feature value to a training-target layer subsequent to the training-target layer to generate the new trained AI model (Haidar [102, 104] teacher and student models may have same layers with one-to-one correspondence, Haidar [63, 73-76, 102-105] teacher and student layer feature outputs (intermediate representation(s) may be converted to vectors with layer-to-layer correspondence, layer attention-related information determined (intermediate representation loss, merged, Haidar [85, 89, 112, 114, 115, 117, 121, 125, 126] intermediate representation loss and layer output(s) used in forward propagation (subsequent layer) to train student layers).
Claim 6 is directed towards a device executing instructions similar in scope to the instructions performed by the method of claim 1, and is rejected under the same rationale.
Haidar further teaches an AI training device in which training data is input to a training-target AI model including layers to generate a new trained AI model (Haidar [128-130] method performed using a device executing software stored in memory, to train an AI model with layers).
Claim(s) 9, 10 is/are dependent on claim 6 above, is/are directed towards a device executing instructions similar in scope to the instructions performed by the method of claim(s) 4, 5 respectively, and is/are rejected under the same rationale.
Claim 11 is directed towards a program executing instructions similar in scope to the instructions performed by the method of claim 1, and is rejected under the same rationale.
Haidar further teaches AI training program involving inputting training data to a training-target AI model including layers to generate a new trained AI model, the program making a computer perform a procedure comprising (Haidar [128-130] method performed using a device executing software stored in memory)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2, 7, is/are rejected under 35 U.S.C. 103 as being unpatentable over Haidar (US 20220335303 A1), in view of Lee (US 20220383072 A1).
Regarding claim 2 Haidar teach(es) the invention as claimed in claim 1 above.
Haidar does not specifically teach in generating the merged feature value, merging the trained-layer feature value with the training-target layer feature value using a cross-attention mechanism
However Lee teaches in generating the merged feature value, merging the trained-layer feature value with the training-target layer feature value using a cross-attention mechanism (Lee [9, 25-27, 93, 99] student and teacher encoder (encoder has layers) feature outputs are merged using cross attention mechanism and merged feature value may be used as one of decoder (decoder has layers) inputs to train student decoder, cross attention mechanism allows determining which part of the output of the first deep neural network model 202 to focus on and apply to the second deep neural network model).
It would have been obvious to one of an ordinary skill in the art before the effective filing date of the claimed invention, to have incorporated the concept taught by Lee of in generating the merged feature value, merging the trained-layer feature value with the training-target layer feature value using a cross-attention mechanism, into the invention suggested by Haidar; since both inventions are directed towards knowledge distillation for student model training, and incorporating the teaching of Lee into the invention suggested by Haidar would provide the added advantage of determining which part of the output of the first deep neural network model 202 to focus on and apply to the second deep neural network model, and the combination would perform with a reasonable expectation of success (Lee [9, 25-27, 93, 99]).
Claim(s) 7 is/are dependent on claim 6 above, is/are directed towards a program executing instructions similar in scope to the instructions performed by the method of claim(s) 2, and is/are rejected under the same rationale.
Claim(s) 3, 8, is/are rejected under 35 U.S.C. 103 as being unpatentable over Haidar (US 20220335303 A1) in view of Lee (US 20220383072 A1), and further in view of Zhou et al “Channel Distillation: Channel-Wise Attention for Knowledge Distillation” dated 2 Jun 2020 and retrieved from arXiv:2006.01683v1 .
Regarding claim 3 Haidar and Lee teach(es) the invention as claimed in claim 2 above.
Haidar does not specifically teach reducing a degree of merging as training progresses
However Zhou teaches reducing a degree of merging as training progresses (Zhou Secs. 3.1- 3.3, teacher and student information are merged for knowledge distillation for student training, to prevent inhibition student learning better in later stages, merging is gradually decreased).
It would have been obvious to one of an ordinary skill in the art before the effective filing date of the claimed invention, to have incorporated the concept taught by Zhou of reducing a degree of merging as training progresses, into the invention suggested by Haidar and Lee; since both inventions are directed towards merging teacher and student information for knowledge distillation for student training, and incorporating the teaching of Zhou into the invention suggested by Haidar and Lee would provide the added advantage of preventing inhibition student learning better in later stages, and the combination would perform with a reasonable expectation of success (Zhou Secs. 3.1- 3.3).
Claim(s) 8, is/are dependent on claim 6 above, is/are directed towards a program executing instructions similar in scope to the instructions performed by the method of claim(s) 3, and is/are rejected under the same rationale.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SANCHITA ROY whose telephone number is (571)272-5310. The examiner can normally be reached Monday-Friday 12-8.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed can be reached at (571) 272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
SANCHITA ROY
Primary Examiner
Art Unit 2146
/SANCHITA ROY/Primary Examiner, Art Unit 2146
