DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant's claim for foreign priority based on an application filed in Korea on December 1st, 2021. It is noted, however, that applicant has not filed a certified copy of the KR-10-2021-0170040 application as required by 37 CFR 1.55.
Receipt is acknowledged of certified copies of paper required by 37 CFR 1.55 (Korean Application KR-10-2022-0165012 filed November 30th, 2022).
Information Disclosure Statement
The information disclosure statements (IDS) submitted on April 12th, 2023; November 8th, 2023; and July 16th, 2026 were filed before the mailing date of the First Action on the Merits (this Office Action). The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the Examiner.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description: “S220” [Figure 2 – See Paragraph 131 where “S230” is listed instead] “410” and “41(N-1)” [Figure 21]. Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference character “300” has been used to designate both “memory” and “controller” [Figure 18].
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The drawings are objected to because:
In Figure 17, the numbering of the comparators and the associated reference characters do not match with the corresponding description in the Specification.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The abstract of the disclosure is objected to because the Abstract is less than 50 words (approximately 45 words) and is written in legalese instead of a series of brief sentences in narrative format describing the inventive concept. A corrected abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text. See MPEP § 608.01(b).
Applicant is reminded of the proper language and format for an abstract of the disclosure.
The abstract should be in narrative form and generally limited to a single paragraph on a separate sheet within the range of 50 to 150 words in length. The abstract should describe the disclosure sufficiently to assist readers in deciding whether there is a need for consulting the full patent text for details.
The language should be clear and concise and should not repeat information given in the title. It should avoid using phrases which can be implied, such as, “The disclosure concerns,” “The disclosure defined by this invention,” “The disclosure describes,” etc. In addition, the form and legal phraseology often used in patent claims, such as “means” and “said,” should be avoided.
The lengthy specification has not been checked to the extent necessary to determine the presence of all possible minor errors. Applicant’s cooperation is requested in correcting any errors of which applicant may become aware in the specification.
The disclosure is objected to because of the following informalities:
In Paragraph 168 line 2, the acronym “SELU” is not defined on first use.
In Paragraph 497 line 4, the phrase “A(N-1, B0” should read as --A(N-1), B0-- for clarity.
Appropriate correction is required.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claim 1 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 10 of U.S. Patent No. 11,836,604.
Although the claims at issue are not identical, they are not patentably distinct from each other because pending claim 1 is within the scope of Patented claim 10 as the programmed activation segments and controller used are obvious variants of those claimed and the CPU for the pending controller as readily recognized by one of ordinary skill in the art.
Claim1 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim13 of U.S. Patent No. 12,327,174.
Although the claims at issue are not identical, they are not patentably distinct from each other because pending claim 1 is within the scope of Patented claim 13 as the programmed activation segments and controller used are obvious variants of those claimed and the CPU for the pending controller as readily recognized by one of ordinary skill in the art and in further view of the Patented “generate segment data …” limitation, “segment the activation function ..” and “receiving, from the PAF circuit, …” limitations for those at least which render obvious pending claim 1 to one of ordinary skill in the art.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1 – 26 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term “preferentially ” in claim 20 is a relative term which renders the claim indefinite. The term “preferentially” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
Claim limitations:
“a neural processing unit …” [Claim 1];
“at least one processing element configured to …” [Claim 1];
“a programmed activation function execution unit configured to …” [Claim 1];
“a controller configured to …” [Claim 1]; and
“an activation function conversion program unit that programs …” [Claim 19];
have been evaluated under the three-prong test set forth in MPEP § 2181, subsection I, but the result is inconclusive. Thus, it is unclear whether this limitation should be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because there is no clear corresponding structure in view of at least Specification Paragraph 33 as software embodiment are contemplated as well (e.g. firmware). The boundaries of this claim limitation are ambiguous; therefore, the claim is indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
In response to this rejection, applicant must clarify whether this limitation should be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. Mere assertion regarding applicant’s intent to invoke or not invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph is insufficient. Applicant may:
(a) Amend the claim to clearly invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, by reciting “means” or a generic placeholder for means, or by reciting “step.” The “means,” generic placeholder, or “step” must be modified by functional language, and must not be modified by sufficient structure, material, or acts for performing the claimed function;
(b) Present a sufficient showing that 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, should apply because the claim limitation recites a function to be performed and does not recite sufficient structure, material, or acts to perform that function;
(c) Amend the claim to clearly avoid invoking 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, by deleting the function or by reciting sufficient structure, material or acts to perform the recited function; or
(d) Present a sufficient showing that 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, does not apply because the limitation does not recite a function or does recite a function along with sufficient structure, material or acts to perform that function.
Regarding claims 2 – 26, the dependent claims do not cure the deficiencies of independent claim 1 and thus are similarly Rejected.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 – 9, 17 – 20, and 24 are rejected under 35 U.S.C. 103 as being unpatentable over Lin, et al. (US PG PUB 2020/0401873 A1 referred to as “Lin” throughout), and further in view of Sather, et al. (US Patent #12,136,039 B1 referred to as “Sather” throughout).
Regarding claim 1, Lin teaches segmenting activation functions for neural networks and implementation details of the segments. Sather teaches architectures to expand on the teachings of Lin for neural networks with processing elements and array based implementations of Lin’s linearized segments.
It would have been obvious to one of ordinary skill art before the effective filing date of the claimed invention to modify the implementation of Lin to include Sather’s architectures implementing the segmented activation functions taught by Lin. The combination teaches
at least one processing element configured to output an operation value by artificial neural network operation [Lin Figures 1 – 2 (see at least reference character 150) as well as Paragraphs 18 – 22 and 29 – 32 (accumulators for generating output from the activation functions / segments used in the layers) where Sather Figures 19 – 24 (see at least reference characters 2010, 2030, 2015, and 2420), 27, 29 – 31 (activation elements with weights), 34 (array of processing elements) as well as Column 43 line 43 – Column 44 line 59 (core / registers to store weights for product determinations) and Column 45 line 52 – Column 46 line 36];
a programmed activation function execution unit configured to generate an activation value by applying at least one programmed activation function [Lin Figures 1 – 2 (see at least reference character 150) as well as Paragraphs 18 – 24 (linear segmentation of activation functions), 27 – 32 (linear segmentation of activation functions and execution units for computing outputs of activation functions) where Sather Figures 19 – 24 (see at least reference characters 2010, 2030, 2015, and 2420), 27, 29 – 31 (activation elements with weights), 34 (array of processing elements) as well as Column 43 line 31 – Column 44 line 59 (core / registers to store weights for product determinations) and Column 45 line 52 – Column 46 line 36] including a plurality of programmable segments to the operation value [Lin Figures 2 – 5 (see at least reference characters 130 and 150) as well as Paragraphs 18 – 24 (linear segmentation of activation functions), 27 – 35 (linear segmentation of activation functions (see Paragraph 35 at least) and execution units for computing outputs of activation functions)]; and
a controller configured to control operation of the at least one processing element and the programmed activation function execution unit [Lin Figures 2 – 5 (see at least reference characters 100, 110, 130, and 150) as well as Paragraphs 27 – 30, and 35 – 38 (circuit implementations of the segmentation and activation function where processor implementations are taught in Paragraphs 17 – 18 as well)].
The motivation to combine Sather with Lin is to combine features in the same / similar field of invention of machine learning / neural network implementation [Sather Column 1 lines 21 – 37] in order to improve operations of neural networks by reducing hardware complexity / computation time [Sather Column 1 lines 21 – 27 and Column 3 lines 46 – 65 where the Examiner observes at least KSR Rationales (D) or (F) are also applicable].
This is the motivation to combine Lin and Sather which will be used throughout the Rejection.
Regarding claim 2, Lin teaches segmenting activation functions for neural networks and implementation details of the segments. Sather teaches architectures to expand on the teachings of Lin for neural networks with processing elements and array based implementations of Lin’s linearized segments.
It would have been obvious to one of ordinary skill art before the effective filing date of the claimed invention to modify the implementation of Lin to include Sather’s architectures implementing the segmented activation functions taught by Lin. The combination teaches
a segment register for storing information about sections of the plurality of programmable segments [Lin Figures 2 – 5 (see at least reference characters 110 and 130) as well as Paragraphs 16 – 19 (registers to store data for the parameters / segments), 20 – 23 and 25 – 31 (linearization of activation functions such as those in Paragraphs 33 – 35 (e.g. sigmoid functions)); Sather Figures 19 – 25 (see at least reference characters 2010, 2030, 2015, and 2420), 27, 29 – 31 (activation elements with weights), 34 (array of processing elements) as well as Column 43 line 31 – Column 44 line 59 (core / registers to store weights for product determinations with registers to store data), Column 45 line 52 – Column 46 line 36, and Column 52 line 37 – Column 53 line 57 (register control for activation function processing using weights / offsets (linear functions))].
See claim 1 for the motivation to combine Lin and Sather.
Regarding claim 3, Lin teaches segmenting activation functions for neural networks and implementation details of the segments. Sather teaches architectures to expand on the teachings of Lin for neural networks with processing elements and array based implementations of Lin’s linearized segments.
It would have been obvious to one of ordinary skill art before the effective filing date of the claimed invention to modify the implementation of Lin to include Sather’s architectures implementing the segmented activation functions taught by Lin. The combination teaches
a segment register for storing a segment boundary value of the plurality of programmable segments [Lin Figures 2 – 5 (see at least reference characters 110 and 130) as well as Paragraphs 16 – 19 (registers to store data for the parameters / segments), 21 – 27 (linearization or piecemeal implementation as in Equations 1 and 4 with boundary values (e.g. the domains of x in the piecemeal functions) of activation functions such as those in Paragraphs 33 – 35 (e.g. sigmoid functions));].
See claim 1 for the motivation to combine Lin and Sather.
Regarding claim 4, Lin teaches segmenting activation functions for neural networks and implementation details of the segments. Sather teaches architectures to expand on the teachings of Lin for neural networks with processing elements and array based implementations of Lin’s linearized segments.
It would have been obvious to one of ordinary skill art before the effective filing date of the claimed invention to modify the implementation of Lin to include Sather’s architectures implementing the segmented activation functions taught by Lin. The combination teaches
the programmed activation function execution unit comprises a plurality of comparators, a selector, at least one multiplier and at least one adder, which are hard-wired [Lin Figures 1 – 5 (see at least reference characters 100, 110, 130, and 150) as well as Paragraphs 16 – 19 and 36 – 38 (processor / hard-wired (obvious variant to one of ordinary skill in the art) / circuit implementations) and 22 – 25 (comparators used and implementations of hardware / circuitry with multipliers and adders described in Paragraph 18 as well); Sather Figures 20 – 27 (see the selection of the scale / shift factors and weights selected by the input and math functions at least in reference characters 2415, 2225, 2325) as well as Column 48 line 45 – Column 49 line 42 and Column 50 lines 14 – 62 (hardwired multiplier and adder implementations for the look up table / segments)].
See claim 1 for the motivation to combine Lin and Sather.
Regarding claim 5, Lin teaches segmenting activation functions for neural networks and implementation details of the segments. Sather teaches architectures to expand on the teachings of Lin for neural networks with processing elements and array based implementations of Lin’s linearized segments.
It would have been obvious to one of ordinary skill art before the effective filing date of the claimed invention to modify the implementation of Lin to include Sather’s architectures implementing the segmented activation functions taught by Lin. The combination teaches
a plurality of comparators configured to compare the operation value with each of a plurality of input segment boundary values and output a section determination data [Lin Figures 1 – 5 (see at least reference characters 100, 110, 130, and 150) as well as Paragraphs 16 – 19 and 36 – 38 (processor / hard-wired (obvious variant to one of ordinary skill in the art) / circuit implementations) and 22 – 25 (comparators used and implementations of hardware / circuitry with multipliers and adders described in Paragraph 18 as well); Sather Figures 20 – 27 (see the selection of the scale / shift factors and weights selected by the input and math functions at least in reference characters 2415, 2225, 2325 where the weight selector / weight values are obvious variants of the claimed operation value) as well as Column 48 line 45 – Column 49 line 42 and Column 50 lines 14 – 62 (hardwired multiplier and adder implementations for the look up table / segments)].
See claim 1 for the motivation to combine Lin and Sather.
Regarding claim 6, Lin teaches segmenting activation functions for neural networks and implementation details of the segments. Sather teaches architectures to expand on the teachings of Lin for neural networks with processing elements and array based implementations of Lin’s linearized segments.
It would have been obvious to one of ordinary skill art before the effective filing date of the claimed invention to modify the implementation of Lin to include Sather’s architectures implementing the segmented activation functions taught by Lin. The combination teaches
a plurality of comparators configured to determine whether to operate by a comparator enable signal [Lin Figures 1 – 5 (see at least reference characters 100, 110, 130, and 150) as well as Paragraphs 16 – 19 and 36 – 38 (processor / hard-wired (obvious variant to one of ordinary skill in the art) / circuit implementations) and 22 – 25 (comparators used and implementations of hardware / circuitry); Sather Figures 20 – 27 (see the selection of the scale / shift factors and weights selected by the input and math functions at least the AND gates for enabling and the selection lines for enabling and see at least reference character 2420 (enable signal used for selection in Figures 23 – 24) or the enable inputs to reference characters 2305 and 2310 in Figures 22 – 24) as well as Column 50 line 47 – Column 51 line 42 (enabling comparators / adders / multipliers) and Column 52 lines 4 – 56 (enabling nodes to compute portions / pieces of activation functions)].
See claim 1 for the motivation to combine Lin and Sather.
Regarding claim 7, Lin teaches segmenting activation functions for neural networks and implementation details of the segments. Sather teaches architectures to expand on the teachings of Lin for neural networks with processing elements and array based implementations of Lin’s linearized segments.
It would have been obvious to one of ordinary skill art before the effective filing date of the claimed invention to modify the implementation of Lin to include Sather’s architectures implementing the segmented activation functions taught by Lin. The combination teaches
a plurality of comparators configured to output a section determination data [See next limitation for citations of Lin and Sather], wherein the programmed activation function execution unit is configured to generate the activation value by applying a gradient and an offset of a corresponding segment among the plurality of programmable segments to the operation value according to the section determination data [Lin Figures 1 – 4 (see at least reference characters 100, 110, 130, and 150) as well as Paragraphs 16 – 19 and 36 – 38 (processor / hard-wired / circuit implementations of comparators, multipliers, and adders) and 20 – 27 (see at least Equations 1 and 4 in which the piece meal equations are implemented in which a gradient / slope and offset / bias to applied to generate output for the segment activated); Sather Figures 20 – 27 (see the selection of the scale / shift factors and weights selected by the input and math functions at least the AND gates for enabling and the selection lines for enabling where the weights / slopes and bias / offsets are ) as well as Column 50 line 47 – Column 51 line 42 (enabling comparators / adders / multipliers where the ) and Column 52 lines 4 – 56 (enabling activation functions to compute portions / pieces of activation functions)].
See claim 1 for the motivation to combine Lin and Sather.
Regarding claim 8, Lin teaches segmenting activation functions for neural networks and implementation details of the segments. Sather teaches architectures to expand on the teachings of Lin for neural networks with processing elements and array based implementations of Lin’s linearized segments.
It would have been obvious to one of ordinary skill art before the effective filing date of the claimed invention to modify the implementation of Lin to include Sather’s architectures implementing the segmented activation functions taught by Lin. The combination teaches
wherein the at least one multiplier multiplies an input value and a gradient of an outputted programmable segment from the selector [Lin Figures 1 – 5 (see at least reference characters 100, 110, 130, and 150) as well as Paragraphs 16 – 19 and 36 – 38 (processor / hard-wired (obvious variant to one of ordinary skill in the art) / circuit implementations) and 20 – 27 (see at least equations 1 and 4 in which the piece meal functions (segments) are implemented by multipliers and adders described in Paragraph 18 as well); Sather Figures 20 – 27 (see the selection of the scale / shift factors and weights selected by the input and math functions at least in reference characters 2415, 2225, 2325) as well as Column 48 line 45 – Column 49 line 42 and Column 50 lines 14 – 62 (hardwired multiplier and adder implementations for the look up table / segments)].
See claim 1 for the motivation to combine Lin and Sather.
Regarding claim 9, Lin teaches segmenting activation functions for neural networks and implementation details of the segments. Sather teaches architectures to expand on the teachings of Lin for neural networks with processing elements and array based implementations of Lin’s linearized segments.
It would have been obvious to one of ordinary skill art before the effective filing date of the claimed invention to modify the implementation of Lin to include Sather’s architectures implementing the segmented activation functions taught by Lin. The combination teaches
wherein the at least one adder adds a value output from the at least one multiplier obtained by multiplying the input value and the gradient for the programmable segment, to an offset for the programmable segment [Lin Figures 1 – 5 (see at least reference characters 100, 110, 130, and 150) as well as Paragraphs 16 – 19 and 36 – 38 (processor / hard-wired (obvious variant to one of ordinary skill in the art) / circuit implementations for the adders and multipliers) and 20 – 27 (see at least equations 1 and 4 in which the piece meal functions (segments) are implemented by multipliers and adders described in Paragraph 18 as well); Sather Figures 20 – 27 (see the selection of the scale / shift factors and weights selected by the input and math functions at least in reference characters 2415, 2225, 2325, 2720 (adder tree to combine / add weighted inputs) in which offset / bias terms are added at the end of the operations) as well as Column 48 line 45 – Column 49 line 42 and Column 50 lines 14 – 62 (hardwired multiplier and adder implementations for the look up table / segments)].
See claim 1 for the motivation to combine Lin and Sather.
Regarding claim 17, Lin teaches segmenting activation functions for neural networks and implementation details of the segments. Sather teaches architectures to expand on the teachings of Lin for neural networks with processing elements and array based implementations of Lin’s linearized segments.
It would have been obvious to one of ordinary skill art before the effective filing date of the claimed invention to modify the implementation of Lin to include Sather’s architectures implementing the segmented activation functions taught by Lin. The combination teaches
a programmable activation function library that stores gradient and offset information for a plurality of programmable segments configuring a programmable activation function [Lin Figures 1 – 5 (see at least reference characters 100, 110, 130, and 150) as well as Paragraphs 16 – 19 and 36 – 38 (processor / hard-wired (obvious variant to one of ordinary skill in the art) / circuit implementations for the adders and multipliers) and 20 – 27 (see at least equations 1 and 4 in which the piece meal functions (segments) are implemented by multipliers and adders described in Paragraph 18 and are implemented in a library / look-up table (obvious variant of the library claimed to one of ordinary skill in the art) in Paragraphs 16, 24 – 25, and 31 and combinable with Sather Column 73 line 54 – Column 74 line 28 (programmable hardware rendering obvious programmable functions and segments in view of Figures 3 – 4 as well as Column 14 line 49 – Column 15 line 19))].
See claim 1 for the motivation to combine Lin and Sather.
Regarding claim 18, Lin teaches segmenting activation functions for neural networks and implementation details of the segments. Sather teaches architectures to expand on the teachings of Lin for neural networks with processing elements and array based implementations of Lin’s linearized segments.
It would have been obvious to one of ordinary skill art before the effective filing date of the claimed invention to modify the implementation of Lin to include Sather’s architectures implementing the segmented activation functions taught by Lin. The combination teaches
wherein the at least one processing element is connected to the programmed activation function execution unit through a multiplexer [Lin Figures 1 – 5 (see at least reference characters 100, 110, 130, and 150) as well as Paragraphs 16 – 19 and 36 – 38 (processor / hard-wired (obvious variant to one of ordinary skill in the art) / circuit implementations) and 20 – 27 (see at least equations 1 and 4 in which the piece meal functions (segments) are implemented by multipliers and adders with selection in Paragraph 26 to combine with Sather); Sather Figures 20 – 27 (see the selection of the scale / shift factors and weights selected by the input and math functions at least in reference characters 2210, 2215 (multiplexer to select inputs), 2415, 2225, 2325, 2720 (adder tree to combine / add weighted inputs) in which offset / bias terms are added at the end of the operations) as well as Column 48 line 45 – Column 49 line 42 and Column 50 lines 14 – 62 (hardwired multiplexers to select values for the activation function computation)].
See claim 1 for the motivation to combine Lin and Sather.
Regarding claim 19, Lin teaches segmenting activation functions for neural networks and implementation details of the segments. Sather teaches architectures to expand on the teachings of Lin for neural networks with processing elements and array based implementations of Lin’s linearized segments.
It would have been obvious to one of ordinary skill art before the effective filing date of the claimed invention to modify the implementation of Lin to include Sather’s architectures implementing the segmented activation functions taught by Lin. The combination teaches
an activation function conversion program unit that programs an activation function into the at least one programmed activation function [Lin Figures 1 – 5 (see at least reference characters 100, 110, 130, and 150) as well as Paragraphs 16 – 19 and 36 – 38 (processor / hard-wired (obvious variant to one of ordinary skill in the art) implementations for the activation function) and 20 – 27 (see at least equations 1 and 4 in which the piece meal functions (segments) are programmed as described in at least Paragraph 18 and are implemented in a library / look-up table (obvious variant of the programmed claimed to one of ordinary skill in the art) in Paragraphs 16, 24 – 25, and 31 and combinable with Sather Column 73 line 54 – Column 74 line 28 (programmable hardware rendering obvious programmable functions and segments in view of Figures 3 – 4 as well as Column 14 line 49 – Column 15 line 19))].
See claim 1 for the motivation to combine Lin and Sather.
Regarding claim 20, Lin teaches segmenting activation functions for neural networks and implementation details of the segments. Sather teaches architectures to expand on the teachings of Lin for neural networks with processing elements and array based implementations of Lin’s linearized segments.
It would have been obvious to one of ordinary skill art before the effective filing date of the claimed invention to modify the implementation of Lin to include Sather’s architectures implementing the segmented activation functions taught by Lin. The combination teaches
the activation function conversion program unit preferentially determines a linear section and a non-linear section of the at least one programmed activation function according to a slope change data [Lin Figures 1 – 5 (see at least reference characters 100, 110, 130, and 150) as well as Paragraphs 16 – 19 and 36 – 38 (processor / hard-wired (obvious variant to one of ordinary skill in the art) / circuit implementations) and 20 – 27 (see at least equations 1 and 4 in which the piece meal functions (segments) are done to approximate non-linear activation functions such as in Paragraph 20 and sections in Paragraphs 21 – 27 determined described in Paragraph 18 as well); Sather Figures 20 – 27 (see the look up tables at least in reference character 2240 to implement linear sections of non-linear functions) as well as Column 48 line 45 – Column 49 line 42 and Column 50 lines 14 – 62 (non linear activation functions implemented in linear section)].
See claim 1 for the motivation to combine Lin and Sather.
Regarding claim 24, Lin teaches segmenting activation functions for neural networks and implementation details of the segments. Sather teaches architectures to expand on the teachings of Lin for neural networks with processing elements and array based implementations of Lin’s linearized segments.
It would have been obvious to one of ordinary skill art before the effective filing date of the claimed invention to modify the implementation of Lin to include Sather’s architectures implementing the segmented activation functions taught by Lin. The combination teaches
the activation function conversion program unit converts the linear section of the at least one programmed activation function into a programmable segment approximated by a linear function [Lin Figures 1 – 5 (see at least reference characters 100, 110, 130, and 150) as well as Paragraphs 16 – 19 and 36 – 38 (processor / hard-wired (obvious variant to one of ordinary skill in the art) / circuit implementations) and 20 – 27 (see at least equations 1 and 4 in which the piece meal functions (segments) are done to approximate non-linear activation functions such as in Paragraph 20 and sections in Paragraphs 21 – 27 determined described in Paragraph 18 as well); Sather Figures 20 – 27 (see the look up tables at least in reference character 2240 to implement linear sections of non-linear functions) as well as Column 48 line 45 – Column 49 line 42 and Column 50 lines 14 – 62 (non linear activation functions implemented in linear section)].
See claim 1 for the motivation to combine Lin and Sather.
Allowable Subject Matter
Claims 10 – 16, 21 – 23, and 25 – 26 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, and ODP Rejections set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim 10, the claim recites using second order functions in the determination of the segments / linear approximations in which the prior art does not fairly teach such consideration in the context of activation functions for neural networks.
Regarding claim 15, the prior art in the implementation of the activation functions does not fairly teach compressing outputs of activation functions with logarithmic functions after the addition step.
Regarding claim 21, the claim is allowable similarly to claim 10 in the use of second derivatives for error analysis in determining section breaks for the segments during the linearization process.
Regarding claim 22, the claim is allowable similarly to claim 10 in the use of second derivatives for error analysis in determining section breaks for the segments during the linearization process.
Regarding claim 25, the prior art does not fairly teach segments with quadratic / higher order polynomial approximations for the segmentation of the activation function in the neural network context.
Regarding claim 26, the prior art does not fairly teach segments with logarithmic approximations for the segmentation of the activation function in the neural network context.
Regarding claims 11 – 14, the claims depend on claim 10 and thus have similar reasons for Allowance.
Regarding claim 16, the claim depends on claim 15 and thus has similar reasons for Allowance.
Regarding claim 23, the claim depends on claim 22 and thus has similar reasons for Allowance.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Stevens (US PG PUB 2021/0295134 A1 referred to as “Stevens” throughout) teaches comparators and selection of values for multiplier / adder activation node circuits similar to Sather.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Tyler W Sullivan whose telephone number is (571)270-5684. The examiner can normally be reached IFP.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Czekaj can be reached at (571)-272-7327. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TYLER W. SULLIVAN/Primary Examiner, Art Unit 2487
