Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The application is a continuation of Application No. 18/438,959, filed February 12, 2024 (now U.S. Patent No. 12,346,277), itself a continuation of Application No. 17/559,233, filed December 22, 2021 (now U.S. Patent No. 11,934,327). To the extent priority is proper, the earliest possible effective filing date for the presently claimed subject matter is December 22, 2021. This application has been examined. Claim 1 has been amended; claims 2-19 have been added. Claims 1-19 are currently pending.
The Group and/or Art Unit location of your application in the PTO has changed. To aid in correlating any papers for this application, all further correspondence regarding this application should be directed to Group Art Unit 2175.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Double Patenting
4. The non-statutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A non-statutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on non-statutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP §§ 706.02(l)(1) - 706.02(l)(3) for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
5. Claims 1-19 are provisionally rejected under the judicially-created doctrine of obviousness-type double patenting (ODP) as being unpatentable over claim 1 of each of:
U.S. Patent No. 11,934,327 B2 (Xi et al., “Systems and Methods for Hardware Acceleration of Data Masking Using a Field Programmable Gate Array,” issued Mar. 19, 2024) - the grandparent patent (App. 17/559,233); and
U.S. Patent No. 12,346,277 B2 (Xi et al., same title, issued Jul. 1, 2025) - the immediate parent patent (App. 18/438,959), of which the instant application is a direct continuation.
Claim 1 of the ’277 patent (the immediate parent) recites: “An electronic circuit comprising a data masking circuit to: receive input data including a two-dimensional array of data values at a plurality of indices of the two-dimensional array of the input data; select between a data value of the data values and an alternative value using a masking multiplexer to generate masked data, the masking multiplexer being controlled by a mask value of a plurality of mask values at indices corresponding to the indices of the input data; and output the masked data.” Instant claim 1 recites the identical “electronic circuit comprising a data masking circuit” performing the identical receive/select/output sequence on the identical two-dimensional array of data values; it differs only in re-expressing the single “select between [a data value] and [an alternative value] using a masking multiplexer” limitation as an explicit per-data-value iteration (“for each data value of the two-dimensional array of data values, select, using a multiplexer, between: a data value -; and an alternative value”) and in the second “a plurality of indices” terminology addressed in the § 112(b) discussion above. Reciting a single selection operation as an explicit iteration over each member of the same array, rather than as a single compound clause, does not narrow or broaden the claimed subject matter and adds no patentable weight. Instant claim 1 is accordingly not patentably distinct from claim 1 of the ’277 patent.
Claim 1 of the ’327 patent (the grandparent) recites the same masking-multiplexer limitations, but recites the data masking circuit as implemented specifically in “[a] field programmable gate array (FPGA) comprising a configurable interconnect fabric connecting a plurality of logic blocks, the configurable interconnect fabric and the logic blocks being configured to implement” the data masking circuit, rather than in a generic “electronic circuit.” An FPGA is itself expressly admitted to be “a hardware device” (spec § 0002), i.e., a species of electronic circuit; every FPGA meeting the limitations of ’327 claim 1 necessarily also meets every limitation of instant claim 1. Instant claim 1 is accordingly also not patentably distinct from, and is in fact broader in scope than, ’327 claim 1 - the paradigmatic obviousness-type double patenting scenario of a later, generic claim directed to the same invention already patented in more specific form. See MPEP §§ 804, 1490; In re Van Ornum, 686 F.2d 937 (CCPA 1982); In re Goodman, 11 F.3d 1046 (Fed. Cir. 1993).
Claim 9 of the ’277 patent recites: “Computer storage media storing a configuration file, the configuration file specifying an accelerator electronic circuit comprising a data masking circuit configured to: receive input data comprising a two-dimensional array including data values at a plurality of indices of the two-dimensional array of the input data; select between a data value of the data values and an alternative value using a masking multiplexer to generate masked data, the masking multiplexer being controlled by a mask value of a plurality of mask values at indices corresponding to the indices of the input data; and output the masked data.” This limitation is word-for-word identical to instant claim 9, again differing only in the per-data-value iterative re-expression of the masking-multiplexer selection addressed for claim 1 above. Instant claim 9 is accordingly not patentably distinct from ’277 claim 9. Claim 9 of the ’327 patent recites the same operations, but specifies that the configuration file configures “a field programmable gate array (FPGA) comprising a configurable interconnect fabric and a plurality of logic blocks” rather than a generic accelerator electronic circuit; for the same species/genus reason given for claim 1, every FPGA-configuring configuration file meeting ’327 claim 9 necessarily also meets instant claim 9, so instant claim 9 is also broader than, and not patentably distinct from, ’327 claim 9.
Claim 16 of the ’277 patent recites: “A method for accelerating computations, the method comprising: receiving, by a machine learning model training application executed by a computing device comprising a processor, memory, and an accelerator electronic circuit, labeled training data; supplying, by the machine learning model training application, the labeled training data to a machine learning model to generate input data; supplying, by the machine learning model training application, the input data to a data masking circuit implemented in the accelerator electronic circuit to generate masked data; receiving, at the data masking circuit of the accelerator electronic circuit, input data including data values at a plurality of indices of the input data; selecting, by a masking multiplexer of the data masking circuit, between a data value of the data values and an alternative value to generate the masked data, the masking multiplexer being controlled by a mask value of a plurality of mask values at indices corresponding to the indices of the input data; outputting the masked data; computing a plurality of output scores of the machine learning model based on the masked data computed by the machine learning model in response to the labeled training data; updating the machine learning model based on the output scores; and outputting the updated machine learning model as a trained machine learning model.” This method is expressly performed “by a computing device comprising a processor, memory, and an accelerator electronic circuit” - the identical three structural elements recited in instant claim 16 (“a processor; an accelerator electronic circuit comprising a data masking circuit; and memory storing instructions that, when executed, cause the processor to” perform the recited steps) - and recites every one of the same operative steps. Reciting, in system/apparatus form, a processor and memory storing instructions that cause the processor to perform a method already claimed as being performed by a computing device comprising a processor, memory, and the same accelerator electronic circuit is nothing more than claiming the well-understood, routine generic computer implementation already disclosed as the operating environment of the patented method claim; a method claim and its corresponding apparatus/system claim, reciting structure for performing the identical steps, are not patentably distinct from one another. See MPEP § 804; In re Mettke, 570 F.3d 1356 (Fed. Cir. 2009). Instant claim 16 is accordingly not patentably distinct from ’277 claim 16. Claim 16 of the ’327 patent recites the identical method performed by “a computing device comprising a processor, memory, and the FPGA” rather than a generic accelerator electronic circuit; for the reasons given above regarding the FPGA/electronic-circuit relationship, instant claim 16 is also broader than, and not patentably distinct from, ’327 claim 16.
The examiner has retrieved and compared the complete, verbatim claim sets of both the ’327 and ’277 patents against each of instant dependent claims 2-8, 10-15, and 17-19; every instant dependent claim corresponds, limitation-for-limitation, to an already-patented claim of both reference patents (claim numbers below refer to the ’277 patent; the correspondingly-numbered claim of the ’327 patent recites the same subject matter in the FPGA-specific language addressed above for claims 1, 9, and 16, and is not patentably distinct for the same reason):
instant claim 2 (mask generation circuit generating mask values based on corresponding indices) corresponds to ’277 claim 2, verbatim;
instant claim 3 (mask selection multiplexer selecting between internally-generated and externally-supplied mask values) corresponds to ’277 claim 5, verbatim;
instant claim 4 (mask generation circuit configured for a triangular mask pattern) corresponds to ’277 claim 6, verbatim;
instant claim 5 (mask pattern selected from upper triangular, lower triangular, block upper triangular, block lower triangular, or alternating) corresponds to ’277 claim 7, verbatim;
instant claim 6 (mask generation circuit receiving one or more mask parameters and selecting the mask pattern based on the parameters) corresponds to ’277 claim 8, including the identical “based on the one or mask parameters” phrasing addressed in the § 112(b) discussion above - confirming that this phrasing, already examined and allowed to issue in both reference patents without an indefiniteness rejection, does not render the instant claim indefinite;
instant claim 7 (masked data supplied to a SoftMax function with a floating-point large-negative alternative value) corresponds to ’277 claim 3, verbatim; and
instant claim 8 (electronic circuit connected to a computer system training a machine learning model on training data masked to accelerate training) corresponds to ’277 claim 4, verbatim.
Each of instant claims 2-8 is accordingly not patentably distinct from the corresponding already-patented claim of each of the ’277 and ’327 patents.
instant claim 10 (configuration file further specifying a mask generation circuit generating mask values based on corresponding indices) corresponds to ’277 claim 10, differing only in the incidental omission of the article “a” before “mask generation circuit” in the issued claim;
instant claim 11 (mask selection multiplexer selecting between internally-generated and externally-supplied mask values, configuration-file context) corresponds to ’277 claim 12, verbatim;
instant claim 12 (mask generation circuit configured for a triangular mask pattern, configuration-file context) corresponds to ’277 claim 13, verbatim;
instant claim 13 (mask pattern selected from the same five enumerated patterns, configuration-file context) corresponds to ’277 claim 14, verbatim;
instant claim 14 (mask generation circuit receiving one or more mask parameters and selecting the mask pattern based on the parameters, configuration-file context) corresponds to ’277 claim 15, again including the identical “based on the one or mask parameters” phrasing; and
instant claim 15 (masked data supplied to a SoftMax function, configuration-file context) corresponds to ’277 claim 11, verbatim.
Each of instant claims 10-15 is accordingly not patentably distinct from the corresponding already-patented claim of each of the ’277 and ’327 patents.
instant claim 17 (mask generation circuit generating mask values within the accelerator electronic circuit based on corresponding indices) corresponds to ’277 claim 17, verbatim;
instant claim 18 (mask selection multiplexer selecting between internally-generated and externally-supplied mask values, system context) corresponds to ’277 claim 18, verbatim; and
instant claim 19 (masked data supplied to a SoftMax function, system context) corresponds to ’277 claim 19, verbatim.
Each of instant claims 17-19 is accordingly not patentably distinct from the corresponding already-patented claim of each of the ’277 and ’327 patents, for the same apparatus/method reasons given for claim 16 above.
A terminal disclaimer under 37 CFR 1.321(c), disclaiming the terminal portion of any patent granted on this application beyond the expiration date of both U.S. Patent Nos. 11,934,327 and 12,346,277, and providing that the patent shall be enforceable only for and during the period it and each reference patent are commonly owned, would obviate the double patenting rejection as to all of claims 1-19.
Claim Rejections - 35 USC § 112
The following is a quotation of the second paragraph of 35 U.S.C. 112:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1, 2, 4, 7, and 9 are rejected under 35 U.S.C. 112(b) as failing to particularly point out and distinctly claim the subject matter regarded as the invention:
Claims 1, 3, 9, 11, 16, and 18 (and claims dependent therefrom) are rejected under 35 U.S.C. 112(b) as indefinite for lack of antecedent basis / ambiguous double recitation of “a plurality of indices.”
Claim 1 first establishes “a plurality of indices of the two-dimensional array of the input data” and refers back to it as “the plurality of indices” when reciting the masking multiplexer’s first input (“a data value of the data values at a corresponding index of the plurality of indices”). The claim then recites that the masking multiplexer is “controlled by a mask value of a plurality of mask values at an index of a plurality of indices corresponding to the indices of the input data.” Because this second “a plurality of indices” is introduced with an indefinite article rather than tied back (e.g., “said plurality of indices” or “the plurality of indices”), it is unclear whether the mask values are indexed by the very same “plurality of indices” already recited for the input data, or by a second, separately-introduced plurality of indices that merely “correspond[s] to” - but is not identical to - the first. Under the broadest reasonable interpretation, this creates ambiguity as to whether one common index space or two distinct (merely corresponding) index spaces are being claimed, rendering the metes and bounds of the mask-control limitation unclear. The identical informality appears in claim 9 (“a plurality of indices of the two-dimensional array of the input data” ... “an index of a plurality of indices corresponding to the indices of the input data”) and claim 16 (same construction), and is inherited by claims 3, 11, and 18 by dependency. Correction is suggested by amending the second recitation to read “...controlled by a mask value of the plurality of mask values at an index of the plurality of indices corresponding to the indices of the input data” or by expressly defining a single shared index variable for both the data values and the mask values.
Claim 16 (and claims 17-19 depending therefrom) is rejected under 35 U.S.C. 112(b) for lack of antecedent basis.
Claim 16 recites that the input data supplied to the data masking circuit includes “data values at a plurality of indices” - it does not characterize the input data as being organized into a two-dimensional array anywhere in the claim. The claim later recites, however, “for each data value of the two-dimensional array of data values, select, using a multiplexer, between...” (emphasis added). “The two-dimensional array of data values” lacks antecedent basis in claim 16, because no two-dimensional array was previously introduced (contrast claim 1, which does recite “a two-dimensional array of data values” in its first clause, and claim 9, which recites “a two-dimensional array including data values” in its first clause - in each of those claims the later reference to “the two-dimensional array of data values” is proper). Correction is suggested either by conforming claim 16’s first data-related clause to recite a two-dimensional array (paralleling claims 1 and 9), or by amending the later clause to remove “the two-dimensional array of.”
Claim 16 is further rejected under 35 U.S.C. 112(b) as indefinite for an internally inconsistent recitation of which component computes the masked data.
Claim 16 recites “supply the input data to the data masking circuit to generate masked data” (i.e., the data masking circuit - not the machine learning model - generates the masked data), but later recites “compute a plurality of output scores of the machine learning model based on the masked data computed by the machine learning model in response to the labeled training data” (emphasis added). This later clause attributes computation of “the masked data” to “the machine learning model,” directly conflicting with the earlier clause attributing that computation to “the data masking circuit.” It is unclear whether “computed by the machine learning model” was intended to modify “the masked data” (as literally written, and which conflicts with the earlier clause) or was intended to modify some other quantity, such as activations or output scores generated by the machine learning model when supplied with the masked data as input (consistent with spec ¶·0092-0093 and Fig. 7). Clarifying amendment is suggested, e.g., “...output scores of the machine learning model, the output scores computed by the machine learning model in response to the masked data...”
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art t which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-19 are rejected under AIA 35 U.S.C. § 103 as being unpatentable over Applicants Admitted Prior Arts (hereinafter AAPA).
In regard to claim 1, AAPA teaches an electronic circuit - the “comparative system” of Fig. 2 - that receives input data (training example input data 210) organized as a two-dimensional array (an LxL attention-score matrix, spec ¶¶ 0029-0030) at a plurality of indices (i,j), and that, for each data value of that array, computes masked data by combining the data value at a corresponding index with an alternative value (a large negative constant, e.g., -10,000.0), the selection at each index being controlled by a mask value from a plurality of mask values (mask(i,j) ∈ {0.0, 1.0}) stored at indices corresponding to the indices of the input data (attention mask 240, device memory 263), and that outputs the resulting masked data (x_masked 230) for further processing. See spec ¶¶ 0029-0038, Equation (1), Fig. 2.
AAPA does not teach that the selection between the data value and the alternative value is performed “using a multiplexer” - AAPA’s comparative system instead performs the selection arithmetically, using a floating-point subtraction circuit 266, a floating-point multiplier 267, and a second floating-point subtraction circuit 268 to implement Equation (1) (spec ¶ 0031).
It is a matter of common, undisputable knowledge in digital circuit design - of which the examiner takes Official Notice - that a two-input conditional selection of the general form “select value A when a control bit is one condition, else select value B” (here, mask(i,j) versus 1 - mask(i,j)) can be implemented equivalently and interchangeably by (i) arithmetic blending of the two candidate values as in AAPA’s Equation (1), or (ii) a two-input multiplexer whose select line is driven by the single-bit control value, choosing directly between the two candidate values without any floating-point arithmetic. This equivalence is confirmed by Applicant’s own specification, which describes the very substitution recited in claim 1 as implementing “the same function” as the arithmetic approach using “a ternary if or conditional operator” (spec ¶ 0096: “y = mask ? x : -10000.0” performs the identical function as the two-adder-plus-multiplier arithmetic of Equation (1)).
It would have been obvious to a person of ordinary skill in the art at the time of filing to substitute AAPA’s known arithmetic (subtraction/multiplication) implementation of the masking selection with the well-known, functionally-equivalent multiplexer-based implementation, because doing so is nothing more than the simple substitution of one known element (an arithmetic selector) for another known, equally suitable element (a multiplexer selector) to obtain the predictable result of masked data - namely, the very result AAPA’s own arithmetic approach already produces. See MPEP § 2143(I)(B); KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398, 416 (2007). A person of ordinary skill would have had a reasonable expectation of success in making this substitution because multiplexer-based value selection was, and is, one of the most basic building blocks of digital logic design, requiring no more than ordinary skill to apply to the masking context already fully disclosed by AAPA.
In regard to claim 2, AAPA teaches (spec ¶ 0029) that, for autoregressive transformer models, mask values are conventionally defined as a deterministic function of the corresponding row/column indices of the input data (a token at sequence position i may not attend to positions j ≥ i+1), i.e., that the plurality of mask values is generated based on the corresponding indices of the input data as a matter of background knowledge in the field.
AAPA’s Fig. 2 comparative system does not itself include a “mask generation circuit” - it instead fetches a pre-computed mask from external memory (attention mask 240, device memory 263) rather than generating the mask values on-device from the indices.
The examiner takes Official Notice that implementing a known, closed-form boolean/arithmetic relationship (such as the known index-comparison relationship admitted at AAPA ¶ 0029) directly in dedicated combinational logic - i.e., a “mask generation circuit” - rather than pre-computing and fetching the same values from memory, is a basic and entirely predictable engineering design choice with a finite number of known, foreseeable implementations (compute on-the-fly in hardware versus store and fetch), both well known in circuit design generally before the priority date.
It would have been obvious to a person of ordinary skill to implement AAPA’s admittedly-known index-based mask relationship as an on-device mask generation circuit rather than pre-computing and fetching the mask, as this is no more than choosing among a finite number of known, predictable design alternatives for realizing an admittedly-known function, each yielding the predictable result of supplying the same mask values used in AAPA’s own system. See MPEP § 2143(A), (G).
In regard to claim 3, AAPA’s Fig. 2 comparative system teaches sourcing the mask values externally, from a device memory (263) storing mask data fetched over a communications bus from system memory (250). Combined with the mask generation circuit rendered obvious at claim 2 above, AAPA and Official Notice together establish that both (i) generating mask values on-device from indices, and (ii) fetching externally-stored mask values, were independently known design options for supplying mask values to a masking operation.
Neither AAPA’s Fig. 2 system nor the mask-generation-circuit modification of claim 2 by itself teaches a “mask selection multiplexer” that selects, at run time and based on a mask selection input, between these two already-known mask sources.
The examiner takes Official Notice that providing a multiplexer to select, under control of a mode/selection signal, between an internally-generated operand and an externally-supplied (e.g., memory-fetched) operand is one of the most basic and well-known patterns in general-purpose digital and computer-architecture design (e.g., an arithmetic-logic-unit operand-select multiplexer choosing between an immediate/generated value and a register- or memory-sourced value under control of an instruction-derived select signal), predating the priority date by decades.
It would have been obvious to combine AAPA’s known external-mask-fetch technique with the mask-generation-circuit modification of claim 2 using a selection multiplexer of the well-known type noticed above, because combining two known alternative sources of the identical needed value (a mask value) via a known selector, to accommodate both use cases (frequently-used masks generated on-chip, and specialized or infrequently-used masks supplied externally), yields nothing more than the predictable result of a system usable in either mode. Applicant’s own specification confirms the predictability and known motivation for this combination: “[s]ome further aspects of embodiments relate to a hybrid circuit configurable to generate and apply a mask on-device or to apply a mask received from system memory over the communications bus (e.g., for less frequently used or specialized masks)” (spec ¶ 0039), corroborating that combining the two known techniques via a selector was well within the level of ordinary skill and yields a predictable, foreseeable result. See MPEP § 2143(A).
In regard to claim 4, AAPA (spec ¶ 0029) admits that, for autoregressive transformer models, “a mask is used to mask out tokens at positions greater than or equal to [i+1]” and depicts (Fig. 1, mask 40) the resulting mask pattern as “an upper triangular mask.” This is the identical i ≤ j / i > j index-comparison relationship recited in Applicant’s own Equations (2)-(3) (spec ¶ 0050) as defining “an upper triangular mask.”
As discussed for claim 2, AAPA’s Fig. 2 system does not implement this relationship in a mask generation circuit; it fetches the mask from memory.
For the reasons given at claim 2 (Official Notice of the predictability of implementing an admittedly-known boolean relationship directly in combinational logic), it would have been obvious to implement AAPA’s admittedly-known upper-triangular masking relationship as a mask generation circuit configured to generate mask values “in accordance with a triangular mask pattern,” yielding the predictable result of the identical, already-known upper-triangular mask.
In regard to claim claim 5, AAPA (spec ¶ 0029, Fig. 1) admits the upper triangular mask pattern discussed at claim 4. An upper triangular mask’s complement (a “lower triangular mask”) and coarse block-wise groupings of the same triangular relationship (“block upper”/“block lower” triangular masks) are straightforward, predictable variants of the identically-admitted relationship - reversing an inequality (i ≤ j vs. i > j) and applying a known relationship at block, rather than single-element, granularity are each well-known, basic techniques for adapting a masking relationship to a coarser or inverted application, and an “alternating mask” (masking every other index, i.e., masking based on the parity of an index) is one of the most basic patterns expressible in digital logic (a single low-order address/index bit).
AAPA’s Fig. 2 system does not implement any of these variants in a mask generation circuit for the reasons given at claim 4.
It would have been obvious to a person of ordinary skill, having been taught the upper-triangular masking relationship and its hardware implementation (claim 4), to further configure the same mask generation circuit to selectably produce these predictable, closely-related variants (the complementary lower-triangular relationship, block-wise groupings of the same relationship, or index-parity masking), as each represents a routine, foreseeable extension applying the identically-admitted comparison technique, yielding predictable results. See MPEP § 2143(A), (G).
In regard to claim 6, claim 6 depends from claim 5 and further recites that the mask generation circuit is configured to receive one or more mask parameters and to select the mask pattern - i.e., among the upper triangular, lower triangular, block upper triangular, block lower triangular, and alternating mask patterns already rendered obvious at claim 5 - based on the one or more mask parameters.
The examiner takes further Official Notice that providing a selection mechanism (e.g., a mode-select multiplexer or decoder) that chooses, under control of one or more input parameters, among a finite number of pre-configured, already-known circuit operating modes is one of the most basic and well-known patterns in general-purpose digital and computer-architecture design (e.g., an ALU’s opcode/function-select-controlled multiplexer choosing among several pre-configured arithmetic or logic operations), predating the priority date by decades - the identical class of known parameter-controlled selection technique already noticed at claim 3 (there selecting between generated and externally-supplied mask sources), here applied instead to select among the several mask-pattern-generation modes already rendered obvious at claim 5.
It would have been obvious to a person of ordinary skill to configure the mask generation circuit of claim 5 to receive one or more mask parameters and select, using such a well-known parameter-controlled selection mechanism, among the upper triangular, lower triangular, block upper triangular, block lower triangular, and alternating mask-generation modes already rendered obvious at claim 5, because doing so is nothing more than applying a known selection technique to choose among a finite number of known, predictable mask-pattern alternatives, yielding the predictable result of a run-time-configurable choice among mask patterns already taught to be obvious. See MPEP § 2143(A), (G).
In regard to claim 7, AAPA teaches, verbatim, every limitation of claim 7: the masked data (x_masked 230) is supplied to a circuit implementing a SoftMax function (SoftMax and other processing 270); the data values and the alternative value are in a floating-point data format (BFloat16, IEEE half-precision FP16, spec ¶ 0033); and the alternative value (-10,000.0, or other application-appropriate large negative constants such as -100,000 or -1,000,000 depending on the magnitude of the data values, spec ¶¶ 0033-0034) is expressly chosen to be “sufficiently large” that supplying it to the SoftMax function “will produce a number that is small enough to be rounded to zero ... in most low-precision floating-point formats” (spec ¶ 0033). This limitation requires no modification beyond the multiplexer substitution already addressed for claim 1, from which claim 7 depends.
In regard to claim 8, AAPA teaches, verbatim, that the (masking) electronic circuit is connected to a computer system configured to train a machine learning model based on a plurality of training data (spec ¶¶ 0002-0008, 0027-0028), and that the data masking circuit masks the training data supplied as input data in order to accelerate training the machine learning model - indeed, this stated purpose (“hardware acceleration of data masking ... improves the performance of machine learning model training processes,” spec ¶ 0008) is the very problem AAPA’s own background section identifies as motivating the invention. This limitation requires no modification beyond the multiplexer substitution already addressed for claim 1, from which claim 8 depends.
In regard to claim 9, claim 9 recites the identical data-masking-circuit operations of claim 1, cast as a configuration file specifying an accelerator electronic circuit rather than as the electronic circuit itself. AAPA independently establishes that FPGA configuration files (“bit files”) that configure hardware to implement a specified function were well known generally: “configuring an FPGA with a configuration file sets the interconnects ... thereby configuring the FPGA to perform the particular function specified by the configuration file (sometimes referred to as a ‘bit file’)” (spec ¶ 0002). Combining this admittedly-known configuration-file technique with the AAPA/Official-Notice combination applied to claim 1 renders obvious a configuration file that specifies an accelerator electronic circuit having the masking-multiplexer structure and operation of claim 1, for the same reasons (simple substitution of a known multiplexer-based selector for AAPA’s known arithmetic selector) given at claim 1.
In regard to claim 10, for the reasons given at claim 2 (mask generation circuit), applied here to the configuration-file context of claim 9.
See the analysis at claim 2, which applies equally to the corresponding “configuration file further specifies a mask generation circuit” limitation of claim 10.
In regard to claim 11, for the reasons given at claim 3 (mask selection multiplexer), applied here to the configuration-file context of claim 9.
See the analysis at claim 3, which applies equally to the corresponding “configuration file further specifies a mask selection multiplexer” limitation of claim 11.
In regard to claim 12, for the reasons given at claim 4 (triangular mask pattern), applied here to the configuration-file context of claim 9.
See the analysis at claim 4, which applies equally to the corresponding limitation of claim 12.
In regard to claim 13, for the reasons given at claim 5 (upper/lower/block/alternating mask patterns), applied here to the configuration-file context of claim 9.
See the analysis at claim 5, which applies equally to the corresponding limitation of claim 13.
In regard to claim 14, for the reasons given at claim 6 (mask-parameter-based selection among mask-generation modes), applied here to the configuration-file context of claim 9.
See the analysis at claim 6, which applies equally to the corresponding limitation of claim 14.
In regard to claim 15, for the same reasons given at claim 7, see the analysis at claim 7, which applies equally to the corresponding SoftMax/large-negative-value limitation of claim 15.
In regard to claim 16, AAPA teaches every element of the claimed system: a processor and memory of a computing device (spec ¶¶ 0099-0100, computing device 800/902); an accelerator electronic circuit comprising a data masking circuit connected to that computing device (spec ¶¶ 0002-0038, Fig. 2); and - as to the recited training-application instructions - AAPA admits, verbatim, a supervised-learning training loop that receives labeled training data (spec ¶ 0091, operation 710), supplies the labeled training data to a machine learning model to generate input data/activations (spec ¶ 0091), supplies that input data to the (masking-augmented) data masking circuit to generate masked data by the same index-based multiplexer-substituted selection addressed at claim 1 (spec ¶ 0092), computes output scores of the machine learning model based on the masked data (spec ¶ 0092, operation 734), updates the machine learning model based on the output scores via backpropagation/gradient descent (spec ¶ 0093, operation 740), and outputs the updated model as a trained machine learning model (spec ¶ 0093) upon completion of training (spec ¶ 0093, operation 750).
AAPA’s Fig. 2 comparative system does not perform the masking-multiplexer selection recited in the “data masking circuit” clause of claim 16 (it uses arithmetic selection, as discussed at claim 1).
For the same reasons given at claim 1 - the well-known, art-recognized equivalence (Official Notice, corroborated by Applicant’s own ¶ 0096) of arithmetic selection and multiplexer-based selection for implementing a two-way conditional choice between a data value and an alternative value.
It would have been obvious to substitute AAPA’s known arithmetic masking selection, within AAPA’s own admittedly-known supervised training loop, with the well-known multiplexer-based selection, yielding the predictable result of masked training data usable in the identical training loop already disclosed by AAPA.
In regard to claim 17, for the reasons given at claim 2 (mask generation circuit), applied here to the system context of claim 16. See the analysis at claim 2, which applies equally to the corresponding limitation of claim 17.
In regard to claim 18, for the reasons given at claim 3 (mask selection multiplexer), applied here to the system context of claim 16. See the analysis at claim 3, which applies equally to the corresponding limitation of claim 18.
In regard to claim 19, for the same reasons given at claim 7.
See the analysis at claim 7, which applies equally to the corresponding SoftMax/large-negative-value limitation of claim 19.
Examiner's note:
If Applicant traverses any statement of Official Notice set forth above, documentary evidence will be provided in the next Office Action (MPEP § 2144.03(C)).
Closest Issues and Potential Allowable Subject Matter
The specific hardware realization of the mask generation circuit that computes triangular/block-triangular/alternating mask patterns in SIMD-width chunks using a decrement circuit, one-padding circuits, a shifter, and comparator-driven select signals keyed to row/column counters (spec ¶¶ 0068-0090, Figs. 6A-6C), and the mask selection multiplexer architecture permitting run-time selection between an internally-generated mask and an externally-supplied mask (spec ¶¶ 0061-0067, Fig. 5A), were not found in the search conducted to be taught or suggested by any single non-family reference or clearly-supported combination thereof - the rejections of claims 2, 3, 6, 10, 11, 14, 17, and 18 above rest on Official Notice of general digital-design principles rather than an applied documentary reference, and are correspondingly more vulnerable to a well-supported traversal.
If claims 2, 3, 6, 10, 11, 14, 17, and 18 are amended to positively recite the specific mask-generation hardware detailed above (e.g., the comparator/shifter/one-padding-circuit arrangement of Fig. 6A/6C, or that the mask selection multiplexer’s selection input is distinct from, and independent of, the selected mask pattern itself), and provided the double patenting rejection is overcome by terminal disclaimer, this subject matter - the specific SIMD-chunked hardware realization of run-time-configurable triangular mask generation and generated/external mask selection - appears on the present record to represent the closest candidate for allowance identified in this search.
Conclusion
All claims are rejected.
The prior arts made of record and not relied upon are considered pertinent to applicant's disclosure.
U.S. Pub No. 2023/0376663, discloses an alternative (non-multiplexer) circuit architecture - complementary-row merging of two rows having complementary numbers of unmasked values - for efficiently computing a normalized (e.g., SoftMax) function over triangularly-masked attention data without wasted computation on masked-out positions. Post-dates the instant priority date and is therefore not applied as prior art, but is relevant background for the triangular-mask-generation and SoftMax-adjacent dependent claims, and should be checked for common inventorship/ownership as a potential further ODP reference.
EP 4,293,575 A1 (priority June 17, 2022), discloses generating an attention mask from a padding mask via an outer-product operation for hardware-accelerated transformer inference. Post-dates the instant priority date; relevant background for mask-generation-circuit claims 2, 10, and 17.
U.S. Pub No. 2022/0391172, discloses fixed-function hardware SoftMax/exponential circuits (relevant to the “SoftMax and other processing” circuits of spec Figs. 3A/4A/5A) but does not disclose mask-based value selection. Post-dates the instant priority date.
U.S. Pub No. 2025/0028563, teaches accelerator Architecture For A Transformer Machine Learning Model. General processing-in-memory transformer accelerator background; post-dates the instant priority date.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to examiner Raymond Phan, whose telephone number is (571) 272-3630. The examiner can normally be reached on Monday-Friday from 6:30AM- 3:00PM. The Group Fax No. (571) 273-8300.
Communications via Internet e-mail regarding this application, other than those under 35 U.S.C. 132 or which otherwise require a signature, may be used by the applicant and should be addressed to [raymond.phan@uspto.gov].
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Jung can be reached at (571) 270-3779. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
All Internet e-mail communications will be made of record in the application file. PTO employees do not engage in Internet communications where there exists a possibility that sensitive information could be identified or exchanged unless the record includes a properly signed express waiver of the confidentiality requirements of 35 U.S.C. 122. This is more clearly set forth in the Interim Internet Usage Policy published in the Official Gazette of the Patent and Trademark on February 25, 1997 at 1195 OG 89.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see hop://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Any inquiry of a general nature or relating to the status of this application should be directed to the TC 2100 central telephone number is (571) 272-2100.
/RAYMOND N PHAN/
Primary Examiner, Art Unit 2175