DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1-30 are pending examination.
Information Disclosure Statement
The Information Disclosure Statement submitted by Applicant on 8/15/2024 has been considered.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-9, 12-26, and 29-30 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (abstract idea) without significantly more.
Regarding claim 1,
Step 1: Claim 1 is directed to a method.
Step 2A, Prong 1: Claim 1 recites the following limitations:
processing, via the geometric algebra transformer, the multivector inputs to generate multivector outputs, (i.e., this limitation comprises mathematical calculations in view of paragraphs [0060]-[0062] of Applicant’s specification)
Hence, the claim recites an abstract idea.
Step 2A, Prong 2: Claim 1 recites the additional elements of “a processor-implemented method of processing data using a geometric algebra transformer, the processor-implemented method comprising:”, “via the geometric algebra transformer”, and “wherein the geometric algebra transformer is trained to process geometric algebra representations associated with the multivector inputs and to be equivariant with respect to translations and rotations.” These limitations are recited at a high-level of generality such that they amount to no more than mere instructions to apply the exception using generic computer components. (see MPEP 2106.05(f)). Furthermore, the additional element of “receiving, at the geometric algebra transformer, multivector inputs processed from raw data associated with a three-dimensional space;” is considered insignificant extra-solution activity. (see MPEP 2106.05(g)). Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 1 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of “a processor-implemented method of processing data using a geometric algebra transformer, the processor-implemented method comprising:”, “via the geometric algebra transformer”, and “wherein the geometric algebra transformer is trained to process geometric algebra representations associated with the multivector inputs and to be equivariant with respect to translations and rotations.” are recited at a high-level of generality such that they amount to no more than mere instructions to apply the exception using generic computer components. (see MPEP 2106.05(f)). Furthermore, the additional element of “receiving, at the geometric algebra transformer, multivector inputs processed from raw data associated with a three-dimensional space;” was considered insignificant extra-solution activity under Step 2A, Prong 2 and it is re-evaluated under Step 2B to determine if it is more than what the courts have considered well-understood, routine, and conventional activity in the field. The court decisions cited in MPEP 2106.05(d)(II) have determined that mere data transmission (as it is currently claimed) is a well-understood, routine, conventional activity in the field supported by Berkheimer. (see also, i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information). Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 2,
Step 2A, Prong 1: Claim 2 recites an abstract idea as inherited from claim 1.
Step 2A, Prong 2: Claim 2 recited the additional element of wherein the multivector inputs comprise multi-component multivectors. This additional element merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 2 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of wherein the multivector inputs comprise multi-component multivectors, merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 3,
Step 2A, Prong 1: Claim 3 recites an abstract idea as inherited from claim 1.
Step 2A, Prong 2: Claim 3 recites the additional element of “wherein the multi-component multivectors comprise embedded geometric objects.” This additional element merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 3 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the multi-component multivectors comprise embedded geometric objects.” merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 4,
Step 2A, Prong 1: Claim 4 recites an abstract idea as inherited from claim 1. Claim 4 recites the additional limitation of:
wherein the embedded geometric objects are embedded into the multi-component multivectors …(i.e., this limitation comprises mathematical calculations in view of paragraphs [0060]-[0064] of Applicant’s specification)
Hence, the claim recites an abstract idea.
Step 2A, Prong 2: Claim 4 recites the additional element of “using a geometric algebra embedding component” to perform the limitation stated above. This additional element is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component. (See MPEP 2106.05(f)). Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 4 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “using a geometric algebra embedding component”, to perform the limitation stated above, is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component. (See MPEP 2106.05(f)). Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 5,
Step 2A, Prong 1: Claim 5 recites an abstract idea as inherited from claim 1.
Step 2A, Prong 2: Claim 5 recites the additional elements of “wherein the embedded geometric objects comprise at least one of a scalar, a vector, a bivector, a trivector, or a pseudoscalar.” These additional elements merely generally link the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 5 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of “wherein the embedded geometric objects comprise at least one of a scalar, a vector, a bivector, a trivector, or a pseudoscalar” merely generally link the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 6,
Step 2A, Prong 1: Claim 6 recites an abstract idea as inherited from claim 1.
Step 2A, Prong 2: Claim 6 recites the additional elements of “wherein the geometric algebra transformer further comprises: an input equilinear layer; a transformer block; and an output equilinear layer.” These additional elements merely generally link the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 6 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of “wherein the geometric algebra transformer further comprises: an input equilinear layer; a transformer block; and an output equilinear layer” merely generally link the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 7,
Step 2A, Prong 1: Claim 7 recites an abstract idea as inherited from claim 1.
Step 2A, Prong 2: Claim 7 recites the additional elements of “wherein the geometric algebra transformer further comprises a plurality of transformer blocks.” These additional elements merely generally link the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 7 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of “wherein the geometric algebra transformer further comprises a plurality of transformer blocks” merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 8,
Step 2A, Prong 1: Claim 8 recites an abstract idea as inherited from claim 1.
Step 2A, Prong 2: Claim 8 recites the additional elements of “wherein the transformer block further comprises: a first normalization layer; a first equilinear layer; a geometric attention layer; a first geometric product engine; a second equilinear layer; a first addition engine; a second normalization layer; a third equilinear layer; a second geometric product engine; a scalar-gated nonlinearity layer; a fourth equilinear layer; and a second addition engine.” These additional elements merely generally link the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 8 Claim 5 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of “wherein the transformer block further comprises: a first normalization layer; a first equilinear layer; a geometric attention layer; a first geometric product engine; a second equilinear layer; a first addition engine; a second normalization layer; a third equilinear layer; a second geometric product engine; a scalar-gated nonlinearity layer; a fourth equilinear layer; and a second addition engine” merely generally link the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 9,
Step 2A, Prong 1: Claim 9 recites an abstract idea as inherited from claim 1.
Step 2A, Prong 2: Claim 9 recites the additional element of “wherein the scalar-gated nonlinearity layer comprises a scalar-gated Gaussian Error Linear Units nonlinearity layer.” This additional element merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 9 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the scalar-gated nonlinearity layer comprises a scalar-gated Gaussian Error Linear Units nonlinearity layer” merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 12,
Step 2A, Prong 1: Claim 12 recites an abstract idea as inherited from claim 1.
Step 2A, Prong 2: Claim 12 recites the additional elements of “wherein the first normalization layer, the first geometric product engine, the first addition engine, the second normalization layer, the second geometric product engine, the scalar-gated nonlinearity layer, and the second addition engine are fixed components and wherein the first equilinear layer, the geometric attention layer, the second equilinear layer, the third equilinear layer, and the fourth equilinear layer are learnable components.” These additional elements merely generally link the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 12 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of “wherein the first normalization layer, the first geometric product engine, the first addition engine, the second normalization layer, the second geometric product engine, the scalar-gated nonlinearity layer, and the second addition engine are fixed components and wherein the first equilinear layer, the geometric attention layer, the second equilinear layer, the third equilinear layer, and the fourth equilinear layer are learnable components” merely generally link the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 13,
Step 2A, Prong 1: Claim 13 recites an abstract idea as inherited from claim 1.
Step 2A, Prong 2: Claim 13 recites the additional elements of “wherein each layer maps between multivector data and is equivariant.” These additional elements merely generally link the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 13 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of “wherein each layer maps between multivector data and is equivariant” merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 14,
Step 2A, Prong 1: Claim 14 recites an abstract idea as inherited from claim 1.
Step 2A, Prong 2: Claim 14 recites the additional elements of “wherein the multivector inputs comprise at least one of a scalar value, a plane with a normal value, a line with a direction value, a point value, a pseudoscalar value, a reflection value through a plane with a normal value, a translation value, a rotation value, or a point reflection value and wherein the geometric algebra transformer represents both geometric objects and transformations of the geometric objects via use of the multivector inputs”. These additional elements merely generally link the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 14 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of wherein the multivector inputs comprise at least one of a scalar value, a plane with a normal value, a line with a direction value, a point value, a pseudoscalar value, a reflection value through a plane with a normal value, a translation value, a rotation value, or a point reflection value and wherein the geometric algebra transformer represents both geometric objects and transformations of the geometric objects via use of the multivector inputs” merely generally link the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 15,
Step 2A, Prong 1: Claim 15 recites an abstract idea as inherited from claim 1.
Step 2A, Prong 2: Claim 15 recites the additional element of “wherein the multivector inputs uniquely represent various geometric types.” This additional element merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 15 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the multivector inputs uniquely represent various geometric types” merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 16,
Step 2A, Prong 1: Claim 16 recites an abstract idea as inherited from claim 1. Claim 16 recites the additional following limitations:
wherein the multivector inputs are generated from a geometric product of vectors… (i.e. this limitation comprises mathematical calculations in view of paragraphs [0043]-[0047] and [0064] of Applicant’s specification)
Hence, the claim recites an abstract idea.
Step 2A, Prong 2: Claim 16 recites the additional element of “wherein the multivector inputs comprises a representation of geometric objects and operators associated with the geometric objects.” This additional element merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 16 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the multivector inputs comprises a representation of geometric objects and operators associated with the geometric objects” merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 17,
Step 2A, Prong 1: Claim 17 recites an abstract idea as inherited from claims 1 and 16.
Step 2A, Prong 2: Claim 17 recites the additional element of “wherein the operators comprise at least one of a rotation or a reflection.” These additional elements merely generally link the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 17 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the operators comprise at least one of a rotation or a reflection” merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 18,
Step 2A, Prong 1: Claim 18 recites an abstract idea as inherited from 1. Claim 18 further recites the additional limitation of:
generat[ing] at least one of a planet trajectory prediction, a robotic planning output, or a molecular modeling output. (i.e., a person can, with the aid of pen and paper, generate at least one of a planet trajectory prediction, a robotic planning output, or a molecular modeling output)
Hence, the claim recites an abstract idea.
Step 2A, Prong 2: Claim 18 recites the additional element of “wherein the geometric algebra transformer generates”, to perform the limitation stated above, is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components. (see MPEP 2106.05(f)). Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 18 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the geometric algebra transformer generates”, to perform the limitation stated above, is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components. (see MPEP 2106.05(f)). Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 19,
Step 1: Claim 19 is directed to a method.
Step 2A, Prong 1: Claim 1 recites the following limitations:
processing,… the multivector inputs… to generate multivector outputs (i.e., this limitation comprises mathematical calculations in view of paragraphs [0060]-[0062] of Applicants specification)
…apply a dot product that subsumes a geometric algebra inner product … (i.e., this limitation comprises mathematical calculations and concepts in view of paragraphs [0043]-[0047] of Applicant’s specification)
Hence, the claim recites an abstract idea.
Step 2A, Prong 2: Claim 19 recites the additional elements of “a processor-implemented method of operating a geometric algebra transformer, the processor-implemented method comprising:”, “via the geometric algebra transformer, …via at least a one normalization layer,”, “a geometric attention layer configured to…”, “at least one equilinear layer, and a scalar-gated nonlinearity layer, …”, and “wherein the geometric algebra transformer is trained to process geometric algebra representations associated with the multivector inputs and to be equivariant with respect to translations and rotations.” These additional elements are recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components. (see MPEP 2106.05(f)) Furthermore, claim 19 recites the additional element of “receive, at the geometric algebra transformer, multivector inputs processed from raw data associated with a three-dimensional space;”, which is considered insignificant extra-solution activity.(see MPEP 2106.05(g)) Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 19 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of of “a processor-implemented method of operating a geometric algebra transformer, the processor-implemented method comprising:”, “via the geometric algebra transformer, …via at least a one normalization layer,”, “a geometric attention layer configured to…”, “at least one equilinear layer, and a scalar-gated nonlinearity layer, …”, and “wherein the geometric algebra transformer is trained to process geometric algebra representations associated with the multivector inputs and to be equivariant with respect to translations and rotations” are recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components. (see MPEP 2106.05(f)). Furthermore, the additional element of “receive, at the geometric algebra transformer, multivector inputs processed from raw data associated with a three-dimensional space;” was determined to be insignificant extra-solution activity consisting of mere data transmission. (see MPEP 2106.05(g)) As such, it is re-evaluated in Step 2B to determine if it is more than what the courts have held as well-understood, routine, and conventional activity in the field. The court decisions cited in MPEP 2106.05(d)(II) have determined that mere data transmission (as it is presently claimed) is a well-understood, routine, and conventional activity in the field supported by Berkheimer. (see also, i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information). Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 20,
Step 1: Claim 20 is directed to an apparatus.
Step 2A, Prong 1: Claim 20 recites the following limitations
process,… , the multivector inputs to generate multivector outputs,.. (i.e., this limitation comprises mathematical calculations in view of paragraphs [0060]-[0062] of Applicant’s specification)
Step 2A, Prong 2: Claim 20 recites the additional elements of “an apparatus for processing data using a geometric algebra transformer, the apparatus comprising: at least one memory; and at least one processor coupled to at least one memory and configured to:”, “…via the geometric algebra transformer…”, and “wherein the geometric algebra transformer is trained to process geometric algebra representations associated with the multivector inputs and to be equivariant with respect to translations and rotations.”. These additional elements are recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components. (see MPEP 2106.05(f)). Furthermore, the claim recites the additional element of “receive, at the geometric algebra transformer, multivector inputs processed from raw data associated with a three-dimensional space;” which is considered insignificant extra-solution activity. (see MPEP 2106.05(g)) Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 20 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of “an apparatus for processing data using a geometric algebra transformer, the apparatus comprising: at least one memory; and at least one processor coupled to at least one memory and configured to:”, “…via the geometric algebra transformer…”, and “wherein the geometric algebra transformer is trained to process geometric algebra representations associated with the multivector inputs and to be equivariant with respect to translations and rotations” are recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components. (see MPEP 2106.05(f)). Furthermore, the additional element of “receive, at the geometric algebra transformer, multivector inputs processed from raw data associated with a three-dimensional space;” has been considered insignificant extra-solution activity. (see MPEP 2106.05(g)) As such, it is re-evaluated under Step 2B to determine if it is more than what the courts have held to be well-understood, routine, and conventional activity in the field. The court decisions cited in MPEP 2106.05(d)(II) have held that mere data transmission (as it is presently claimed) is a well-understood, routine, and conventional activity in the field supported by Berkheimer. (see also, i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information). Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 21,
Step 2A, Prong 1: Claim 21 recites an abstract idea as inherited from claim 20.
Step 2A, Prong 2: Claim 21 recites the additional elements of “wherein the multivector inputs comprise multi-component multivectors and wherein the multi-component multivectors comprise embedded geometric objects.” These additional elements merely generally link the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 21 not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of “wherein the multivector inputs comprise multi-component multivectors and wherein the multi-component multivectors comprise embedded geometric objects” merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 22,
Step 2A, Prong 1: Claim 22 recites an abstract idea as inherited from claim 20. Claim 22 further recites the following limitations:
wherein the embedded geometric objects are embedded into the multi-component multivectors…(i.e., this limitation comprises mathematical calculations in view of paragraphs [0060]-[0064] of Applicant’s specification)
Hence, the claim recites an abstract idea.
Step 2A, Prong 2: Claim 22 recites the additional element of “…using a geometric algebra embedding component”, which is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components. (see MPEP 2106.05(f)) Furthermore, the claim recites the additional element of “wherein the embedded geometric objects comprise at least one of a scalar, a vector, a bivector, a trivector, or a pseudoscalar”. This additional element merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 22 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “…using a geometric algebra embedding component”, is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components. (see MPEP 2106.05(f)) Furthermore, the additional element of “wherein the embedded geometric objects comprise at least one of a scalar, a vector, a bivector, a trivector, or a pseudoscalar” merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 23,
Step 2A, Prong 1: Claim 23 recites an abstract idea as inherited from claim 20.
Step 2A, Prong 2: Claim 23 recites the additional elements of “wherein the geometric algebra transformer further comprises: an input equilinear layer; a transformer block; and an output equilinear layer.” These additional elements merely generally link the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 23 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of “wherein the geometric algebra transformer further comprises: an input equilinear layer; a transformer block; and an output equilinear layer” merely generally link the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 24,
Step 2A, Prong 1: Claim 24 recites an abstract idea as inherited from claim 20.
Step 2A, Prong 2: Claim 24 recites the additional element of “wherein the geometric algebra transformer further comprises a plurality of transformer blocks.” This additional element merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 24 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the geometric algebra transformer further comprises a plurality of transformer blocks” merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 25,
Step 2A, Prong 1: Claim 25 recites an abstract idea as inherited from claim 20.
Step 2A, Prong 2: Claim 25 recites the additional elements of “wherein the transformer block further comprises: a first normalization layer; a first equilinear layer; a geometric attention layer; a first geometric product engine; a second equilinear layer; a first addition engine; a second normalization layer; a third equilinear layer; a second geometric product engine; a scalar-gated nonlinearity layer; a fourth equilinear layer; and a second addition engine.” These additional elements merely generally link the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 25 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of “wherein the transformer block further comprises: a first normalization layer; a first equilinear layer; a geometric attention layer; a first geometric product engine; a second equilinear layer; a first addition engine; a second normalization layer; a third equilinear layer; a second geometric product engine; a scalar-gated nonlinearity layer; a fourth equilinear layer; and a second addition engine” merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 26,
Step 2A, Prong 1: Claim 26 recites an abstract idea as inherited from claim 20.
Step 2A, Prong 2: Claim 26 recites the additional element of “wherein the scalar-gated nonlinearity layer comprises a scalar- gated Gaussian Error Linear Units nonlinearity layer”. This additional element merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 26 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the scalar-gated nonlinearity layer comprises a scalar- gated Gaussian Error Linear Units nonlinearity layer” merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 29,
Step 2A, Prong 1: Claim 29 recites an abstract idea as inherited from claim 20.
Step 2A, Prong 2: Claim 29 recites the additional elements of “wherein the first normalization layer, the first geometric product engine, the first addition engine, the second normalization layer, the second geometric product engine, the scalar-gated nonlinearity layer, and the second addition engine are fixed components and wherein the first equilinear layer, the geometric attention layer, the second equilinear layer, the third equilinear layer, and the fourth equilinear layer are learnable components.” These additional elements merely generally link the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 29 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of “wherein the first normalization layer, the first geometric product engine, the first addition engine, the second normalization layer, the second geometric product engine, the scalar-gated nonlinearity layer, and the second addition engine are fixed components and wherein the first equilinear layer, the geometric attention layer, the second equilinear layer, the third equilinear layer, and the fourth equilinear layer are learnable components” merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)) Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 30,
Step 1: Claim 30 is directed to an apparatus.
Step 2A, Prong 1: Claim 30 recites the following limitation:
process,… , the multivector inputs… to generate multivector outputs… (i.e., this limitation comprises mathematical calculations in view of paragraphs [0060]-[0062] of Applicant’s specification)
…apply a dot product that subsumes a geometric algebra inner product… (i.e., this limitation comprises mathematical calculations and concepts in view of paragraphs [0043]-[0047] of Applicant’s specification)
Hence, the claim recites an abstract idea.
Step 2A, Prong 2: Claim 30 recites the additional elements of “an apparatus for operating a geometric algebra transformer, the apparatus comprising: at least one memory; and at least one processor coupled to at least one memory and configured to:”, “…via the geometric algebra transformer…”, “via at least a one normalization layer,” “a geometric attention layer configured to”, “…, at least one equilinear layer, and a scalar-gated nonlinearity layer,” and “…, wherein the geometric algebra transformer is trained to process geometric algebra representations associated with the multivector inputs and to be equivariant with respect to translations and rotations.” These additional elements are recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components. (see MPEP 2106.05(f)) Furthermore, the claim recites the additional element of “receive, at a geometric algebra transformer, multivector inputs processed from raw data associated with a three-dimensional space;” which is considered insignificant extra-solution activity. (see MPEP 2106.05(g)). Hence the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 30 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of “an apparatus for operating a geometric algebra transformer, the apparatus comprising: at least one memory; and at least one processor coupled to at least one memory and configured to:”, “…via the geometric algebra transformer…”, “via at least a one normalization layer,” “a geometric attention layer configured to”, “…, at least one equilinear layer, and a scalar-gated nonlinearity layer,” and “…, wherein the geometric algebra transformer is trained to process geometric algebra representations associated with the multivector inputs and to be equivariant with respect to translations and rotations.” are recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components. (see MPEP 2106.05(f)) Furthermore, the additional element of “receive, at a geometric algebra transformer, multivector inputs processed from raw data associated with a three-dimensional space;” was determined to be insignificant extra-solution activity consisting of data transmission. (see MPEP 2106.05(g)) As such, it is re-evaluated under Step 2B to determine if it is more than what the courts have held as well-understood, routine, and conventional activity in the field. The court decisions cited in MPEP 2106.05(d)(II) have determined that mere data transmission (as it is presently claimed) is a well-understood, routine, and conventional activity in the field supported by Berkheimer. (see also, i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information). Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 2, 3, 4, 5, 14, 15, 16, 17, and 18 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Spellings, “Geometric Algebra Attention Networks for Small Point Clouds” (Oct. 3, 2022) in view of Zhao et al., “Point Transformer”, (2021)
Regarding claim 1, Spellings teaches a processor-implemented method of processing data using a geometric algebra transformer (Spellings, Abstract, teaches we present rotation- and permutation-equivariant architectures for deep learning on these small point clouds, composed of a set of products of terms from the geometric algebra and reductions over those products using an attention mechanism.; Spellings, pg. 4 further teaches machine learning approaches in the past have treated this problem using image-type data and learning an image translation. We treat this problem much like language translation problems using transformers,…), the processor-implemented method comprising:
receiving, at the geometric algebra transformer, multivector inputs processed from raw data associated with a three-dimensional space (Spellings, pg. 4, par. 3, teaches in this work, we use the geometric product to combine groups input vectors and systematically extract rotation-invariant quantities of interest for use in the network layers; Spellings, pg. 4, par. 1, teaches machine learning approaches in the past have treated this problem using simple multilayer perceptrons or by encoding the geometry of the problem as image-type data and learning an image translation process. We treat this problem much like language translation problems using transformers.; .; Spellings, pg. 17, par. 1, teaches the geometric algebra specifies a binary operator, the geometric product, that works on multivectors. Multivectors can be expressed as linear combinations of terms from a fixed basis set for a given space such as R3 [i.e., R3 as in three-dimensional space); in three-dimensional space, this yields scalars, vectors, bivectors (which specify signed areas within a plane and have 3 components), and trivectors (which specify signed volumes and have 1 components).; Spellings, Abstract, further teaches Often problems in the physical sciences deal with relatively small sets of points in two- or three-dimensional space wherein translation, rotation, and permutation equivariance are important or even vital for models to be useful in practice. In this work, we present rotation- and permutation-equivariant architectures for deep learning on these small point clouds, composed of a set of products of terms from the geometric algebra and reductions over those products using an attention mechanism.; Spellings, Abstract, teaches in this work, we present rotation- and permutation-equivariant architectures for deep learning on these small point clouds, composed of a set of products of terms from the geometric algebra and reductions over those products using an attention mechanism. The geometric algebra provides valuable mathematical structure by which to combine vector, scalar, and other types of geometric inputs in a systematic way to account for rotation invariance or covariance, while attention yields a powerful way to impose permutation equivariance. We demonstrate the usefulness of these architectures by training models to solve sample problems relevant to physics, chemistry, and biology.); and
processing, via the geometric algebra transformer, the multivector inputs to generate multivector outputs, wherein the geometric algebra transformer is trained to process geometric algebra representations associated with the multivector inputs and to be equivariant with respect to translations and rotations (Spellings, pg. 5, par. 6, teaches we first calculate the multivector geometric products of all combinations of input vectors indexed by I, j, k, and so on, up to a specified rank. We then use V, M, J, and S – together with a function extracting the rotation-invariant attributes of a geometric product (which are the scalar component, trivector component, ant the norms of the vector and bivector components, depending on how many input vectors are joined via the geometric product) into qi, k…- as follows for a network producing permutation-covariant outputs yi, for each input point – See equation (1); Spellings, pg. 4, par. 2 further teaches in this work we formulate deep neural networks using learnable functions consisting of two main parts: (1) a set of geometric products (from the geometric algebra in three spatial dimensions) of input vectors which encode geometric information in a rotation-equivariant manner; and (2) a permutation-equivariant reduction over these products using an attention mechanism.; Spellings, pg. 3, par. 2, further teaches although the approach described herein applies geometric algebra to train deep learning models on point clouds in a new way, using geometric algebra (also known as Clifford Algebra) to structure the operations of neural networks is not a novel concept…The approach we present here is similar to many of the ideas presented above; however, rather than specifying particular rotation-invariant quantities to utilize or learning maps that operate on irreducible representations, we leverage the structure provided by geometric algebra to calculate rotation-invariant and -covariant quantities of interest. Finally, we use an attention mechanism to attain permutation equivariance in a flexible manner.; Spellings, pg. 4, par. 1, teaches machine learning approaches in the past have treated this problem using simple multilayer perceptrons or by encoding the geometry of the problem as image-type data and learning an image translation process. We treat this problem much like language translation problems using transformers.).
Regarding claim 2, Spellings teaches all of the elements of claim 1, and Spellings further teaches wherein the multivector inputs comprise multi-component multivectors (Spellings, pg. 4, par. 3, teaches the primary objects death with using geometric algebra are multivectors, which consist of linear combinations of basis elements; in three dimensions, these basis elements are one scalar component, three vector components, three bivector components, and one trivector component.).
Regarding claim 3, Spellings teaches all of the limitations of claim 2, and Spellings further teaches wherein the multi-component multivectors comprise embedded geometric objects (Spellings, pg. 4, par. 2, teaches in this work we formulate deeop neural networks using learnable functions consisting of two main parts: (1) a set of geometric products (from the geometric algebra in three spatial dimensions) of input vectors which encode geometric information [i.e., as in embedded geometric information] in a rotation-equivariant manner; and (2) a permutation-equivariant reduction over these products using an attention mechanism.; Spellings, pg. 4, par. 3 further teaches geometric algebra provides mathematical structure to deal with geometric objects – such as point and planes – using a common language for arbitrary numbers of spatial dimensions. The primary objects dealt with using geometric algebra are multivectors, which consist of linear combinations of basis elements; in three dimensions, these basis elements are one scalar component, three vector components, three bivector components, and one trivector component.[i.e., as in multi-component multivectors comprising encoded geometric information – understood to read on the limitation as claimed]).
Regarding claim 4, Spellings teaches all of the limitations of claim 3, and Spellings further teaches wherein the embedded geometric objects are embedded into the multi-component multivectors using a geometric algebra embedding component (Spellings, pg. 4, par. 2, teaches in this work we formulate deeop neural networks using learnable functions consisting of two main parts: (1) a set of geometric products (from the geometric algebra in three spatial dimensions) of input vectors which encode geometric information [i.e., as in embedded geometric information] in a rotation-equivariant manner; and (2) a permutation-equivariant reduction over these products using an attention mechanism.; Spellings, pg. 4, par. 3 further teaches geometric algebra provides mathematical structure to deal with geometric objects – such as point and planes – using a common language for arbitrary numbers of spatial dimensions. The primary objects dealt with using geometric algebra are multivectors, which consist of linear combinations of basis elements; in three dimensions, these basis elements are one scalar component, three vector components, three bivector components, and one trivector component.).
Regarding claim 5, Spellings teaches all of the limitations of claim 3, and Spellings further teaches wherein the embedded geometric objects comprise at least one of a scalar, a vector, a bivector, a trivector, or a pseudoscalar (Spellings, pg. 4, par. 2, teaches in this work we formulate deeop neural networks using learnable functions consisting of two main parts: (1) a set of geometric products (from the geometric algebra in three spatial dimensions) of input vectors which encode geometric information [i.e., as in embedded geometric information] in a rotation-equivariant manner; and (2) a permutation-equivariant reduction over these products using an attention mechanism.; Spellings, pg. 4, par. 3 further teaches geometric algebra provides mathematical structure to deal with geometric objects – such as point and planes – using a common language for arbitrary numbers of spatial dimensions. The primary objects dealt with using geometric algebra are multivectors, which consist of linear combinations of basis elements; in three dimensions, these basis elements are one scalar component, three vector components, three bivector components, and one trivector component.).
Regarding claim 14, Spellings teaches all of the limitations of claim 1, and Spellings further teaches wherein the multivector inputs comprise at least one of a scalar value, a plane with a normal value, a line with a direction value, a point value, a pseudoscalar value, a reflection value through a plane with a normal value, a translation value, a rotation value, or a point reflection value and wherein the geometric algebra transformer represents both geometric objects and transformations of the geometric objects via use of the multivector inputs (Spellings, pg. 4 teaches The primary objects dealt with using geometric algebra are multivectors, which consist of linear combinations of basis elements; in three dimensions, these basis elements are one scalar component, three vector components, three bivector components, and one trivector component. We can calculate the so-called geometric product of two multivectors to yield a new multivector; for example, the geometric product of two vectors yields a scalar (that is the dot product of the vectors) plus a bivector (related to the cross product of the vectors). The geometric product is not commutative; for two general multivectors A and B, we denote the product C as C = AB. In this work, we use the geometric product to combine groups of input vectors and systematically extract rotation-invariant quantities of interest—such as distances, bond angles, and volumes—for use in network layers.).
Regarding claim 15, Spellings teaches all of the limitations of claim 1, and Spellings further teaches wherein the multivector inputs uniquely represent various geometric types (Spellings, Abstract, teaches the geometric algebra provides valuable mathematical structure by which to combine vector, scalar, and other types of geometric inputs in a systematic way to account for rotation invariance or covariance, while attention yields a powerful way to impose permutation equivariance; Spellings, pg. 9 further teaches We find the architectures formulated here to be useful for a variety of tasks. Rather than being limited to operating on bond distances and angles as in SchNet [Schütt et al., 2017], PhysNet [Unke and Meuwly,2019], and DimeNet [Klicpera et al., 2019], geometric algebra provides a systematic way to build functions
with the desired rotation- and permutation-equivariance, with the flexibility to incorporate other types of geometric objects (such as the orientation quaternion commonly used for anisotropic particles in molecular dynamics methods [Kamberaj et al., 2005]) into the framework.).
Regarding claim 16, Spellings teaches all of the limitations of claim 1, and Spellings further teaches the multivector inputs are generated from a geometric product of vectors and wherein the multivector inputs comprises a representation of geometric objects and operators associated with the geometric objects (We begin with a description of the bare essentials of geometric algebra used in this work. For more details, we refer the reader to Appendix A. Briefly, geometric algebra provides mathematical structure to deal with geometric objects—such as points and planes—using a common language for arbitrary numbers of spatial dimensions. The primary objects dealt with using geometric algebra are multivectors, which consist of linear
combinations of basis elements; in three dimensions, these basis elements are one scalar component, three vector components, three bivector components, and one trivector component. We can calculate the so-called geometric product of two multivectors to yield a new multivector; for example, the geometric product of two
vectors yields a scalar (that is the dot product of the vectors) plus a bivector (related to the cross product of the vectors). The geometric product is not commutative; for two general multivectors A and B, we denote the product C as C = AB. In this work, we use the geometric product to combine groups of input vectors and systematically extract rotation-invariant quantities of interest—such as distances, bond angles, and
volumes—for use in network layers.).
Regarding claim 17, Spellings teaches all of the limitations of claim 16, and Spellings further teaches wherein the operators comprise at least one of a rotation or a reflection (Spellings, pgs. 1-2 teaches, we are interested in modeling attributes of the system—such as the identity of a particle’s local self-assembly environment, or the potential energy of a group of atoms—which are invariant with respect to rotation of the input coordinates, as well as permutation in the ordering of points. As shown in Figure 1, here we attain rotation invariance).
Regarding claim 18, Spellings teaches all of the limitations of claim 1, and Spellings further teaches wherein the geometric algebra transformer generates at least one of a planet trajectory prediction, a robotic planning output, or a molecular modeling output (Spellings, pg. 3, teaches here we train models using our attention mechanism to predict the atomic forces calculated using ab initio molecular dynamics and density functional theory [Chmiela et al., 2017]. These models are conservative, permutation-invariant, and rotation-equivariant by construction—all crucial attributes when using models in simulation.).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 6, 7, 8, 13, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Spellings in view of Zhao et al., “Point Transformer”, (2021)
Regarding claim 6, Spellings teaches all of the limitations of claim 1, however, Spellings does not distinctly disclose wherein the geometric algebra transformer further comprises: an input equilinear layer; a transformer block; and an output equilinear layer.
Nevertheless, Zhao teaches wherein the geometric algebra transformer further comprises: an input equilinear layer; a transformer block; and an output equilinear layer (Zhao Figure 2 teaches point transformer layer; and Zhao Figure 4 teaching point transformer block comprising an input linear layer, a point transformer layer and an output linear layer; Zhao, Section 3.5 further teaching we construct complete 3D point cloud understanding networks based on the point transformer block…The network architectures are visualized in Figure 3).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the geometric algebra transformer, as taught by Spellings, with the Point Transformer features, as taught by Zhao, to provide a more natural fit for point cloud processing with the developed transformer architecture for 3D point clouds.
Regarding claim 7, the combination of Spellings in view of Zhao teaches all of the limitations of claim 6, and the combination further teaches wherein the geometric algebra transformer further comprises a plurality of transformer blocks (Zhao, Figure 3 illustrates a plurality of point transformer blocks).
Motivation to combine same as stated for claim 6.
Regarding claim 8, the combination of Spellings in view of Zhao teaches all of the limitations of claim 6, and Spellings teaches a first geometric product engine and a second geometric product engine (Spellings, pg. 4, par. 3 teaches in this work we use the geometric product to combine groups of input vectors and systematically extract rotation-invariant quantities of interest – such as distances, bond angles, and volumes – for use in network layers.)
and Zhao teaches: wherein the transformer block further comprises: a first normalization layer; a first equilinear layer; (Zhao, pg. 16263, Col. 1, par. 2, teaches each input feature goes through a linear transformation, followed by batch normalization.;) a geometric attention layer; (Zhao, Section 3.4 further teaches the transformer block integrates a self-attention layer;) …; a second equilinear layer; (Zhao, Figure 2 teaches first and second equilinear layers; Figure 4, teaches first and second linear layers) a first addition engine (Zhao, Figure 2 and Section 3.5 teaches feature aggregation layer as part of the point transformer layer;); a second normalization layer; a third equilinear layer; (Zhao, Figure 3 teaches plurality of point transformer blocks each with a normalization layer as stated above and an equilinear layer as stated above) … a scalar-gated nonlinearity layer; a fourth equilinear layer; ; (Zhao, Figure 3 teaches plurality of point transformer blocks each with a normalization layer as stated above and an equilinear layer as stated above; Zhao, Section 3.3 teaches the encoding function θ is an MLP with two linear layers and one ReLu nonlinearity) and a second addition engine (Zhao, Figure 2 and Section 3.5 teaches feature aggregation layer as part of the point transformer layer; Zhao Figure 3 teaches plurality of point transformer blocks each with a feature aggregation layer).
Motivation to combine same as stated in claim 6.
Regarding claim 13, the combination of Spellings in view of Zhao teaches all of the limitations of claim 8, and the combination further teaches wherein each layer maps between multivector data and is equivariant (Spellings, pg. 4, teaches The primary objects dealt with using geometric algebra are multivectors, which consist of linear combinations of basis elements; in three dimensions, these basis elements are one scalar component, three vector components, three bivector components, and one trivector component. We can calculate the so-called geometric product of two multivectors to yield a new multivector; in this work, we use the geometric product to combine groups of input vectors and systematically extract rotation-invariant quantities of interest—such as distances, bond angles, and volumes—for use in network layers.; Spellings, Conclusion, further teaches In this work, we have presented a strategy for developing rotation- and permutation-equivariant neural network architectures by combining geometric algebra and attention mechanisms. These architectures operate directly on vector, scalar, and other geometric quantities of interest to produce outputs which respect desirable symmetries by construction. We believe that the mathematical simplicity and the insights derived from inspecting attention maps are particularly appealing aspects of the algorithms presented here.).
Regarding claim 19, Spellings teaches a processor-implemented method of operating a geometric algebra transformer, the processor-implemented method comprising:
receiving, at a geometric algebra transformer, multivector inputs processed from raw data associated with a three-dimensional space (Spellings, pg. 4, par. 3, teaches in this work, we use the geometric product to combine groups input vectors and systematically extract rotation-invariant quantities of interest for use in the network layers; Spellings, pg. 4, par. 1, teaches machine learning approaches in the past have treated this problem using simple multilayer perceptrons or by encoding the geometry of the problem as image-type data and learning an image translation process. We treat this problem much like language translation problems using transformers.; Spellings, pg. 17, par. 1, teaches the geometric algebra specifies a binary operator, the geometric product, that works on multivectors. Multivectors can be expressed as linear combinations of terms from a fixed basis set for a given space such as R3 [i.e., R3 as in three-dimensional space); in three-dimensional space, this yields scalars, vectors, bivectors (which specify signed areas within a plane and have 3 components), and trivectors (which specify signed volumes and have 1 components); Spellings, Abstract, further teaches often problems in the physical sciences deal with relatively small sets of points in two- or three-dimensional space wherein translation, rotation, and permutation equivariance are important or even vital for models to be useful in practice. In this work, we present rotation- and permutation-equivariant architectures for deep learning on these small point clouds, composed of a set of products of terms from the geometric algebra and reductions over those products using an attention mechanism.); and
processing, via the geometric algebra transformer, the multivector inputs via at least a one normalization layer, a geometric attention layer configured to apply a dot product that subsumes a geometric algebra inner product (Spellings, pg. 4 – Attention from Geometric Products- teaches we can calculate the so-called geometric product of two multivectors to yield a new multivector; for example, the geometric product of two vectors yields a scalar (that is the dot product of the vectors) plus a bivector (related to the cross product of the vectors). The geometric product is not commutative; for two general multivectors A and B, we denote the product C as C = AB. In this work, we use the geometric product to combine groups of input vectors and systematically extract rotation-invariant quantities of interest—such as distances, bond angles, and volumes—for use in network layers.; Spellings, pg. 6, further teaches we demonstrate the utility of our geometric algebra attention scheme by training deep networks to solve three problems appearing in physics, chemistry, and biology. We emphasize that the architectures and hyperparameters presented here are primarily intended to demonstrate how attention layers can be composed and physical constraints can be imposed via architecture (such as creating conservative force fields for molecular force regression and controlling the rotation- and permutation-equivariance of models), rather than to present highly-optimized architectures and training hyperparameters for any given dataset. For simplicity, all of the geometric algebra attention models presented here utilize pairwise attention with a working dimension of 32 units. Value functions V, score functions S, and rescaling functions R are simple multilayer perceptrons with a hidden width of 64 units, with layer normalization applied to the hidden layer of V.; ),
wherein the geometric algebra transformer is trained to process geometric algebra representations associated with the multivector inputs and to be equivariant with respect to translations and rotations (Spellings, pg. 5, par. 6, teaches we first calculate the multivector geometric products of all combinations of input vectors indexed by I, j, k, and so on, up to a specified rank. We then use V, M, J, and S – together with a function extracting the rotation-invariant attributes of a geometric product (which are the scalar component, trivector component, ant the norms of the vector and bivector components, depending on how many input vectors are joined via the geometric product) into qi, k…- as follows for a network producing permutation-covariant outputs yi, for each input point – See equation (1); Spellings, pg. 4, par. 2 further teaches in this work we formulate deep neural networks using learnable functions consisting of two main parts: (1) a set of geometric products (from the geometric algebra in three spatial dimensions) of input vectors which encode geometric information in a rotation-equivariant manner; and (2) a permutation-equivariant reduction over these products using an attention mechanism.; Spellings, pg. 3, par. 2, further teaches although the approach described herein applies geometric algebra to train deep learning models on point clouds in a new way, using geometric algebra (also known as Clifford Algebra) to structure the operations of neural networks is not a novel concept…The approach we present here is similar to many of the ideas presented above; however, rather than specifying particular rotation-invariant quantities to utilize or learning maps that operate on irreducible representations, we leverage the structure provided by geometric algebra to calculate rotation-invariant and -covariant quantities of interest. Finally, we use an attention mechanism to attain permutation equivariance in a flexible manner.; Spellings, pg. 4, par. 1, teaches machine learning approaches in the past have treated this problem using simple multilayer perceptrons or by encoding the geometry of the problem as image-type data and learning an image translation process. We treat this problem much like language translation problems using transformers.).
However, Spellings does not distinctly disclose:
at least one equilinear layer, and a scalar-gated nonlinearity layer, to generate multivector outputs, …
Nevertheless, Zhao teaches at least one equilinear layer, and a scalar-gated nonlinearity layer, to generate multivector outputs, … (Zhao Figure 2 teaches point transformer layer; and Zhao Figure 4 teaching point transformer block comprising an input linear layer, a point transformer layer and an output linear layer; Zhao, Section 3.5 further teaching we construct complete 3D point cloud understanding networks based on the point transformer block…The network architectures are visualized in Figure 3. Zhao, Figure 3 teaches plurality of point transformer blocks each with a normalization layer and an equilinear layer; Zhao, Section 3.3 teaches the encoding function θ is an MLP with two linear layers and one ReLu nonlinearity; Zhao Section 3.4 further teaches We construct a residual point transformer block with the point transformer layer at its core, as shown in Figure 4(a). The transformer block integrates the self-attention layer, linear projections that can reduce dimensionality and accelerate processing, and a residual connection. The input is a set of feature vectors x with associated 3D coordinates p. The point transformer block facilitates information exchange between these localized feature vectors, producing new feature vectors for all data points as its output. [Note: producing multivector outputs, as claimed] The information aggregation adapts both to the content of the feature vectors and their layout in 3D.)
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the geometric algebra transformer, as taught by Spellings, with the Point Transformer features, as taught by Zhao, to provide a more natural fit for point cloud processing with the developed transformer architecture for 3D point clouds.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Spellings in view of Zhao et al., as applied to claim 8, and further in view of Li et al., “A high speed reconfigurable architecture for softmax and GELU in vision transformer”, (March 2023)
Regarding claim 9, the combination of Spellings in view of Zhao teaches all of the limitations of claim 8, however the combination does not distinctly disclose wherein the scalar-gated nonlinearity layer comprises a scalar-gated Gaussian Error Linear Units nonlinearity layer.
Nevertheless, Li teaches wherein the scalar-gated nonlinearity layer comprises a scalar-gated Gaussian Error Linear Units nonlinearity layer (Li, Abstract, teaches transformers have been widely used in various computer vision applications. Compared to traditional convolutional neural networks (CNNs), transformer’s inference includes plenty of non-linear operations, such as softmax and Gaussian error linear units (GELU). As the scale of transformers grows, an efficient hardware implementation of these operations is significant.; Li, pg. 1, col. 1, Introduction teaches GELU is a common activation function of Transformers and will be used many times during the processing of ViT. Therefore, softmax and GELU operations make up a large portion of the ViT runtime, as shown in Fig. 1).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the geometric algebra transformer, as taught by Spellings in view of Zhao, to further include the Gaussian error linear units, as taught by Li, as it is the most commonly used activation function in transformers. (Li, Abstract and Li pg. 1, col. 1)
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Spellings in view of Zhao, as applied to claim 8, and further in view of Kaul et al., “CpT: Convolutional Point Transformer for 3D Point Cloud Processing (21 Nov. 2021)
Regarding claim 12, the combination of Spellings in view of Zhao teaches all of the limitations of claim 8, however the combination does not distinctly disclose wherein the first normalization layer, the first geometric product engine, the first addition engine, the second normalization layer, the second geometric product engine, the scalar-gated nonlinearity layer, and the second addition engine are fixed components and wherein the first equilinear layer, the geometric attention layer, the second equilinear layer, the third equilinear layer, and the fourth equilinear layer are learnable components.
Nevertheless, Kaul teaches wherein the first normalization layer, the first geometric product engine, the first addition engine, the second normalization layer, the second geometric product engine, the scalar-gated nonlinearity layer, and the second addition engine are fixed components and wherein the first equilinear layer, the geometric attention layer, the second equilinear layer, the third equilinear layer, and the fourth equilinear layer are learnable components (Kaul, pg. 3, col. 2, teaches the The overall architecture of the Convolutional point Transformer (CpT) is shown in Figure 1. Our main contributions are the Point Embedding Module (Section 3.1) and the InterPoint Attention Module with a convolutional atten tion projection (Section 3.2). When an input point cloud of size N × 3 is passed through the architecture, a graph of the points is computed via finding its K-Nearest Neighbors based on Euclidean distance. This representation is then passed through a point embedding layer that maps the input data into a representation implicitly inclusive of the nearest neighbours of the points. This is done via a 2D convolution operation whose degree of overlap across points can be controlled through the length of the stride. A dot product attention operation [i.e., understood as a fixed geometric product engine] is then applied to this embedded representation which is followed by an InterPoint Attention Module. The dot product attention can be seen as learning relevant features of a points embedding as a function of it’s K nearest neighbors [i.e. a learning layer]. Such an attention mechanism learns to attend to the features of the points rather than the points themselves (column-wise matrix attention), i.e. for a set of points in a batch, it learns to weight individual feature transformations. The InterPoint Attention on the other hand can be interpreted as learning the relationships between different the points themselves, within a batch (a row-wise ma trix attention operating per point embedding, rather than per individual feature of the points). This forms one layer of the CpT.[Note: the InterPoint Attention layer understood as a “learnable” geometric attention layer] We update the graph following the InterPoint Attention operation which is then passed into the next CpT layer. A third and final CpT layer with a higher feature space embedding dimension is used without InterPoint Attention to learn the representation of the point cloud. InterPoint attention in the deeper layers does not provide a lot of context, as transformers eventually can learn relations that cross beyond the locality of their point set em bedding dimensions. This also means that transformers are able to visualize inputs beyond their limited receptive field in the deeper layers, making them a better choice for feature space embedding compared to shared-weighted MLPs (1D Convolutions). Adding this attention block to the deeper layers only marginally improves performance but significantly increases computation. The output of each transformer layer is then concatenated and passed through a final shared-weighted MLP. This is then Global Max-Pooled to get a global feature vector which forms the representation of the point cloud [i.e., understood as a fixed scalar-gated nonlinearity layer]. This representation is then processed further through a series of MLPs to classify the point cloud, or obtain a semantic point label for each point in the point cloud. Residual connections, layer normalization, and addition and MLP layers are used after the attention layers as in conventional transformer model structures [i.e., Residual connections, layer normalization, and addition and MLP layers understood as fixed components].
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the geometric algebra transformer, as taught by Spellings in view of Zhao, to further include the features of the convolutional point transformer, as taught by Kaul. The added InterPoint Attention Module to the Transformer learns to enhance the output by learning to relate each point in the input to every other point. This helps capture better geometric relationships between the points and aids in better learning of the local and global concepts in the data. The resultant transformer block, i.e., the CpT layer (shown in Figure 2) is a combination of a dot product attention operation, followed by an InterPoint Attention module. (Kaul, pg. 2, col. 1)
Claims 20, 21, and 22 is rejected under 35 U.S.C. 103 as being unpatentable over Spellings et al. in view of Kranski et al. (US 20220314434 A1, filed Apr. 1, 2022 and published Oct. 6, 2022)
Regarding claim 20, Spellings teaches:
…processing data using a geometric algebra transformer…(Spellings, pg. 4 further teaches machine learning approaches in the past have treated this problem using image-type data and learning an image translation. We treat this problem much like language translation problems using transformers,…)
receive, at the geometric algebra transformer, multivector inputs processed from raw data associated with a three-dimensional space (Spellings, pg. 4, par. 3, teaches in this work, we use the geometric product to combine groups input vectors and systematically extract rotation-invariant quantities of interest for use in the network layers; Spellings, pg. 4, par. 1, teaches machine learning approaches in the past have treated this problem using simple multilayer perceptrons or by encoding the geometry of the problem as image-type data and learning an image translation process. We treat this problem much like language translation problems using transformers.; Spellings, Abstract, teaches in this work, we present rotation- and permutation-equivariant architectures for deep learning on these small point clouds, composed of a set of products of terms from the geometric algebra and reductions over those products using an attention mechanism. The geometric algebra provides valuable mathematical structure by which to combine vector, scalar, and other types of geometric inputs in a systematic way to account for rotation invariance or covariance, while attention yields a powerful way to impose permutation equivariance. We demonstrate the usefulness of these architectures by training models to solve sample problems relevant to physics, chemistry, and biology.); and
process, via the geometric algebra transformer, the multivector inputs to generate multivector outputs, wherein the geometric algebra transformer is trained to process geometric algebra representations associated with the multivector inputs and to be equivariant with respect to translations and rotations (Spellings, pg. 5, par. 6, teaches we first calculate the multivector geometric products of all combinations of input vectors indexed by I, j, k, and so on, up to a specified rank. We then use V, M, J, and S – together with a function extracting the rotation-invariant attributes of a geometric product (which are the scalar component, trivector component, ant the norms of the vector and bivector components, depending on how many input vectors are joined via the geometric product) into qi, k…- as follows for a network producing permutation-covariant outputs yi, for each input point – See equation (1); Spellings, pg. 4, par. 2 further teaches in this work we formulate deep neural networks using learnable functions consisting of two main parts: (1) a set of geometric products (from the geometric algebra in three spatial dimensions) of input vectors which encode geometric information in a rotation-equivariant manner; and (2) a permutation-equivariant reduction over these products using an attention mechanism.; Spellings, pg. 3, par. 2, further teaches although the approach described herein applies geometric algebra to train deep learning models on point clouds in a new way, using geometric algebra (also known as Clifford Algebra) to structure the operations of neural networks is not a novel concept…The approach we present here is similar to many of the ideas presented above; however, rather than specifying particular rotation-invariant quantities to utilize or learning maps that operate on irreducible representations, we leverage the structure provided by geometric algebra to calculate rotation-invariant and -covariant quantities of interest. Finally, we use an attention mechanism to attain permutation equivariance in a flexible manner.; Spellings, pg. 4, par. 1, teaches machine learning approaches in the past have treated this problem using simple multilayer perceptrons or by encoding the geometry of the problem as image-type data and learning an image translation process. We treat this problem much like language translation problems using transformers.).
However, Spellings does not distinctly disclose an apparatus for processing data… comprising: at least one memory; and at least one processor coupled to at least one memory…
Nevertheless, Kranski teaches an apparatus for processing data … comprising: at least one memory; and at least one processor coupled to at least one memory… (Kranski, [0032] teaches some embodiments of robot systems 102 may include an even more expansive ensemble of control models 116. For example, a machine learning subsystem 114 may pipeline a convolutional neural network (or vision transformer) that extracts features from 2D image data, a geometric deep learning model that extracts features from 3D point clouds from depth sensors, and an encoder model that maps both sets of those features for a given time slice into respective vectors in latent embedding spaces, and a reinforcement learning model that controls the robot; Kranski, [0033] teaches in some examples, one or more of a convolutional neural network, vision transformer, or geometric deep learning model may be implemented with a hardware ML Accelerator, such as in addition to a downstream encoder model implemented with a hardware ML Accelerator. Embodiments are not limited to only one ML model (or type) or a specific collection of ML models, which is not to suggest that any other description is limiting.; Kranski, [0149] FIG. 5 is a physical architecture block diagram that shows an example of a computing device (or data processing system) by which some aspects of the above techniques may be implemented. Various portions of systems and methods described herein, may include or be executed on one or more computer systems similar to computing system 1000. Further, processes and modules described herein may be executed by one or more processing systems similar to that of computing system 1000.; Kranski [0150] further teaches Computing system 1000 may include one or more processors (e.g., processors 1010a-1010n) coupled to system memory 1020… A processor may include a central processing unit (CPU) that carries out program instructions to perform the arithmetical, logical, and input/output operations of computing system 1000.)
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the geometric algebra transformer, as taught by Spellings, with the computing system of Kranski, given that performance of robots and other controlled dynamic mechanical systems is constrained by computing resources and particularly those used to implement machine learning techniques. (Kranski, [0018])
Regarding claim 21, the combination of Spellings in view of Kranski teaches all of the limitations of claim 20, and the combination further teaches wherein the multivector inputs comprise multi-component multivectors (Spellings, pg. 4, par. 3, teaches the primary objects death with using geometric algebra are multivectors, which consist of linear combinations of basis elements; in three dimensions, these basis elements are one scalar component, three vector components, three bivector components, and one trivector component.) and wherein the multi-component multivectors comprise embedded geometric objects (Spellings, pg. 4, par. 2, teaches in this work we formulate deeop neural networks using learnable functions consisting of two main parts: (1) a set of geometric products (from the geometric algebra in three spatial dimensions) of input vectors which encode geometric information [i.e., as in embedded geometric information] in a rotation-equivariant manner; and (2) a permutation-equivariant reduction over these products using an attention mechanism.; Spellings, pg. 4, par. 3 further teaches geometric algebra provides mathematical structure to deal with geometric objects – such as point and planes – using a common language for arbitrary numbers of spatial dimensions. The primary objects dealt with using geometric algebra are multivectors, which consist of linear combinations of basis elements; in three dimensions, these basis elements are one scalar component, three vector components, three bivector components, and one trivector component.[i.e., as in multi-component multivectors comprising encoded geometric information – understood to read on the limitation as claimed]).
Regarding claim 22, the combination of Spellings in view of Kranski teaches all of the limitations of claim 21, and the combination further teaches wherein the embedded geometric objects are embedded into the multi-component multivectors using a geometric algebra embedding component and wherein the embedded geometric objects comprise at least one of a scalar, a vector, a bivector, a trivector, or a pseudoscalar (Spellings, pg. 4, par. 2, teaches in this work we formulate deep neural networks using learnable functions consisting of two main parts: (1) a set of geometric products (from the geometric algebra in three spatial dimensions) of input vectors which encode geometric information [i.e., as in embedded geometric information] in a rotation-equivariant manner; and (2) a permutation-equivariant reduction over these products using an attention mechanism.; Spellings, pg. 4, par. 3 further teaches geometric algebra provides mathematical structure to deal with geometric objects – such as point and planes – using a common language for arbitrary numbers of spatial dimensions. The primary objects dealt with using geometric algebra are multivectors, which consist of linear combinations of basis elements; in three dimensions, these basis elements are one scalar component, three vector components, three bivector components, and one trivector component.).
Claims 23, 24, 25, and 30 are rejected under 35 U.S.C. 103 as being unpatentable over Spellings et al. in view of Kranski et al. , as applied to claim 20, and further in view of Zhao et al.
Regarding claim 23, the combination of Spellings in view of Kranski teaches all of the limitations of claim 20, however the combination does not distinctly disclose wherein the geometric algebra transformer further comprises: an input equilinear layer; a transformer block; and an output equilinear layer.
Nevertheless, Zhao teaches wherein the geometric algebra transformer further comprises: an input equilinear layer; a transformer block; and an output equilinear layer (Zhao Figure 2 teaches point transformer layer; and Zhao Figure 4 teaching point transformer block comprising an input linear layer, a point transformer layer and an output linear layer; Zhao, Section 3.5 further teaching we construct complete 3D point cloud understanding networks based on the point transformer block…The network architectures are visualized in Figure 3).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the geometric algebra transformer, as taught by Spellings in view of Kranski, with the Point Transformer features, as taught by Zhao, to provide a more natural fit for point cloud processing with the developed transformer architecture for 3D point clouds.
Regarding claim 24, the combination of Spellings in view of Kranski and Zhao teaches all of the limitations of claim 23, and the combination further teaches wherein the geometric algebra transformer further comprises a plurality of transformer blocks (Zhao, Figure 3 illustrates a plurality of point transformer blocks).
Motivation to combine same as stated for claim 23.
Regarding claim 25, the combination of Spellings in view of Kranski and Zhao teaches all of the limitations of claim 23, and Spellings further teaches a first geometric product engine and a second geometric product engine (Spellings, pg. 4, par. 3 teaches in this work we use the geometric product to combine groups of input vectors and systematically extract rotation-invariant quantities of interest – such as distances, bond angles, and volumes – for use in network layers.)
and Zhao further teaches: wherein the transformer block further comprises: a first normalization layer; a first equilinear layer; (Zhao, pg. 16263, Col. 1, par. 2, teaches each input feature goes through a linear transformation, followed by batch normalization.;) a geometric attention layer; (Zhao, Section 3.4 further teaches the transformer block integrates a self-attention layer;) …; a second equilinear layer; (Zhao, Figure 2 teaches first and second equilinear layers; Figure 4, teaches first and second linear layers) a first addition engine (Zhao, Figure 2 and Section 3.5 teaches feature aggregation layer as part of the point transformer layer;); a second normalization layer; a third equilinear layer; (Zhao, Figure 3 teaches plurality of point transformer blocks each with a normalization layer as stated above and an equilinear layer as stated above) … a scalar-gated nonlinearity layer; a fourth equilinear layer; ; (Zhao, Figure 3 teaches plurality of point transformer blocks each with a normalization layer as stated above and an equilinear layer as stated above; Zhao, Section 3.3 teaches the encoding function θ is an MLP with two linear layers and one ReLu nonlinearity) and a second addition engine (Zhao, Figure 2 and Section 3.5 teaches feature aggregation layer as part of the point transformer layer; Zhao Figure 3 teaches plurality of point transformer blocks each with a feature aggregation layer).
Motivation to combine same as stated in claim 23.
Regarding claim 30, Spellings teaches:
an apparatus for operating a geometric algebra transformer (Spellings, pg. 4 further teaches machine learning approaches in the past have treated this problem using image-type data and learning an image translation. We treat this problem much like language translation problems using transformers,…)
receive, at a geometric algebra transformer, multivector inputs processed from raw data associated with a three-dimensional space (Spellings, pg. 4, par. 3, teaches in this work, we use the geometric product to combine groups input vectors and systematically extract rotation-invariant quantities of interest for use in the network layers; Spellings, pg. 4, par. 1, teaches machine learning approaches in the past have treated this problem using simple multilayer perceptrons or by encoding the geometry of the problem as image-type data and learning an image translation process. We treat this problem much like language translation problems using transformers.; Spellings, pg. 17, par. 1, teaches the geometric algebra specifies a binary operator, the geometric product, that works on multivectors. Multivectors can be expressed as linear combinations of terms from a fixed basis set for a given space such as R3 [i.e., R3 as in three-dimensional space); in three-dimensional space, this yields scalars, vectors, bivectors (which specify signed areas within a plane and have 3 components), and trivectors (which specify signed volumes and have 1 components); Spellings, Abstract, further teaches often problems in the physical sciences deal with relatively small sets of points in two- or three-dimensional space wherein translation, rotation, and permutation equivariance are important or even vital for models to be useful in practice. In this work, we present rotation- and permutation-equivariant architectures for deep learning on these small point clouds, composed of a set of products of terms from the geometric algebra and reductions over those products using an attention mechanism.);
and process, via the geometric algebra transformer, the multivector inputs via at least a one normalization layer, a geometric attention layer configured to apply a dot product that subsumes a geometric algebra inner product,… (Spellings, pg. 4 – Attention from Geometric Products- teaches we can calculate the so-called geometric product of two multivectors to yield a new multivector; for example, the geometric product of two vectors yields a scalar (that is the dot product of the vectors) plus a bivector (related to the cross product of the vectors). The geometric product is not commutative; for two general multivectors A and B, we denote the product C as C = AB. In this work, we use the geometric product to combine groups of input vectors and systematically extract rotation-invariant quantities of interest—such as distances, bond angles, and volumes—for use in network layers.; Spellings, pg. 6, further teaches we demonstrate the utility of our geometric algebra attention scheme by training deep networks to solve three problems appearing in physics, chemistry, and biology. We emphasize that the architectures and hyperparameters presented here are primarily intended to demonstrate how attention layers can be composed and physical constraints can be imposed via architecture (such as creating conservative force fields for molecular force regression and controlling the rotation- and permutation-equivariance of models), rather than to present highly-optimized architectures and training hyperparameters for any given dataset. For simplicity, all of the geometric algebra attention models presented here utilize pairwise attention with a working dimension of 32 units. Value functions V, score functions S, and rescaling functions R are simple multilayer perceptrons with a hidden width of 64 units, with layer normalization applied to the hidden layer of V.; )
wherein the geometric algebra transformer is trained to process geometric algebra representations associated with the multivector inputs and to be equivariant with respect to translations and rotations (Spellings, pg. 5, par. 6, teaches we first calculate the multivector geometric products of all combinations of input vectors indexed by I, j, k, and so on, up to a specified rank. We then use V, M, J, and S – together with a function extracting the rotation-invariant attributes of a geometric product (which are the scalar component, trivector component, ant the norms of the vector and bivector components, depending on how many input vectors are joined via the geometric product) into qi, k…- as follows for a network producing permutation-covariant outputs yi, for each input point – See equation (1); Spellings, pg. 4, par. 2 further teaches in this work we formulate deep neural networks using learnable functions consisting of two main parts: (1) a set of geometric products (from the geometric algebra in three spatial dimensions) of input vectors which encode geometric information in a rotation-equivariant manner; and (2) a permutation-equivariant reduction over these products using an attention mechanism.; Spellings, pg. 3, par. 2, further teaches although the approach described herein applies geometric algebra to train deep learning models on point clouds in a new way, using geometric algebra (also known as Clifford Algebra) to structure the operations of neural networks is not a novel concept…The approach we present here is similar to many of the ideas presented above; however, rather than specifying particular rotation-invariant quantities to utilize or learning maps that operate on irreducible representations, we leverage the structure provided by geometric algebra to calculate rotation-invariant and -covariant quantities of interest. Finally, we use an attention mechanism to attain permutation equivariance in a flexible manner.; Spellings, pg. 4, par. 1, teaches machine learning approaches in the past have treated this problem using simple multilayer perceptrons or by encoding the geometry of the problem as image-type data and learning an image translation process. We treat this problem much like language translation problems using transformers.).
However, Spellings does not distinctly disclose:
…the apparatus comprising: at least one memory; and at least one processor coupled to at least one memory…
…at least one equilinear layer, and a scalar-gated nonlinearity layer, to generate multivector outputs,…
Nevertheless, Kranski teaches …the apparatus comprising: at least one memory; and at least one processor coupled to at least one memory… (Kranski, [0032] teaches some embodiments of robot systems 102 may include an even more expansive ensemble of control models 116. For example, a machine learning subsystem 114 may pipeline a convolutional neural network (or vision transformer) that extracts features from 2D image data, a geometric deep learning model that extracts features from 3D point clouds from depth sensors, and an encoder model that maps both sets of those features for a given time slice into respective vectors in latent embedding spaces, and a reinforcement learning model that controls the robot; Kranski, [0033] teaches in some examples, one or more of a convolutional neural network, vision transformer, or geometric deep learning model may be implemented with a hardware ML Accelerator, such as in addition to a downstream encoder model implemented with a hardware ML Accelerator. Embodiments are not limited to only one ML model (or type) or a specific collection of ML models, which is not to suggest that any other description is limiting.; Kranski, [0149] FIG. 5 is a physical architecture block diagram that shows an example of a computing device (or data processing system) by which some aspects of the above techniques may be implemented. Various portions of systems and methods described herein, may include or be executed on one or more computer systems similar to computing system 1000. Further, processes and modules described herein may be executed by one or more processing systems similar to that of computing system 1000.; Kranski [0150] further teaches Computing system 1000 may include one or more processors (e.g., processors 1010a-1010n) coupled to system memory 1020… A processor may include a central processing unit (CPU) that carries out program instructions to perform the arithmetical, logical, and input/output operations of computing system 1000.)
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the geometric algebra transformer, as taught by Spellings, with the computing system of Kranski, given that performance of robots and other controlled dynamic mechanical systems is constrained by computing resources and particularly those used to implement machine learning techniques. (Kranski, [0018])
However, the combination does not distinctly disclose …at least one equilinear layer, and a scalar-gated nonlinearity layer, to generate multivector outputs,…
Nevertheless, Zhao teaches …at least one equilinear layer, and a scalar-gated nonlinearity layer, to generate multivector outputs,… (Zhao Figure 2 teaches point transformer layer; and Zhao Figure 4 teaching point transformer block comprising an input linear layer, a point transformer layer and an output linear layer; Zhao, Section 3.5 further teaching we construct complete 3D point cloud understanding networks based on the point transformer block…The network architectures are visualized in Figure 3).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the geometric algebra transformer, as taught by Spellings in view of Kranski, with the Point Transformer features, as taught by Zhao, to provide a more natural fit for point cloud processing with the developed transformer architecture for 3D point clouds.
Claim 26 is rejected under 35 U.S.C. 103 as being unpatentable over Spellings et al. in view of Kranski et al. and Zhao, as applied to claim 25, and further in view of Li et al., “A high speed reconfigurable architecture for softmax and GELU in vision transformer”, (March 2023)
Regarding claim 26, the combination of Spellings in view of Kranski and Zhao teaches all of the limitations of claim 25, however, the combination does not distinctly disclose wherein the scalar-gated nonlinearity layer comprises a scalar- gated Gaussian Error Linear Units nonlinearity layer.
Nevertheless, Li teaches wherein the scalar-gated nonlinearity layer comprises a scalar-gated Gaussian Error Linear Units nonlinearity layer (Li, Abstract, teaches transformers have been widely used in various computer vision applications. Compared to traditional convolutional neural networks (CNNs), transformer’s inference includes plenty of non-linear operations, such as softmax and Gaussian error linear units (GELU). As the scale of transformers grows, an efficient hardware implementation of these operations is significant.; Li, pg. 1, col. 1, Introduction teaches GELU is a common activation function of Transformers and will be used many times during the processing of ViT. Therefore, softmax and GELU operations make up a large portion of the ViT runtime, as shown in Fig. 1).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the geometric algebra transformer, as taught by Spellings in view of Kranski and Zhao, to further include the Gaussian error linear units as taught by Li, as it is the most commonly used activation function in transformers. (Li, Abstract and Li pg. 1, col. 1)
Claim 29 is rejected under 35 U.S.C. 103 as being unpatentable over Spellings et al. in view of Kranski et al. and Zhao et al., as applied to claim 25, and further in view of Kaul et al., “CpT: Convolutional Point Transformer for 3D Point Cloud Processing (21 Nov. 2021)
Regarding claim 29, the combination of Spellings in view of Kranski and Zhao teaches all of the limitations of claim 25, however, the combination does not distinctly disclose wherein the first normalization layer, the first geometric product engine, the first addition engine, the second normalization layer, the second geometric product engine, the scalar-gated nonlinearity layer, and the second addition engine are fixed components and wherein the first equilinear layer, the geometric attention layer, the second equilinear layer, the third equilinear layer, and the fourth equilinear layer are learnable components.
Nevertheless, Kaul teaches wherein the first normalization layer, the first geometric product engine, the first addition engine, the second normalization layer, the second geometric product engine, the scalar-gated nonlinearity layer, and the second addition engine are fixed components and wherein the first equilinear layer, the geometric attention layer, the second equilinear layer, the third equilinear layer, and the fourth equilinear layer are learnable components.
(Kaul, pg. 3, col. 2, teaches the The overall architecture of the Convolutional point Transformer (CpT) is shown in Figure 1. Our main contributions are the Point Embedding Module (Section 3.1) and the InterPoint Attention Module with a convolutional atten tion projection (Section 3.2). When an input point cloud of size N × 3 is passed through the architecture, a graph of the points is computed via finding its K-Nearest Neighbors based on Euclidean distance. This representation is then passed through a point embedding layer that maps the input data into a representation implicitly inclusive of the nearest neighbours of the points. This is done via a 2D convolution operation whose degree of overlap across points can be controlled through the length of the stride. A dot product attention operation [i.e., understood as a fixed geometric product engine] is then applied to this embedded representation which is followed by an InterPoint Attention Module. The dot product attention can be seen as learning relevant features of a points embedding as a function of it’s K nearest neighbors [i.e. a learning layer]. Such an attention mechanism learns to attend to the features of the points rather than the points themselves (column-wise matrix attention), i.e. for a set of points in a batch, it learns to weight individual feature transformations. The InterPoint Attention on the other hand can be interpreted as learning the relationships between different the points themselves, within a batch (a row-wise ma trix attention operating per point embedding, rather than per individual feature of the points). This forms one layer of the CpT.[Note: the InterPoint Attention layer understood as a “learnable” geometric attention layer] We update the graph following the InterPoint Attention operation which is then passed into the next CpT layer. A third and final CpT layer with a higher feature space embedding dimension is used without InterPoint Attention to learn the representation of the point cloud. InterPoint attention in the deeper layers does not provide a lot of context, as transformers eventually can learn relations that cross beyond the locality of their point set em bedding dimensions. This also means that transformers are able to visualize inputs beyond their limited receptive field in the deeper layers, making them a better choice for feature space embedding compared to shared-weighted MLPs (1D Convolutions). Adding this attention block to the deeper layers only marginally improves performance but significantly increases computation. The output of each transformer layer is then concatenated and passed through a final shared-weighted MLP. This is then Global Max-Pooled to get a global feature vector which forms the representation of the point cloud [i.e., understood as a fixed scalar-gated nonlinearity layer]. This representation is then processed further through a series of MLPs to classify the point cloud, or obtain a semantic point label for each point in the point cloud. Residual connections, layer normalization, and addition and MLP layers are used after the attention layers as in conventional transformer model structures [i.e., Residual connections, layer normalization, and addition and MLP layers understood as fixed components].
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the geometric algebra transformer, as taught by Spellings in view of Kranski and Zhao, to further include the features of the convolutional point transformer, as taught by Kaul. The added InterPoint Attention Module to the Transformer learns to enhance the output by learning to relate each point in the input to every other point. This helps capture better geometric relationships between the points and aids in better learning of the local and global concepts in the data. The resultant transformer block, i.e., the CpT layer (shown in Figure 2) is a combination of a dot product attention operation, followed by an InterPoint Attention module. (Kaul, pg. 2, col. 1)
Allowable Subject Matter
Claims 10, 11, 27, and 28 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The following prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Nan et al., “Learning Geometric Feature Embedding with Transformers for Image Matching” (Dec. 2022)
Ruhe et al., “Geometric Clifford Algebra Networks”, (13 Feb. 2023)
Qin et al., “Geometric Transformer for Fast and Robust Point Cloud Registration” (2022)
Hendrycks et al., “Gaussian Error Linear Units (GELUs), (8 Jul. 2020)
Yu et al., “PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers” (2021)
Zheng et al., “A Survey on Transformers for Point Cloud Processing: An Updated Overview”, (Aug. 2022)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BEATRIZ RAMIREZ BRAVO whose telephone number is 571-272-2156. The examiner can normally be reached Mon. - Fri. 7:30a.m.-5:00p.m..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, USMAAN SAEED can be reached at 571-272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/B.R.B./Examiner, Art Unit 2146
/USMAAN SAEED/Supervisory Patent Examiner, Art Unit 2146
Prosecution Insights