DETAILED ACTION
This action is in response to the application field on 02/28/2024. Claims 1-12 are pending and have been examined
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 03/01/2024. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1, 2, 4, 5, 7, 8, 10, and 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Satorras et al. (E(n) Equivariant Graph Neural Networks) (hereafter referred to as Satorras) in view of Haviv et al. (US 11514647 B1) (hereafter referred to as Havi).
Regarding claim 1, Satorras teaches
provide an input dataset represented by an input group representation into input layers connected to a Group Representation Network (GRepsNet) (Satorras, Section 3, “In this section we present Equivariant Graph Neural Networks (EGNNs). Following the notation from background Section 2.2, we consider a graph G = (V,E) with nodes vi ∈ V and edges eij ∈ E. In addition to the feature node embeddings hi ∈ Rnf we now also consider an-dimensional coordinate xi ∈ Rn associated with each of the graph nodes...Our Equivariant Graph Convolutional Layer (EGCL) takes as input the set of node embeddings hl = {hl0,...,hlM−1}, coordinate embeddings xl = {xl0,...,xlM−1} and edge information E =(eij) and outputs a transformation on hl+1 and xl+1. Concisely: hl+1,xl+1 = EGCL[hl,xl,E]”. Examiner notes that the Equivariant Graph Neural Network (EGNN) maps to the Group Representation Network (GRepNet).
transform the input dataset by using the GRepsNet configured to generate an output dataset represented by an output group representation from the input dataset, wherein the GRepsNet is designed to be equivariant to a predetermined transformation group (Satorras, Section 3, “Our model will preserve equivariance to rotations and translations on these set of coordinates xi and it will also preserve equivariance to permutations on the set of nodes V in the same fashion as GNNs. Our Equivariant Graph Convolutional Layer (EGCL) takes as input the set of node embeddings hl = {hl 0,...,hl M−1}, coordinate embeddings xl = {xl 0,...,xl M−1} and edge information E = (eij) and outputs a transformation on hl+1 and xl+1. Concisely: hl+1,xl+1 = EGCL[hl,xl,E]. The equations that define this layer are the following: “
PNG
media_image1.png
349
779
media_image1.png
Greyscale
output the output dataset from output layers connected to the GRepsNet (Satorras, Section 3, “Finally, equations 5 and 6 follow the same updates than standard GNNs. Equation 5 just aggregates all the incoming messages from neighbor nodes N(i) to node vi and Equation 6 performs the node operation φv which takes as input the aggregated messages mi, the node emedding hli and outputs the updated node embedding hl+1”).
Satorras does not teach, but Haviv does teach
A non-transitory computer-readable medium having stored thereon a set of instructions for data transformations (Haviv, column 2, lines 29-31, “ There may be provided various methods, systems and non-transitory computer readable medium for generating translation and rotation invariant representation of an object“).
Satorras and Haviv are considered analogous to the claimed invention because they deal with invariance. It would have been obvious to one having ordinary skill in the art prior to the effective filling date to have modified Satorras to include a non-transitory computer-readable medium from Haviv. One of the ordinary skill in the art would have known to apply the known technique to apply Jenni’s technique of a non-transitory computer-readable medium to perform the instructions from Satorras. Therefore, applying Haviv’s technique would yield the predicable result of running instructions on a non-transitory computer-readable medium (See MPEP 2141 (III)(D) Applying a known technique to a known device ready for improvement to yield predicable results).
Regarding claim 2, Satorras and Haviv teach the non-transitory computer readable medium of claim 1, Satorras further teaches
the GRepsNet is formed by one or more linear neural layers with no point-wise non- linearities or bias terms when the input group representation is first or higher-order tensors (Satorras, Appendix C, “In this Appendix section we describe the implementation details of the experiments. First, we describe those parts of our model that are the same across all experiments. Our EGNN model from Section 3 contains the following three main learnable functions. The edge function φe(eq.3) is a two layers MLP with two Swish non-linearities: Input−→{LinearLayer()−→Swish()−→LinearLayer()−→Swish()}−→Output” and “In Equation 4 we update the position of each particle xi as a vector field in a radial direction. In other words, the position of each particle xi is updated by the weighted sum of all relative differences (xi−xj)∀j. The weights of this sum are provided as the output of the function φx : Rnf → R1 that takes as input the edge embedding mij from the previous edge operation and outputs a scalar value” (Satorras, Section 3)).
wherein the linear neural layers combine the input group representation with the output group representation linearly which maintains equivariance to the predetermined transformation group (Satorras, Section 3, “Our model will preserve equivariance to rotations and translations on these set of coordinates xi and it will also preserve equivariance to permutations on the set of nodes V in the same fashion as GNNs” and “In this section we prove that our model is translation equivariant on x for any translation vector g ∈ Rn and it is rotation and reflection equivariant on x for any orthogonal matrix Q ∈ Rn×n…Thus concluding that a transformation Qxl+g on xl will result in the same transformation on xl+1 while hl+1 will remain invariant to it such that Qxl+1 +g,hl+1 = EGCL(Qxl +g,hl) is satisfied” (Satorras, Appendix A)).
Regarding claim 4, Satorras and Haviv teach the non-transitory computer readable medium of claim 1, Satorras further teaches
the GRepsNet is formed by one or more invariant non-linearities applied to the outputs of linear neural layers (Satorras, Appendix C, “In this Appendix section we describe the implementation details of the experiments. First, we describe those parts of our model that are the same across all experiments. Our EGNN model from Section 3 contains the following three main learnable functions. The edge function φe(eq.3) is a two layers MLP with two Swish non-linearities: Input−→{LinearLayer()−→Swish()−→LinearLayer()−→Swish()}−→Output”).
wherein the outputs of the non-linearities are configured to interact with the output of the linear neural layers and maintain the equivariance to a predetermined transformation group (Satorras, Section 3, “In Equation 4 we update the position of each particle xi as a vector field in a radial direction. In other words, the position of each particle xi is updated by the weighted sum of all relative differences (xi−xj)∀j. The weights of this sum are provided as the output of the function φx : Rnf → R1 that takes as input the edge embedding mij” and “Next, Equation 4 computes xl+1 i by a weighted sum of differences (xi − xj) which is added to xi, this transforms as a type-1 vector and preserves equivariance (see Appendix A)” (Satorras, Section 3.1)).
wherein the invariant non-linearities are represented by Euclidean norm (Satorras, Section 3, “In equation 3 we now input the relative squared distance between two coordinates ||xli− xlj||^2 into the edge operation φe”. Examiner notes that the squared distance maps to the Euclidean norm).
Regarding claim 5, Satorras and Haviv teach the non-transitory computer readable medium of claim 1, Satorras further teaches
the GRepsNet consists of multiple layers, wherein each of the multiple layers is equivariant to the predetermined transformation group (Satorras, Appendix C.2, “All models consist of 4 layers, 64 features for the hidden layers and the Swish activation function as a non linearity” and “Our model will preserve equivariance to rotations and translations on these set of coordinates xi and it will also preserve equivariance to permutations on the set of nodes V in the same fashion as GNNs” (Satorras, Section 3)).
Regarding claim 7, Satorras teaches
providing an input dataset represented by an input group representation into input layers connected to a Group Representation Network (GRepsNet) (Satorras, Section 3, “In this section we present Equivariant Graph Neural Networks (EGNNs). Following the notation from background Section 2.2, we consider a graph G = (V,E) with nodes vi ∈ V and edges eij ∈ E. In addition to the feature node embeddings hi ∈ Rnf we now also consider an-dimensional coordinate xi ∈ Rn associated with each of the graph nodes...Our Equivariant Graph Convolutional Layer (EGCL) takes as input the set of node embeddings hl = {hl0,...,hlM−1}, coordinate embeddings xl = {xl0,...,xlM−1} and edge information E =(eij) and outputs a transformation on hl+1 and xl+1. Concisely: hl+1,xl+1 = EGCL[hl,xl,E]”. Examiner notes that the Equivariant Graph Neural Network (EGNN) maps to the Group Representation Network (GRepNet).
transforming the input dataset by using the GRepsNet configured to generate an output dataset represented by an output group representation from the input dataset, wherein the GRepsNet is designed to be equivariant to a predetermined transformation group (Satorras, Section 3, “Our model will preserve equivariance to rotations and translations on these set of coordinates xi and it will also preserve equivariance to permutations on the set of nodes V in the same fashion as GNNs. Our Equivariant Graph Convolutional Layer (EGCL) takes as input the set of node embeddings hl = {hl 0,...,hl M−1}, coordinate embeddings xl = {xl 0,...,xl M−1} and edge information E = (eij) and outputs a transformation on hl+1 and xl+1. Concisely: hl+1,xl+1 = EGCL[hl,xl,E]. The equations that define this layer are the following: “
PNG
media_image1.png
349
779
media_image1.png
Greyscale
outputting the output dataset from output layers connected to the GRepsNet (Satorras, Section 3, “Finally, equations 5 and 6 follow the same updates than standard GNNs. Equation 5 just aggregates all the incoming messages from neighbor nodes N(i) to node vi and Equation 6 performs the node operation φv which takes as input the aggregated messages mi, the node emedding hli and outputs the updated node embedding hl+1”).
Satorras does not teach, but Haviv does teach
A data transformation system, comprising: a memory configured to store a Group Representation Network (GRepsNet) and a set of instructions for data transformations using the GRepsNet; and a processor coupled to the memory and configured to perform the set of instructions including steps (Haviv, column 2, lines 29-31, “There may be provided various methods, systems and non-transitory computer readable medium for generating translation and rotation invariant representation of an object” and “The calculating of the translation and rotation invariant features may include: decomposing the second function to a sum of irreducible representations of three dimensional rotation group; the irreducible representations may be of different types; and calculating multiple tensor products, each tensor product may be calculated between a combination of at least two irreducible representations of a same type of irreducible representations” (Haviv, column 14, lines 64 – column 15, lines 1-4). Examiner notes that the irreducible representations map to the Group Representation Network).
Satorras and Haviv are considered analogous to the claimed invention because they deal with equivariant models. It would have been obvious to one having ordinary skill in the art prior to the effective filling date to have modified Satorras to include system from Haviv. One of the ordinary skill in the art would have known to apply the known technique to apply Haviv’s technique of a non-transitory computer-readable medium to perform the instructions from Satorras. Therefore, applying Jenni’s technique would yield the predicable result of running instructions on a computer (See MPEP 2141 (III)(D) Applying a known technique to a known device ready for improvement to yield predicable results).
Regarding claim 8, Satorras and Haviv teach the system of claim 7, Satorras further teaches
the GRepsNet is formed by one or more linear neural layers with no-point wise non- linearities or bias terms when the input group representation is first or higher-order tensors (Satorras, Appendix C, “In this Appendix section we describe the implementation details of the experiments. First, we describe those parts of our model that are the same across all experiments. Our EGNN model from Section 3 contains the following three main learnable functions. The edge function φe(eq.3) is a two layers MLP with two Swish non-linearities: Input−→{LinearLayer()−→Swish()−→LinearLayer()−→Swish()}−→Output” and “In Equation 4 we update the position of each particle xi as a vector field in a radial direction. In other words, the position of each particle xi is updated by the weighted sum of all relative differences (xi−xj)∀j. The weights of this sum are provided as the output of the function φx : Rnf → R1 that takes as input the edge embedding mij from the previous edge operation and outputs a scalar value” (Satorras, Section 3)).
wherein the linear neural layers combine the input group representation with the output group representation linearly (Satorras, Section 3, “Our model will preserve equivariance to rotations and translations on these set of coordinates xi and it will also preserve equivariance to permutations on the set of nodes V in the same fashion as GNNs” and “In this section we prove that our model is translation equivariant on x for any translation vector g ∈ Rn and it is rotation and reflection equivariant on x for any orthogonal matrix Q ∈ Rn×n…Thus concluding that a transformation Qxl+g on xl will result in the same transformation on xl+1 while hl+1 will remain invariant to it such that Qxl+1 +g,hl+1 = EGCL(Qxl +g,hl) is satisfied” (Satorras, Appendix A)).
Regarding claim 10, Satorras and Haviv teach the system of claim 8, Satorras further teaches
the GRepsNet is formed by one or more invariant non-linearities applied to the outputs of linear neural layers (Satorras, Appendix C, “In this Appendix section we describe the implementation details of the experiments. First, we describe those parts of our model that are the same across all experiments. Our EGNN model from Section 3 contains the following three main learnable functions. The edge function φe(eq.3) is a two layers MLP with two Swish non-linearities: Input−→{LinearLayer()−→Swish()−→LinearLayer()−→Swish()}−→Output”).
wherein the outputs of the non-linearities are configured to interact with the output of the linear neural layers and maintain the equivariance to a predetermined transformation group (Satorras, Section 3, “In Equation 4 we update the position of each particle xi as a vector field in a radial direction. In other words, the position of each particle xi is updated by the weighted sum of all relative differences (xi−xj)∀j. The weights of this sum are provided as the output of the function φx : Rnf → R1 that takes as input the edge embedding mij” and “Next, Equation 4 computes xl+1 i by a weighted sum of differences (xi − xj) which is added to xi, this transforms as a type-1 vector and preserves equivariance (see Appendix A)” (Satorras, Section 3.1)).
wherein the invariant non-linearities are represented by Euclidean norm (Satorras, Section 3, “In equation 3 we now input the relative squared distance between two coordinates ||xli− xlj||^2 into the edge operation φe”. Examiner notes that the squared distance maps to the Euclidean norm).
Regarding claim 11, Satorras and Haviv teach the system of claim 7, Satorras further teaches
the GRepsNet consists of multiple layers, wherein each of the multiple layers is equivariant to the predetermined transformation group (Satorras, Appendix C.2, “All models consist of 4 layers, 64 features for the hidden layers and the Swish activation function as a non linearity” and “Our model will preserve equivariance to rotations and translations on these set of coordinates xi and it will also preserve equivariance to permutations on the set of nodes V in the same fashion as GNNs” (Satorras, Section 3)).
Claim(s) 3 and 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Satorras in view of Haviv and Thomas et al. (Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds) (hereafter referred to as Thomas). Regarding claim 3, Satorras and Haviv teach the non-transitory computer readable medium of claim 2, Satorras and Haviv do not teach, but Thomas teaches
the GRepsNet is configured to create additional higher-order group representations (Thomas, Section 4.1.2, “In order to produce output that we can feed into downstream layers, we need to combine the layer input and filters in such a way that the output also transforms appropriately (by inhabiting a representation of SO(3)). A tensor product of representations is a prescription for combining two representations DX and DY to get another representation DX ⊗ DY over the vector space X ⊗ Y. The crucial property of the tensor product is that it is equivariant: DX ⊗DY =DX⊗Y”. Examiner notes that the product of the representation maps to the higher-order group representation).
Satorras, Haviv, and Thomas are considered analogous to the claimed invention because they deal with invariance. It would have been obvious to one having ordinary skill in the art prior to the effective filling date to have modified Satorras and Haviv to include the tensor product of representation from Thomas. Thomas teaches “3D rotation equivariance removes the need for data augmentation to identify features in arbitrary orientations” (Thomas, Abstract) (See MPEP 2141 (III)(G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention).
Regarding claim 9, Satorras and Haviv teach the system of claim 8, Satorras and Haviv do not teach, but Thomas teaches
the GRepsNet is configured to create additional higher-order group representations (Thomas, Section 4.1.2, “In order to produce output that we can feed into downstream layers, we need to combine the layer input and filters in such a way that the output also transforms appropriately (by inhabiting a representation of SO(3)). A tensor product of representations is a prescription for combining two representations DX and DY to get another representation DX ⊗ DY over the vector space X ⊗ Y. The crucial property of the tensor product is that it is equivariant: DX ⊗DY =DX⊗Y”. Examiner notes that the product of the representation maps to the higher-order group representation).
Satorras, Haviv, and Thomas are considered analogous to the claimed invention because they deal with invariance. It would have been obvious to one having ordinary skill in the art prior to the effective filling date to have modified Satorras and Haviv to include the tensor product of representation from Thomas. Thomas teaches “3D rotation equivariance removes the need for data augmentation to identify features in arbitrary orientations” (Thomas, Abstract) (See MPEP 2141 (III)(G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention).
Claim(s) 6 and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Satorras in view of Haviv and Cohen et al. (Group Equivariant Convolutional Networks (hereafter referred to as Cohen). Regarding claim 6, Satorras and Haviv teach the non-transitory computer readable medium of claim 2, Satorras and Haviv do not teach, but Cohen teaches
when an invariance is required at the output dataset, a pooling invariant layer is additionally arranged at the output layers to pool over group dimensions in the output dataset (Cohen, Section 6.3, “We can obtain full G-equivariance by choosing our pooling region U to be a subgroup H ⊂ G...The feature map that results from pooling over cosets is invariant to the right-action of H...As an example, in a p4 feature map, we can pool over all four rotations at each spatial position (the cosets of the sub group R of rotations around the origin). The resulting feature map is a function on Z2 ∼ =p4/R, i.e. it will transform in the same way as the input image”)
Satorras, Haviv, and Cohen are considered analogous to the claimed invention because they deal with invariance. It would have been obvious to one having ordinary skill in the art prior to the effective filling date to have modified Satorras and Haviv to include the pooling layer from Cohen. Cohen teaches that ”since all layer types are equivariant, we can freely stack them into deep networks and expect G-conv parameter sharing to be effective at arbitrary depth” (Cohen, Section 6.3) (See MPEP 2141 (III)(G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention).
Regarding claim 12, Satorras and Haviv teach the system of claim 7, Satorras and Haviv do not teach, but Cohen teaches
when an invariance is required at the output dataset, a pooling invariant layer is additionally arranged at the output layers to pool over group dimensions in the output dataset (Cohen, Section 6.3, “We can obtain full G-equivariance by choosing our pooling region U to be a subgroup H ⊂ G...The feature map that results from pooling over cosets is invariant to the right-action of H...As an example, in a p4 feature map, we can pool over all four rotations at each spatial position (the cosets of the sub group R of rotations around the origin). The resulting feature map is a function on Z2 ∼ =p4/R, i.e. it will transform in the same way as the input image”)
Satorras, Haviv, and Cohen are considered analogous to the claimed invention because they deal with invariance. It would have been obvious to one having ordinary skill in the art prior to the effective filling date to have modified Satorras and Haviv to include the pooling layer from Cohen. Cohen teaches that ”since all layer types are equivariant, we can freely stack them into deep networks and expect G-conv parameter sharing to be effective at arbitrary depth” (Cohen, Section 6.3) (See MPEP 2141 (III)(G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Deng et al. (Vector Neurons: A General Framework for SO(3)-Equivariant Networks) discloses a general framework built on top of Vector Neuron representations for creating SO(3)-equivariant neural networks for point cloud processing. Batatia et al. (A General Framework for Equivariant Neural Networks on Reductive Lie Groups) discloses a general Equivariant Neural Network architecture capable of respecting the symmetries of the finite dimensional representations of any reductive Lie Group G.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to STEVEN VO whose telephone number is (571)272-9622. The examiner can normally be reached Monday - Friday from 7-3 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.V./ Examiner, Art Unit 2148 /Ryan Barrett/Primary Examiner, Art Unit 2148
Prosecution Insights