The present application, filed on or after 16 March 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This office action is in response to Applicant’s submission filed on 14 September 2026. THIS ACTION IS NON-FINAL.
In response to the restriction requirement, Applicant’s election without traverse of the instant application in the reply filed on 14 September 2026 is acknowledged.
Status of Claims
Claims 1-20 are pending.
Claims 10-14 are withdrawn.
Claims 6, 9, 18 and 19 are rejected under 35 U.S.C. 112(b) as indefinite.
Claims 1-9, 15-20 are rejected under 35 U.S.C. 101 for being directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
Claims 1-9, 15-20 are rejected under 35 U.S.C. 103 as unpatentable.
Claims 17 and 18 are objected to.
Claim Objections
Claims 17 and 18 recite "the method of claim 1." For purposes of this rejection, they are treated as depending from method claim 13. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
A claim is indefinite if, when read in light of the specification, it fails to inform, with reasonable certainty, those skilled in the art about the scope of the invention. Nautilus, Inc. v. Biosig Instruments, Inc., 110 USPQ.2d 1688, U.S. Supreme Court (2014).
Claims 6, 9, 18 and 19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Claim 6 recites “a class estimation of an object corresponding to a bounding box.” Parent claim 5 already recites “a bounding box of an object detected in the input data item.” It is unclear whether the object and bounding box of claim 6 are the same as, or different from, those of claim 5, the claim is therefore indefinite. For examination, they are interpreted as the same object and bounding box.
Claim 9 recites “wherein the queries have different dimensions and the ensemble query is based on respective transformations of the queries that have a same dimension.” The relative clause “that have a same dimension” can be read as modifying either “transformations” or “the queries.” Read the second way, the clause contradicts the immediately preceding recitation that “the queries have different dimensions,” so the metes and bounds of the claim are unclear. For examination, the clause is interpreted as modifying “transformations” (see Spec. ¶¶ [0047]–[0048]).
Claims 18 and 19 each recite “the estimated value of the input data item.” There is insufficient antecedent basis for this limitation. Independent claim 16 and intervening claim 17 recite “a predicted value of the input data item,” not an “estimated value.” For examination, the phrase is interpreted as “the predicted value.”
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Judicial Exception
Claims 1-9, 15-20 of the claimed invention are directed to a judicial exception, an abstract idea, without significantly more.
Regarding claims 1-9, 15-20,
(Independent Claims)
With regards to claim 1,
Step 1: The claim recites a method (a series of steps), which is a process and falls into one of the statutory categories.
Step 2A – Prong 1: The claim, in part, recites:
(a) “forming an ensemble query corresponding to the queries” (mathematical concept and/or mental process). As drafted, and under its broadest reasonable interpretation (BRI) in light of the specification, this limitation covers combining sets of numerical values (query vectors) into one aggregate data structure. The specification confirms the combination is a mathematical operation: “the ensemble query may be generated by performing an ensemble operation on the queries, e.g., concatenation” (¶ [0046]), and “when the N projected queries … each has a size 1×D, the ensemble query … may form a tensor of a size N×D” (¶ [0051]). Arranging numerical vectors into a matrix is a mathematical relationship or calculation (MPEP 2106.04(a)(2)(I)). At the high level of generality claimed, nothing precludes a person from also forming such a combined list of values with pen and paper (MPEP 2106.04(a)(2)(III)).
(b) “generating a predicted value of the input data item … the prediction model inferring the predicted value from the ensemble query” (mental process and/or mathematical concept). As drafted, and under its BRI, this limitation covers evaluating the combined data and drawing a conclusion from it, for example judging where an object is and what it is. That is observation, evaluation, judgment and/or opinion, which, but for the recitation of a generic “prediction model,” a person could practically perform mentally or with pen and paper (for example, deciding from several detectors’ outputs where a box should be drawn and what class to assign). The claim recites the inference only as a result, with no particular way of reaching it. To the extent the inference is performed by numerical computation, it is a mathematical calculation. If a claim limitation, under its BRI, covers performance of the limitation in the mind but for the recitation of generic computer components, it falls within the “Mental Processes” grouping of abstract ideas.
Accordingly, the claim recites an abstract idea.
Step 2A – Prong 2: This judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements of:
(a) “an ensemble model based on feature-level consolidation,” “respective transformer models,” and “a prediction model that comprises a transformer decoder.” These are recited at a high level of generality, as generic machine-learning models applied to perform the abstract combining and inferring steps. They amount to mere instructions to implement an abstract idea with generic machine-learning technology, or merely use a computer as a tool to perform an abstract idea (see MPEP 2106.05(f); Recentive, slip op. at 10–12, which held that claims applying established machine-learning methods to a new data environment, with no disclosed improvement to the machine-learning model itself, are directed to an abstract idea). The claim does not recite how the transformer decoder is structured, trained or constrained in a manner that improves the functioning of the model itself (compare Desjardins, which relied on claimed training-process details).
(b) “obtaining queries by inputting a same input data item to respective transformer models, the transformer models generating respective queries from the input data item.” This is insignificant extra-solution activity of pre-solution data gathering, obtaining the data that the abstract combining and inferring steps operate on (MPEP 2106.05(g)). To the extent the transformer models themselves generate the queries, they are generic machine-learning tools (MPEP 2106.05(f)).
(c) Limiting the abstract idea to the environment of “ensemble” transformer models is a field-of-use and technological-environment limitation (MPEP 2106.05(h)).
Accordingly, the additional elements, individually or in combination, do not integrate the judicial exception into a practical application. The claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above, the claim recites the additional elements of: (a) generic “transformer models” and a “prediction model that comprises a transformer decoder,” which are mere instructions to implement an abstract idea using generic machine-learning and computing technology (MPEP 2106.05(f)); (b) obtaining queries by inputting a data item to models, which is extra-solution activity of pre-solution data gathering (MPEP 2106.05(g)). The courts have found limitations directed to obtaining and processing information electronically, recited at a high level of generality, to be well-understood, routine, and conventional (see MPEP 2106.05(d)(II): “receiving or transmitting data over a network,” “storing and retrieving information in memory”). The use of transformer models that output object queries, and of a transformer decoder that decodes queries into predictions, was itself well-understood, routine and conventional before the effective filing date, as evidenced by Carion et al., “End-to-End Object Detection with Transformers,” ECCV 2020, Sec. 3.2, p. 7, and Xie et al., “SparseFusion,” arXiv:2304.14340, Table 1 and Sec. 2 (surveying transformer-query detectors such as TransFusion, FUTR3D, UVTR, DeepInteraction and CMT) (Berkheimer evidence per MPEP 2106.07(a)(III)(C)); and (c) the ensemble environment, which is field of use (MPEP 2106.05(h)). Accordingly, the additional elements, individually or in combination, do not amount to significantly more than the judicial exception. The claim is not patent eligible.
With regards to claim 16,
Step 1: The claim recites an apparatus comprising one or more processors, which is a machine and falls into one of the statutory categories.
Step 2A – Prong 1: The claim recites “obtain an ensemble query corresponding to the queries” and “obtain a predicted value of the input data item by applying the ensemble query to a prediction model,” which are the same mathematical concepts and mental processes identified for claim 1. Accordingly, the claim recites an abstract idea.
Step 2A – Prong 2: The claim recites the additional elements of: (a) “An apparatus comprising one or more processors configured to …,” which is mere instructions to implement an abstract idea on a generic computing device, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)); (b) “generate, by transformer-based models, respective queries corresponding to an input data item inputted to the transformer-based models,” which is pre-solution data gathering performed by generic machine-learning tools (MPEP 2106.05(g), (f)); and (c) “a prediction model comprising a transformer decoder,” which is a generic machine-learning tool (MPEP 2106.05(f)). Accordingly, the additional elements, individually or in combination, do not integrate the judicial exception into a practical application. The claim is directed to an abstract idea.
Step 2B: As discussed above, the additional elements (generic processors; generic transformer-based models and decoder; data gathering) are mere instructions to apply the exception using generic computer and machine-learning components (MPEP 2106.05(f)), and well-understood, routine, conventional activity (MPEP 2106.05(d)(II)). Individually or in combination they do not amount to significantly more than the judicial exception. The claim is not patent eligible.
With regards to claim 15,
Claim 15 recites a non-transitory computer-readable storage medium storing instructions that cause a processor to perform the method of claim 1. At Step 1 it is an article of manufacture (a statutory category). It recites the same abstract idea as claim 1 (Step 2A – Prong 1). The only additional elements beyond those of claim 1 are the generic storage medium and processor. These are mere instructions to implement the abstract idea on a generic computer (MPEP 2106.05(f)), and they neither integrate the exception into a practical application (Step 2A – Prong 2) nor amount to significantly more (Step 2B). The claim is not patent eligible.
(Dependent claims)
Claims 2–9 depend on claim 1 and claims 17–20 depend on claim 16; each includes all the limitations of its parent claim. Therefore, claims 2–9 and 17–20 recite the same abstract ideas.
With regards to claims 2 and 17, the claims recite the further limitation of projecting the queries “to respective projected queries that share a same feature space” and forming the ensemble query as “a concatenation of the projected queries.” This is a further mathematical concept. Projecting a vector into a feature space by a linear layer is a matrix multiplication (Spec. ¶ [0047]: “the projection networks may each include a linear layer”), and concatenating vectors is a mathematical operation (¶ [0051]).
The claim recites the additional element of “respective projection networks” (a “projection network” in claim 17), recited only as generic neural-network layers that perform the mathematical projection. This is merely using a computer or generic machine learning as a tool to perform an abstract idea (MPEP 2106.05(f)). Accordingly, at Step 2A – Prong 2, the additional elements individually or in combination do not integrate the judicial exception into a practical application, and at Step 2B they individually or in combination do not amount to significantly more than the judicial exception. The claim is not patent eligible.
With regards to claim 3, the claim recites the further limitation that the prediction model comprises “a prediction network that generates the predicted value,” which further specifies the generic tool that performs the abstract inferring step of claim 1.
The claim recites the additional element of “a prediction network,” a generic machine-learning component recited at a high level of generality. This is merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). Accordingly, at Step 2A – Prong 2, the additional elements individually or in combination do not integrate the judicial exception into a practical application, and at Step 2B they individually or in combination do not amount to significantly more than the judicial exception. The claim is not patent eligible.
With regards to claim 4, the claims recite the further limitations of “obtaining output embedding data by applying the ensemble query to the transformer decoder” and obtaining the predicted or estimated value “by applying the output embedding data to the prediction network.” These are further mathematical transformations of numerical data followed by the evaluation/inference identified in claim 1(b) (mathematical concept / mental process).
The claim recites the additional element of the generic “transformer decoder” and “prediction network,” which are merely used as tools to perform the abstract idea (MPEP 2106.05(f)). Accordingly, at Step 2A – Prong 2, the additional elements individually or in combination do not integrate the judicial exception into a practical application, and at Step 2B they individually or in combination do not amount to significantly more than the judicial exception. The claim is not patent eligible.
With regards to claim 5, the claim recites the further limitation that the input data item comprises “an image or a point cloud” and the predicted value comprises “a bounding box of an object … or class information of the object.” Identifying an object in an image and outlining it with a box, or naming its class, is observation, evaluation and judgment that a person can practically perform visually and with pen and paper (mental process).
The claim recites the additional element of the image or point cloud data type, which merely limits the abstract idea to a particular type of data and technological environment (MPEP 2106.05(h)) and is data gathering (MPEP 2106.05(g)). Accordingly, at Step 2A – Prong 2, the additional elements individually or in combination do not integrate the judicial exception into a practical application, and at Step 2B they individually or in combination do not amount to significantly more than the judicial exception. The claim is not patent eligible.
With regards to claim 6, the claim recites the further limitation that the prediction model comprises “a network configured for bounding box regression … and … for a class estimation.” Regression is a mathematical calculation, and class estimation is a mental evaluation/judgment or mathematical calculation.
The claim recites the additional element of “a network,” a generic machine-learning component used as a tool (MPEP 2106.05(f)). Accordingly, at Step 2A – Prong 2, the additional elements individually or in combination do not integrate the judicial exception into a practical application, and at Step 2B they individually or in combination do not amount to significantly more than the judicial exception. The claim is not patent eligible.
With regards to claim 7, the claim recites the further limitation that the prediction model comprises “a neural network trained based on a loss function related to a difference between the predicted value … and ground truth data.” Computing a loss from a difference, and optimizing parameters against it, is a mathematical concept.
The claim recites the additional element of “a neural network trained,” recited at a high level of generality. Training a generic neural network on ground truth is incident to the nature of machine learning and is merely using a computer as a tool (MPEP 2106.05(f); Recentive). Accordingly, at Step 2A – Prong 2, the additional elements individually or in combination do not integrate the judicial exception into a practical application, and at Step 2B they individually or in combination do not amount to significantly more than the judicial exception. The claim is not patent eligible.
With regards to claims 8 and 18, the claims recite the further limitation that the projection networks (projection network) and the prediction model comprise “a neural network trained based on a loss function related to a difference between the predicted [estimated] value … and ground truth data.” This is a further mathematical concept (loss computation and optimization).
The claim recites the additional element of the generic trained neural networks, which are merely used as tools to perform the abstract idea (MPEP 2106.05(f)). Accordingly, at Step 2A – Prong 2, the additional elements individually or in combination do not integrate the judicial exception into a practical application, and at Step 2B they individually or in combination do not amount to significantly more than the judicial exception. The claim is not patent eligible.
With regards to claim 9, the claim recites the further limitation that “the queries have different dimensions and the ensemble query is based on respective transformations of the queries that have a same dimension.” Mapping vectors of different lengths to vectors of a common length (e.g., by a D×d linear map) is a further mathematical concept.
Except citing generic computer elements to implement the abstract idea, there is no additional element showing integration into a practical application or adding something significantly more to the abstract idea. The claim is not patent eligible.
With regards to claim 19, the claims recite the further limitation of training the prediction model and the projection network(s) based on the loss function. Loss-based parameter optimization is a further mathematical concept.
The claim recites the additional element of training generic neural networks, recited at a high level of generality without any particular technique that improves the functioning of the models. This is merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f); Recentive). Accordingly, at Step 2A – Prong 2, the additional elements individually or in combination do not integrate the judicial exception into a practical application, and at Step 2B they individually or in combination do not amount to significantly more than the judicial exception. The claim is not patent eligible.
With regards to claim 20, the claim recites the further limitation that the input data item comprises “an image and a point cloud” and the predicted value comprises “a bounding box … or a classification of the object,” which is a further mental process for the reasons given for claim 5.
The claim recites the additional element of the image-and-point-cloud data type, which merely limits the abstract idea to a particular data environment (MPEP 2106.05(h)) and is data gathering (MPEP 2106.05(g)). Accordingly, at Step 2A – Prong 2, the additional elements individually or in combination do not integrate the judicial exception into a practical application, and at Step 2B they individually or in combination do not amount to significantly more than the judicial exception. The claim is not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1–9, 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over Xie et al., “SparseFusion: Fusing Multi-Modal Sparse Representations for Multi-Sensor 3D Object Detection,” arXiv:2304.14340, Apr. 27, 2023 [hereafter SparseFusion] in view of Carion et al., “End-to-End Object Detection with Transformers,” arXiv:2005.12872 (May 2020), ECCV 2020 [hereafter Carion].
With regards to claim 1, SparseFusion teaches
“A method of operating an ensemble model based on feature-level consolidation, the method comprising:” (SparseFusion, FIG.1-3,
PNG
media_image1.png
393
728
media_image1.png
Greyscale
Abstract, p. 1, ‘SparseFusion utilizes the outputs of parallel detectors in the LiDAR and camera modalities as sparse candidates for fusion. … Then, we can fuse the multi-modality candidates in a unified 3D space by a lightweight self-attention module’; Sec. 1, p. 2, ‘we perform 3D object detection on the inputs from each modality in two parallel branches. Then, the instance features from each branch are projected into a unified 3D space’. SparseFusion combines multiple independently operating detectors (an ensemble) by fusing their instance-level features rather than their final boxes, which is an ensemble model based on feature-level consolidation).
“obtaining queries by inputting a same input data item to respective transformer models,” (SparseFusion, FIG.1-3, Sec. 3.1, p. 4, ‘LiDAR candidates. We follow TransFusion-L [1] and use only one decoder layer for LiDAR modality detection. … We initialize N_L object queries’ and ‘Camera candidates. … we extend deformable-DETR [58] with 3D box regression heads. We also initialize N_C object queries’; Sec. B.1, p. 12, ‘We follow TransFusion-L [1] to adopt a transformer-based LiDAR detector’ and ‘We extend Deformable-DETR [58] to the 3D object detection task’; Sec. 4.1, p. 6, the nuScenes dataset ‘provides point clouds collected using a 32-beam LiDAR and six images from multi-view cameras’; Sec. 3.2, p. 5, geometric transfer (LiDAR points projected to images and ‘used as the input to the camera branch’) and semantic transfer (image features combined with the BEV features used for LiDAR query initialization). Each of the two transformer detectors (respective transformer models) receives the same multi-sensor sample of one scene (point cloud and multi-view images), i.e., a same input data item. This reading is consistent with the instant specification, where an input data item may comprise “an image and a point cloud” (Spec. ¶ [0058]; claim 20)
PNG
media_image2.png
214
369
media_image2.png
Greyscale
).
“the transformer models generating respective queries from the input data item;” (SparseFusion, FIG.1-3, Sec. 3.1, p. 4, ‘These queries interact with the BEV features through a cross-attention layer to generate the updated object queries Q_L … These updated queries Q_L represent the instance-level features of objects in the LiDAR modality’ and ‘queries on its image interact with the corresponding image features using a deformable attention layer [58]. The outputs of all perspective views comprise the updated queries Q_C^P’. Q_L and Q_C are the respective queries generated by the respective transformer models).
“forming an ensemble query corresponding to the queries; and” (SparseFusion, FIG.1-3, Sec. 3.1, p. 4, Eq. (2), ‘We now concatenate the candidates together: Q_LC = {q_LC,i} (i = 1 … N_L+N_C) = {f_L(q_L,i)} ∪ {f_C(q_C,i)}, where f_L(·), f_C(·) are learnable projectors’. The concatenated query set Q_LC is an ensemble query corresponding to the queries Q_L and Q_C).
“generating a predicted value of the input data item by applying the ensemble query to a prediction model … the prediction model inferring the predicted value from the ensemble query.” (SparseFusion, FIG.1-3, Sec. 3.1, p. 4, ‘Afterward, we make novel use of a self-attention module to fuse the two modalities. … The output of the self-attention module is used for final classification and regression of the bounding boxes’; Sec. B.1 (Fusion Branch), p. 13, ‘Q_LC is fed into a self-attention module and a feedforward network to get the final fused instance features Q_F. … Finally, we attach a LiDAR view prediction head to Q_F to predict the object category and 3D LiDAR view bounding boxes as the final results’. The attention-plus-feed-forward block with prediction head is a prediction model that infers the predicted values (boxes and categories) from the ensemble query).
SparseFusion does not explicitly detail that the attention-based fusion module to which the ensemble query is applied is “a transformer decoder.”
“However Carion teaches a prediction model that comprises a transformer decoder, the prediction model inferring the predicted value from the [object] query” (Carion, FIG.1-2,
PNG
media_image3.png
229
670
media_image3.png
Greyscale
Sec. 3.2, p. 7, ‘Transformer decoder. The decoder follows the standard architecture of the transformer, transforming N embeddings of size d using multi-headed self- and encoder-decoder attention mechanisms. … The N object queries are transformed into an output embedding by the decoder. They are then independently decoded into box coordinates and class labels by a feed forward network’; Fig. 2, p. 7, ‘A transformer decoder then takes as input a small fixed number of learned positional embeddings, which we call object queries’. Carion’s decoder receives a set of object queries and outputs predicted boxes and classes, i.e., a prediction model comprising a transformer decoder that infers predicted values from queries).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, and having the teachings of SparseFusion and Carion before him or her, to modify the process / system to generate prediction with ensemble models of SparseFusion to include decoder in Carion.
The motivation for doing so would have been for object detection (Carion, Abstract).
With regards to claim 2, SparseFusion in view of Carion teaches
“The method of claim 1, wherein the forming of the ensemble query corresponding to the queries comprises: inputting the queries to respective projection networks that project the queries to respective projected queries that share a same feature space;” (SparseFusion, FIG.1-3, Sec. 3.1, p. 4, Eq. (2), ‘f_L(·), f_C(·) are learnable projectors’; Sec. B.1, p. 13, ‘We process the LiDAR candidates Q_L … and camera candidates Q_C … with two separate modules f_L(·), f_C(·), each consisting of a fully-connected layer and layer normalization’; Sec. 3.1, p. 4, ‘candidates from both modalities represent bounding boxes in the same LiDAR coordinate space’; Fig. 2 caption, p. 5, ‘multi-modality instance-level representation in the unified LiDAR space’. The per-branch projectors are respective projection networks outputting projected queries in a shared (unified) feature space).
“and wherein the ensemble query comprises a concatenation of the projected queries.” (SparseFusion, FIG.1-3, Sec. B.1, p. 13, ‘Then, we concatenate the candidates as Q_LC = {q_LC,i} (i = 1 … N_L+N_C)’).
With regards to claim 3, SparseFusion in view of Carion teaches
“The method of claim 1, wherein the prediction model comprises [the transformer decoder and] a prediction network that generates the predicted value.” (SparseFusion, FIG.1-3, Sec. B.1, p. 13, ‘we attach a LiDAR view prediction head to Q_F to predict the object category and 3D LiDAR view bounding boxes as the final results’).
SparseFusion does not explicitly detail “… transformer decoder…”
However Carion teaches “… transformer decoder…” (Carion, FIG.1-2, Sec. 3.2, p. 7, ‘Transformer decoder. The decoder follows the standard architecture of the transformer, transforming N embeddings of size d using multi-headed self- and encoder-decoder attention mechanisms. … The N object queries are transformed into an output embedding by the decoder. They are then independently decoded into box coordinates and class labels by a feed forward network’; Fig. 2, p. 7, ‘A transformer decoder then takes as input a small fixed number of learned positional embeddings, which we call object queries’. Carion, Fig. 2, p. 7, ‘We pass each output embedding of the decoder to a shared feed forward network (FFN) that predicts either a detection (class and bounding box) or a “no object” class’. The prediction head / FFN is a prediction network; the transformer decoder is supplied by Carion as combined for claim 1).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, and having the teachings of SparseFusion and Carion before him or her, to modify the process / system to generate prediction with ensemble models of SparseFusion to include decoder in Carion.
The motivation for doing so would have been for object detection (Carion, Abstract).
With regards to claim 4, SparseFusion in view of Carion teaches
“The method of claim 3,”
SparseFusion does not explicitly detail “… wherein the generating of the predicted value of the input data item comprises: obtaining output embedding data by applying the ensemble query to the transformer decoder; and obtaining the predicted value of the input data item by applying the output embedding data to the prediction network.”
However Carion teaches
“wherein the generating of the predicted value of the input data item comprises: obtaining output embedding data by applying the ensemble query to the transformer decoder; and” (Carion, FIG.1-2, Sec. 3.2, p. 7, ‘The N object queries are transformed into an output embedding by the decoder’).
“obtaining the predicted value of the input data item by applying the output embedding data to the prediction network.” (Carion, FIG.1-2, Sec. 3.2, p. 7, ‘They are then independently decoded into box coordinates and class labels by a feed forward network’).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, and having the teachings of SparseFusion and Carion before him or her, to modify the process / system to generate prediction with ensemble models of SparseFusion to include processing with decoder as shown in Carion.
The motivation for doing so would have been for object detection (Carion, Abstract).
With regards to claim 5, SparseFusion in view of Carion teaches
“The method of claim 1, wherein the input data item comprises an image or a point cloud, and” (SparseFusion, FIG.1-3, Sec. 4.1, p. 6, ‘It provides point clouds collected using a 32-beam LiDAR and six images from multi-view cameras’).
“the predicted value of the input data item comprises a bounding box of an object detected in the input data item or class information of the object.” (SparseFusion, FIG.1-3, Sec. B.1, p. 13, ‘predict the object category and 3D LiDAR view bounding boxes as the final results’).
With regards to claim 6, SparseFusion in view of Carion teaches
“The method of claim 5, wherein the prediction model comprises: a network configured for bounding box regression for object detection and is configured for a class estimation of an object corresponding to a bounding box.” (SparseFusion, FIG.1-3, Sec. B.2 (LiDAR View Head), p. 15, ‘The same head is used for the LiDAR detector, view transformation, and the fusion branch … 1. It predicts the category of each object. … 2. It regresses the offset of the center of each object on the BEV plane … 4. It regresses the logarithms of the XYZ scale of the 3D bounding box’).
With regards to claim 7, SparseFusion in view of Carion teaches
“The method of claim 1, wherein the prediction model comprises: a neural network trained based on a loss function related to a difference between the predicted value of the input data item and ground truth data of the input data item.” (SparseFusion, FIG.1-3, Sec. 3.3, p. 5, ‘the predictions of each modality-specific detector are independently matched with the ground-truth using the Hungary algorithm [16]. The object classification is optimized with focal loss [26] and the 3D bounding box regression is optimized with L1 loss’ and Eq. (4), which includes the fusion-stage term L_detect^fusion; see also Carion, Sec. 3.1, pp. 5–6, Hungarian loss and ‘ℓ1 loss and the generalized IoU loss’ between predicted and ground-truth boxes).
With regards to claim 8, SparseFusion in view of Carion teaches
“The method of claim 2, wherein the projection networks and the prediction model comprise: a neural network trained based on a loss function related to a difference between the predicted value of the input data item and ground truth data of the input data item.” (SparseFusion, FIG.1-3, Sec. 3.3, pp. 5–6, ‘Our entire network is optimized using L = αL_init + βL_detect’, where L_detect includes L_detect^fusion (Eq. (4)); Sec. 4.2, p. 6, ‘we freeze the pretrained LiDAR components and train the entire fusion framework for 6 epochs’. The learnable projectors f_L, f_C and the fusion module with prediction head are trained end-to-end against ground truth).
With regards to claim 9, SparseFusion in view of Carion teaches
“The method of claim 2. … the ensemble query is based on respective transformations of the queries that have a same dimension (SparseFusion, FIG.1-3, Sec. B.1, p. 13, f_L, f_C ‘each consisting of a fully-connected layer and layer normalization’; Sec. 4.2, pp. 6–7, ‘The hidden dimensions in the entire model except the backbones are 128’).
SparseFusion does not explicitly detail “wherein the queries have different dimensions.”
However, Carion teaches “wherein the queries have different dimensions ….” (Carion, FIG.1-2, Sec. 3.2, p. 6 , “a 1x1 convolution reduces the channel dimension of the high-level activation map f from C to a smaller dimension d”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, and having the teachings of SparseFusion and Carion before him or her, to modify the process / system to generate prediction with ensemble models of SparseFusion to include different dimension as shown in Carion.
The motivation for doing so would have been for object detection (Carion, Abstract).
With regards to claim 15, SparseFusion in view of Carion teaches the method of claim 1 (see above). SparseFusion and Carion further teach
“A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1.” (SparseFusion, FIG.1-3, Sec. 4.2, p. 6, ‘Our implementation is based on the MMDetection3D framework’; Abstract, p. 1, ‘Our code will be made publicly available at https://github.com/yichen928/SparseFusion’; Sec. 4.2, p. 7, ‘We train our method on four NVIDIA A6000 GPUs’; Carion, Sec. 3.2, p. 6, ‘Inference code for DETR can be implemented in less than 50 lines in PyTorch’, and App. A.6, pp. 25–26, Listing 1. Software executed on GPU/processor hardware is necessarily stored as instructions in non-transitory memory accessible to the processor).
With regards to claim 16, SparseFusion teaches
“An apparatus comprising one or more processors configured to:” (SparseFusion, FIG.1-3, Sec. 4.2, p. 7, ‘We train our method on four NVIDIA A6000 GPUs’; Table 4 caption, p. 7, ‘The inference speed is measured on a single NVIDIA A6000 GPU’).
“generate, by transformer-based models, respective queries corresponding to an input data item inputted to the transformer-based models;” (SparseFusion, FIG.1-3, Sec. 3.1, p. 4, TransFusion-L LiDAR detector generating ‘updated object queries Q_L’ and deformable-DETR camera detector generating ‘updated queries Q_C^P’; Sec. B.1, p. 12, ‘transformer-based LiDAR detector’; see claim 1 mapping).
“obtain an ensemble query corresponding to the queries; and” (SparseFusion, FIG.1-3, Sec. 3.1, p. 4, Eq. (2), ‘We now concatenate the candidates together’).
“obtain a predicted value of the input data item by applying the ensemble query to a prediction model comprising a transformer decoder.” (SparseFusion, FIG.1-3, Sec. B.1, p. 13, Q_LC fed to self-attention + FFN and a prediction head predicting ‘the object category and 3D LiDAR view bounding boxes as the final results’; Carion, Sec. 3.2, p. 7, ‘Transformer decoder … The N object queries are transformed into an output embedding by the decoder. They are then independently decoded into box coordinates and class labels’).
SparseFusion does not explicitly detail that the prediction model comprises “a transformer decoder”; Carion teaches it. The rationale and motivation to combine are the same as set forth for claim 1.
With regards to claim 17, SparseFusion in view of Carion teaches
“The apparatus of claim 16, wherein the one or more processors are further configured to, in obtaining the ensemble query: obtain projected queries respectively corresponding to the queries, wherein the projected queries are obtained based on a projection network that projects the queries into the projected queries which are in a same feature space; and wherein the ensemble query comprises a concatenation of the projected queries.” (SparseFusion, FIG.1-3, Sec. 3.1, p. 4, Eq. (2), ‘f_L(·), f_C(·) are learnable projectors’; Sec. B.1, p. 13, ‘each consisting of a fully-connected layer and layer normalization. Then, we concatenate the candidates as Q_LC’; Fig. 2 caption, p. 5, ‘unified LiDAR space’).
With regards to claim 18, SparseFusion in view of Carion teaches
“The apparatus of claim 17, wherein the projection network and the prediction model comprise: a neural network trained based on a loss function related to a difference between the estimated value of the input data item and ground truth data of the input data item.” (SparseFusion, FIG.1-3, Sec. 3.3, pp. 5–6, Hungarian matching to ground truth, focal loss and L1 loss, Eq. (4) including L_detect^fusion, and ‘Our entire network is optimized using L’; Sec. 4.2, p. 6, ‘train the entire fusion framework’).
With regards to claim 19, SparseFusion in view of Carion teaches
“The apparatus of claim 17, wherein the one or more processors are further configured to: train the projection network and the prediction model based on a loss function related to a difference between the estimated value of the input data item and ground truth data of the input data item.” (SparseFusion, FIG.1-3, Sec. 4.2, pp. 6–7, ‘we freeze the pretrained LiDAR components and train the entire fusion framework for 6 epochs’ … ‘We train our method on four NVIDIA A6000 GPUs’; Sec. 3.3, pp. 5–6, loss against ground truth (Eq. (4))).
With regards to claim 20, SparseFusion in view of Carion teaches
“The apparatus of claim 16, wherein the input data item comprises an image and a point cloud,” (SparseFusion, FIG.1-3, Abstract, p. 1, ‘LiDAR-camera 3D object detection’; Sec. 4.1, p. 6, ‘point clouds collected using a 32-beam LiDAR and six images from multi-view cameras’).
“the predicted value of the input data item comprises a bounding box of an object detected in the input data item or a classification of the object.” (SparseFusion, FIG.1-3, Sec. B.1, p. 13, ‘predict the object category and 3D LiDAR view bounding boxes as the final results’).
Additional Relevant Art
The prior art made of record is considered pertinent to applicant’s disclosure and is recorded on Form PTO-892. Applicant is required under 37 C.F.R. § 1.111 (c) to consider these references fully when responding to this action, with particular attention paid to:
Khin et al., US-PGPUB NO.20260187474A1 [hereafter Khin] shows transformer model for handling queries.
Zhang et al., US-PATENT NO.11782793B2 [hereafter Zhang] shows decoder for prediction.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TSU-CHANG LEE whose telephone number is 571-272-3567. The fax number is 571-273-3567.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Omar Fernandez Rivas, can be reached 571-272-2589.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TSU-CHANG LEE/
Primary Examiner, Art Unit 2128
Prosecution Insights