DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Priority is acknowledged from Provisional application 63/449,285 with a filing date of 03/01/2023.
Information Disclosure Statement
The information disclosure statement (“IDS”) filed on 03/25/2024 was reviewed and the listed references were noted.
Drawings
The 6-page drawings have been considered and placed on record in the file.
Claim Objections
Claim 1 is objected to because of the following informalities: the claim recites “…extracting, a motion-guided slot learning mechanism, mid-level features…” in which it is assumed that a typographical error has occurred. For compact prosecution, the limitation is being interpreted as “…extracting, via a motion-guided slot learning mechanism, mid-level features…”. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 5 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention. Examiner respectfully requests that appropriate corrections be made to clarify the scope of the claims.
Claim 5 recites the limitation “the agent”. There is insufficient antecedent basis for these limitations in the claim.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-3, 5-10, 12-16, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Gao et al. (US 20210374950 A1) in view of Kipf et al. ("Conditional object-centric learning from video” 2021), Pronovost (US 20240104934 A1), Harikumar et al. (US 20230419551 A1), and Hu et al. (CN 111080671 A).
Regarding Claim 1, Gao teaches "A method for learning a representation of a sequence of frames, comprising: encoding, via an encoder network, the sequence of frames to obtain a set of feature maps"; (Gao, Abstract, teaches receiving a set of images and determining a sequence of image patches based on the set of images wherein a first learning network including an encoder is configured to extract feature maps based on the sequence of image patches, i.e., encoder network encoding a sequence of frames to obtain a set of feature maps).
However, Gao does not explicitly teach "extracting, a motion-guided slot learning mechanism, mid-level features from the set of feature maps; quantizing the mid-level features via a vector quantization process to obtain a set of tokens; decoding, via a decoder network, the tokens to obtain a reconstructed sequence of frames; and optimizing a combination of a reconstruction loss and a motion loss to train the encoder and decoder networks”.
In an analogous field of endeavor, Kipf teaches "extracting, a motion-guided slot learning mechanism, mid-level features from the set of feature maps"; (Kipf, Figure 1 and Section 2 Slot Attention for Video, teaches a corresponding video frame is passed through a CNN encoder for each time-step wherein the resulting grid of visual features is flattened into a set of vectors with a size of the flattened grid and a dimensionality of the CNN feature maps wherein the corrector updates the slot representation based on the visual features from the encoder using the iterative attention mechanism introduced in Slot Attention wherein the sole prediction target is optical flow for each individual frame, i.e., extracting mid-level features from the set of feature maps being the resulting grid of visual features which are flattened into a set of vectors by a motion-guided slot learning mechanism being the updating of the slot representations of the visual features using the iteration attention mechanism of Slot attention wherein the model is trained to predict optical flow or motion).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Gao by including the extraction of features from feature maps using a slot learning mechanism guided by motion taught by Kipf. One of ordinary skill in the art would be motivated to combine the references since it improves segmentation (Kipf, Abstract, teaches the motivation of combination to be to significantly improve instance segmentation).
However, the combination of references of Gao in view of Kipf does not explicitly teach “quantizing the mid-level features via a vector quantization process to obtain a set of tokens; decoding, via a decoder network, the tokens to obtain a reconstructed sequence of frames; and optimizing a combination of a reconstruction loss and a motion loss to train the encoder and decoder networks”.
In an analogous field of endeavor, Pronovost teaches "quantizing the mid-level features via a vector quantization process to obtain a set of tokens"; (Pronovost, Paras. 21, 40, 59, and 78, teaches a quantizer to receive feature vectors as input and output discretized feature vectors wherein the codebook can convert a set of discretized feature vectors from a quantizer to tokens, i.e., quantizing the features via a vector quantization process to obtain a set of tokens).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Gao and Kipf wherein the features are mid-level features by including the quantizing of the features to obtain a set of tokens taught by Pronovost. One of ordinary skill in the art would be motivated to combine the references since it considers object trajectory to improve vehicle safety (Pronovost, Para. 13, teaches the motivation of combination to be to consider object trajectory during vehicle planning and improve vehicle safety as a vehicle navigates).
However, the combination of references of Gao in view of Kipf and Pronovost does not explicitly teach “decoding, via a decoder network, the tokens to obtain a reconstructed sequence of frames; and optimizing a combination of a reconstruction loss and a motion loss to train the encoder and decoder networks”.
In an analogous field of endeavor, Harikumar teaches "decoding, via a decoder network, the tokens to obtain a reconstructed sequence of frames"; (Harikumar, Paras. 6, 35, 66, 68, 83, and 121, teaches one or more images are input to an encoder to learn discrete image tokens which are decodable into a reconstructed image having one or more image features corresponding to the one or more edge features by the decoder, i.e., decoder network decoding tokens to obtain reconstructed images).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Gao, Kipf, and Pronovost wherein the input images are a sequence of image frames by including the decoding of the tokens to reconstruct the images taught by Harikumar. One of ordinary skill in the art would be motivated to combine the references since it predicts tokens to represent new images (Harikumar, Para. 1, teaches the motivation of combination to be to predict tokens which are representative of new images).
However, the combination of references of Gao in view of Kipf, Pronovost, and Harikumar does not explicitly teach “and optimizing a combination of a reconstruction loss and a motion loss to train the encoder and decoder networks”.
In an analogous field of endeavor, Hu teaches "and optimizing a combination of a reconstruction loss and a motion loss to train the encoder and decoder networks"; (Hu, Claims 2 and 3 and Pg. 9 Para. 1, teaches using a loss function and Adam to optimize the training network wherein the loss function includes a sum or combination of L_rec representing the reconstruction loss and L_disp and L_mob represent the displacement error or loss and the regression error or loss of the motion parameter respectively, i.e., optimize a combination of reconstruction and motion loss to train the network).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Gao, Kipf, Pronovost, and Harikumar wherein the network includes encoder and decoder networks by including the training of the network by optimizing a combination of reconstruction and motion loss taught by Hu. One of ordinary skill in the art would be motivated to combine the references since it optimizes the training network (Hu, Pg. 9 Para. 1, teaches the motivation of combination to be to optimize the training network).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date.
Regarding Claim 2, the combination of references of Gao in view of Kipf, Pronovost, Harikumar, and Hu teaches "The method of claim 1, wherein: the motion-guided slot learning mechanism determines a motion map from the sequence of frames; and the motion map guides the slot learning process"; (Kipf, Section 2 Slot Attention for Video, teaches the corrector updates the slot representation based on the visual features from the encoder using the iterative attention mechanism introduced in Slot Attention wherein slot representation after application of the corrector are decoded to produce per-slot RGB predictions of the optical flow in which the sole prediction target is optical flow for each individual frame, i.e., the motion-guided slot learning mechanism determines a motion map being the per-slot production of the optical flow wherein the motion map or optical flow guides the learning process for training as the prediction target).
The proposed combination as well as the motivation for combining the Gao in view of Kipf, Pronovost, Harikumar, and Hu references presented in the rejection of Claim 1, applies to claim 2. Thus, the method recited in claim 2 is met by Gao in view of Kipf, Pronovost, Harikumar, and Hu.
Regarding Claim 3, the combination of references of Gao in view of Kipf, Pronovost, Harikumar, and Hu teaches "The method of claim 1, wherein: the vector quantization process trains a Vector Quantized-Variational AutoEncoder (VQ-VAE) to learn a latent embedding space of quantized feature vectors, and the VQ-VAE quantizes the mid-level features"; (Harikumar, Paras. 66-68, teaches the autoencoder being a Vector Quantized Variational Autoencoder configured to learn discrete latent representation of an image such as image tokens and compress the image into a quantized codebook used to predict subsequent image tokens in which the codebook is a mapping between a token and a learned vector representation or an embedding which is learned while training an encoder, i.e., vector quantization process trains a VQ-VAE to learn latent embedding space of quantized feature vectors, wherein a codebook can also be used to quantize the bottleneck referring to the most compressed or most encoded version of the input image, i.e., VQ-VAE quantizes the mid-level features output from the encoder).
The proposed combination as well as the motivation for combining the Gao in view of Kipf, Pronovost, Harikumar, and Hu references presented in the rejection of Claim 1, applies to claim 3. Thus, the method recited in claim 3 is met by Gao in view of Kipf, Pronovost, Harikumar, and Hu.
Regarding Claim 5, the combination of references of Gao in view of Kipf, Pronovost, Harikumar, and Hu teaches "The method of claim 1, wherein the agent is an autonomous or semi-autonomous vehicle"; (Pronovost, Para. 23, teaches determining the top-down view based on the sensor data captured from or associated with a sensor of an autonomous vehicle in the environment, i.e., agent is an autonomous vehicle).
The proposed combination as well as the motivation for combining the Gao in view of Kipf, Pronovost, Harikumar, and Hu references presented in the rejection of Claim 1, applies to claim 5. Thus, the method recited in claim 5 is met by Gao in view of Kipf, Pronovost, Harikumar, and Hu.
Regarding Claim 6, the combination of references of Gao in view of Kipf, Pronovost, Harikumar, and Hu teaches "The method of claim t, further comprising controlling the agent to navigate through an environment based on training the encoder and decoder networks"; (Pronovost, FIG. 3 and Paras. 17, 56-57, and 91, teaches the model determining a response by the vehicle to the object trajectory in the environment and control the vehicle in the environment based at least in part on the response and wherein techniques can be performed as the vehicle navigates the environment and wherein input data representing object trajectories associated with one or more objects, object state data, and scene data can be input into an encoder in which the output data from the decoder can be used to train the codebook, the quantizer, the encoder, and the decoder, i.e., controlling the agent to navigate through an environment based on training of encoder and decoder networks).
The proposed combination as well as the motivation for combining the Gao in view of Kipf, Pronovost, Harikumar, and Hu references presented in the rejection of Claim 1, applies to claim 6. Thus, the method recited in claim 6 is met by Gao in view of Kipf, Pronovost, Harikumar, and Hu.
Regarding Claim 7, the combination of references of Gao in view of Kipf, Pronovost, Harikumar, and Hu teaches "The method of claim 1, further comprising capturing the sequence of frames via one or more sensors associated with an agent"; (Pronovost, Para. 35, teaches a vehicle computing device associated with the vehicle may detect objects based on sensor data received from one or more sensors such as cameras mounted on the vehicle, i.e., capturing a sequence of frames via one or more sensors associated with an agent being the sensors capturing sensor data associated with the vehicle).
The proposed combination as well as the motivation for combining the Gao in view of Kipf, Pronovost, Harikumar, and Hu references presented in the rejection of Claim 1, applies to claim 7. Thus, the method recited in claim 7 is met by Gao in view of Kipf, Pronovost, Harikumar, and Hu.
Claim 8 recites a computer-readable storage medium storing a program with instructions corresponding to the steps recited in Claim 1. Therefore, the recited programming instructions of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Gao in view of Kipf, Pronovost, Harikumar, and Hu references, presented in rejection of Claim 1, apply to this claim. Finally, the combination of the Gao in view of Kipf, Pronovost, Harikumar, and Hu references discloses a computer readable storage medium (for example, see Gao, Paragraph 11).
Claim 9 recites a computer-readable storage medium storing a program with instructions corresponding to the steps recited in Claim 2. Therefore, the recited programming instructions of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Gao in view of Kipf, Pronovost, Harikumar, and Hu references, presented in rejection of Claim 1, apply to this claim. Finally, the combination of the Gao in view of Kipf, Pronovost, Harikumar, and Hu references discloses a computer readable storage medium (for example, see Gao, Paragraph 11).
Claim 10 recites a computer-readable storage medium storing a program with instructions corresponding to the steps recited in Claim 3. Therefore, the recited programming instructions of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Gao in view of Kipf, Pronovost, Harikumar, and Hu references, presented in rejection of Claim 1, apply to this claim. Finally, the combination of the Gao in view of Kipf, Pronovost, Harikumar, and Hu references discloses a computer readable storage medium (for example, see Gao, Paragraph 11).
Claim 12 recites a computer-readable storage medium storing a program with instructions corresponding to the steps recited in Claim 5. Therefore, the recited programming instructions of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Gao in view of Kipf, Pronovost, Harikumar, and Hu references, presented in rejection of Claim 1, apply to this claim. Finally, the combination of the Gao in view of Kipf, Pronovost, Harikumar, and Hu references discloses a computer readable storage medium (for example, see Gao, Paragraph 11).
Claim 13 recites a computer-readable storage medium storing a program with instructions corresponding to the steps recited in Claim 6. Therefore, the recited programming instructions of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Gao in view of Kipf, Pronovost, Harikumar, and Hu references, presented in rejection of Claim 1, apply to this claim. Finally, the combination of the Gao in view of Kipf, Pronovost, Harikumar, and Hu references discloses a computer readable storage medium (for example, see Gao, Paragraph 11).
Claim 14 recites a system with elements corresponding to the steps recited in Claim 1. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Gao in view of Kipf, Pronovost, Harikumar, and Hu references, presented in rejection of Claim 1, apply to this claim. Finally, the combination of the Gao in view of Kipf, Pronovost, Harikumar, and Hu references discloses a processor and a memory for executing instructions (for example, see Gao, Paragraphs 10 and 77).
Claim 15 recites a system with elements corresponding to the steps recited in Claim 2. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Gao in view of Kipf, Pronovost, Harikumar, and Hu references, presented in rejection of Claim 1, apply to this claim. Finally, the combination of the Gao in view of Kipf, Pronovost, Harikumar, and Hu references discloses a processor and a memory for executing instructions (for example, see Gao, Paragraphs 10 and 77).
Claim 16 recites a system with elements corresponding to the steps recited in Claim 3. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Gao in view of Kipf, Pronovost, Harikumar, and Hu references, presented in rejection of Claim 1, apply to this claim. Finally, the combination of the Gao in view of Kipf, Pronovost, Harikumar, and Hu references discloses a processor and a memory for executing instructions (for example, see Gao, Paragraphs 10 and 77).
Claim 18 recites a system with elements corresponding to the steps recited in Claim 5. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Gao in view of Kipf, Pronovost, Harikumar, and Hu references, presented in rejection of Claim 1, apply to this claim. Finally, the combination of the Gao in view of Kipf, Pronovost, Harikumar, and Hu references discloses a processor and a memory for executing instructions (for example, see Gao, Paragraphs 10 and 77).
Claim 19 recites a system with elements corresponding to the steps recited in Claim 6. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Gao in view of Kipf, Pronovost, Harikumar, and Hu references, presented in rejection of Claim 1, apply to this claim. Finally, the combination of the Gao in view of Kipf, Pronovost, Harikumar, and Hu references discloses a processor and a memory for executing instructions (for example, see Gao, Paragraphs 10 and 77).
Claim 20 recites a system with elements corresponding to the steps recited in Claim 7. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Gao in view of Kipf, Pronovost, Harikumar, and Hu references, presented in rejection of Claim 1, apply to this claim. Finally, the combination of the Gao in view of Kipf, Pronovost, Harikumar, and Hu references discloses a processor and a memory for executing instructions (for example, see Gao, Paragraphs 10 and 77).
Claims 4, 11, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Gao in view of Kipf, Pronovost, Harikumar, Hu, and Wang et al. (US 20240169479 A1).
Regarding Claim 4, the combination of references of Gao in view of Kipf, Pronovost, Harikumar, and Hu does not explicitly teach "The method of claim 1, wherein the decoder network includes a self-attention mechanism to learn temporal dependencies in the sequence of frames".
In an analogous field of endeavor, Wang teaches "The method of claim 1, wherein the decoder network includes a self-attention mechanism to learn temporal dependencies in the sequence of frames"; (Wang, Para. 16, teaches generating key frames using a latent diffusion model including a convolutional network architecture such as a 3D U-Net decoder wherein the key frame generation process implements a directed self-attention design to provide temporal dependency among the generated key frames, i.e., decoder network includes a self-attention mechanism to learn temporal dependencies in the sequence of frames).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Gao, Kipf, Pronovost, Harikumar, and Hu wherein the frames are a sequence of frames by including the decoder having a self-attention mechanism to learn temporal dependencies in the frames taught by Wang. One of ordinary skill in the art would be motivated to combine the references since it improves data efficiency and sampling speed (Wang, Para. 12, teaches the motivation of combination to be to improve data efficiency and sampling speed).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date.
Claim 11 recites a computer-readable storage medium storing a program with instructions corresponding to the steps recited in Claim 4. Therefore, the recited programming instructions of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Gao in view of Kipf, Pronovost, Harikumar, Hu, and Wang references, presented in rejection of Claim 4, apply to this claim. Finally, the combination of the Gao in view of Kipf, Pronovost, Harikumar, Hu, and Wang references discloses a computer readable storage medium (for example, see Gao, Paragraph 11).
Claim 17 recites a system with elements corresponding to the steps recited in Claim 4. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Gao in view of Kipf, Pronovost, Harikumar, Hu, and Wang references, presented in rejection of Claim 4, apply to this claim. Finally, the combination of the Gao in view of Kipf, Pronovost, Harikumar, Hu, and Wang references discloses a processor and a memory for executing instructions (for example, see Gao, Paragraphs 10 and 77).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW STEVEN BUDISALICH whose telephone number is (703)756-5568. The examiner can normally be reached Monday - Friday 8:30am-5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571) 272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW S BUDISALICH/Examiner, Art Unit 2662
/AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662