DETAILED ACTION
Response to Arguments
Applicant’s amendments to drawings submitted 12/9/2025 have been recorded. As such Examiner’s previous objections to drawings are withdrawn.
Applicant has amended claims 1, 8, 9, 13, 20; added claims 21; and canceled claims 7, 12, and 19 leaving claims 1-6, 8-11, 13-18, and 20-21 currently pending.
Applicant’s arguments, filed 12/9/2025, with respect to the rejection(s) of claim(s) 7, 12, and 19 under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of He (He K, Chen X, Xie S, Li Y, Dollár P, Girshick R. Masked Autoencoders Are Scalable Vision Learners. arXiv e-prints. 2021 Nov:arXiv-2111.). Therefore this action is made NON-FINAL
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 4-6, 8-9, 13, 16-18, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bachmann (Bachmann R, Mizrahi D, Atanov A, Zamir A. MultiMAE: Multi-modal Multi-task Masked Autoencoders. arXiv preprint arXiv:2204.01678. 2022 Apr 4.) in view of He (He K, Chen X, Xie S, Li Y, Dollár P, Girshick R. Masked Autoencoders Are Scalable Vision Learners. arXiv e-prints. 2021 Nov:arXiv-2111.).
Regarding claim 13, Bachmann discloses A system for training a model, comprising: a hardware processor; (Bachmann Section 3.5 Pre-training details – found on p. 5; models are trained using A100 GPUs.) and a memory that stores a computer program which, when executed by the hardware processor, causes the hardware processor to: (Bachmann Abstract – found on p. 1; computer code is disclosed as available on their website.) pre-train a backbone model with a pre-training decoder, using an unlabeled dataset with multiple distinct sensor data modalities that derive from different sensor types; (Bachmann Section 2. Related Work – Self-training – found on p. 3; the use of creating pseudo labels from unlabeled data for training is disclosed. Fig. 2 and Section 3.4 Scene depth; RGB sensors and depth sensors are disclosed as two different sensor modalities to generate data. It can be seen in Fig. 2 that multiple modalities are used to pre-train MultiMAE (backbone).) and fine-tune the backbone model with an output decoder after pre-training, using a labeled dataset with the multiple distinct sensor data modalities. (Bachmann Fig. 2 and Section 4.3. Transfers with multiple modalities ¶1-2 – found on p. 6-7; it can be seen on the right of Fig.2 that Multi-modal fine-tuning is performed using depth and RGB images (from Section 3.4 Scene depth; they are different sensors).)
Bachmann does not explicitly disclose holding parameters of the backbone model fixed while he output decoder.
He, however discloses holding parameters of the backbone model fixed while he output decoder. (He Section 4.3 Partial Fine-tuning ¶1-2 – found on p. 7, table 1, and Fig. 1; fine-tuning the last several layers of the last transformer block while other layers are frozen is disclosed. The last transformer block would refer to the decoder from table 1 and Fig. 1.)
It would have been obvious, before the effective filing date of the claimed invention, to one of ordinary skill in the art to modify the system of Bachmann with the teachings of He by including partial fine-tuning of the decoder in order to achieve close to full fine-tuning accuracy in a shorter training time (He Section 4.3 Partial Fine-tuning ¶1-2 – found on p. 7).
Regarding claim 16, the combination of Bachmann and He discloses the claim limitations with respect to claim 13, as described above. They further disclose wherein the computer program further causes the hardware processor to mask a part of the unlabeled dataset. (Bachmann Fig. 2 and Section 2. Related Work – Self-training – found on p. 3; masked-out patches can be seen in Fig. 2 on the left for MulitMAE pre-training. Masked input modalities are also specifically mentioned.)
Regarding claim 17, the combination of Bachmann and He discloses the claim limitations with respect to claim 16, as described above. They further disclose wherein the computer program further causes the hardware processor to reconstruct the masked part of the unlabeled dataset to generate a reconstruction. (Bachmann Fig. 2 and Section 3.2. Decoders – found on p. 4; decoder reconstruct masked-out tokens from visible tokens.)
Regarding claim 18, the combination of Bachmann and He discloses the claim limitations with respect to claim 17, as described above. They further disclose wherein the computer program further causes the hardware processor to update parameters of the backbone model based on a reconstruction loss between the masked part of the unlabeled dataset and the reconstruction. (Bachmann Section 3.2. Decoders – found on p. 4; losses are computed using only the masked tokens. Section 3.4.Pseudo labeled multi-task training dataset – RGB and per-patch standardized RGB – found on p. 4-5; loss between ground truth and predicted pixels is used.)
Regarding claims 1 and 4-6, they are the corresponding method claims to claims 13 and 16-18 respectively and are rejected for similar reasons.
Regarding claim 9, Bachmann discloses A computer-implemented method for training a model, comprising: pre-training a backbone model with a pre-training decoder, using an unlabeled dataset with multiple distinct sensor data modalities that derive from different sensor types, (Bachmann Section 2. Related Work – Self-training – found on p. 3; the use of creating pseudo labels from unlabeled data for training is disclosed. Fig. 2 and Section 3.4 Scene depth; RGB sensors and depth sensors are disclosed as two different sensor modalities to generate data. It can be seen in Fig. 2 that multiple modalities are used to pre-train MultiMAE (backbone).) including: masking a part of the unlabeled dataset; reconstructing the masked part of the unlabeled dataset (Bachmann Fig. 2 and Section 2. Related Work – Self-training – found on p. 3; masked-out patches can be seen in Fig. 2 on the left for MulitMAE pre-training. Masked input modalities are also specifically mentioned.) to generate a reconstruction; (Bachmann Fig. 2 and Section 3.2. Decoders – found on p. 4; decoder reconstruct masked-out tokens from visible tokens.) and updating parameters of the backbone model based on a reconstruction loss between the masked part of the unlabeled dataset and the reconstruction; (Bachmann Section 3.2. Decoders – found on p. 4; losses are computed using only the masked tokens. Section 3.4.Pseudo labeled multi-task training dataset – RGB and per-patch standardized RGB – found on p. 4-5; loss between ground truth and predicted pixels is used.) and fine-tuning the backbone model with an output decoder after pre-training, using a labeled dataset with the multiple distinct sensor data modalities, (Bachmann Fig. 2 and Section 4.3. Transfers with multiple modalities ¶1-2 – found on p. 6-7; it can be seen on the right of Fig.2 that Multi-modal fine-tuning is performed using depth and RGB images (from Section 3.4 Scene depth; they are different sensors).) including optimizing parameters of the output decoder according to a task-specific loss function distinct from the reconstruction loss. (Bachmann Section 4.3. Transfers with multiple modalities¶2 – found on p. 6-7; no masking is performed in transfer learning therefore the loss function is different between pre-training and optimization of output decoders. Bachmann Appendix Section B.3. NYUv2 depth estimation ¶3 and B.4. Taskonomy dense regression tasks ¶2 – found on p. 17; disclose the reverse Huber loss and L1 loss respectively for depth estimation vs principal curvature, z-buffer depth, texture edges, occlusion edges, 2D keypoints, 3D keypoints, surface normal, and reshading. Showing that the loss function is task specific.)
Bachmann does not explicitly disclose holding parameters of the backbone model fixed while he output decoder.
He, however discloses holding parameters of the backbone model fixed while he output decoder. (He Section 4.3 Partial Fine-tuning ¶1-2 – found on p. 7, table 1, and Fig. 1; fine-tuning the last several layers of the last transformer block while other layers are frozen is disclosed. The last transformer block would refer to the decoder from table 1 and Fig. 1.)
It would have been obvious, before the effective filing date of the claimed invention, to one of ordinary skill in the art to modify the system of Bachmann with the teachings of He by including partial fine-tuning of the decoder in order to achieve close to full fine-tuning accuracy in a shorter training time (He Section 4.3 Partial Fine-tuning ¶1-2 – found on p. 7)
Regarding claim 20, the combination of Bachmann and He discloses the claim limitations with respect to claim 19, as described above. They further disclose wherein the computer program further causes the hardware processor to optimize the parameters of the output decoder according to a task-specific loss function distinct from a loss function used in the pre-training. (Bachmann Section 4.3. Transfers with multiple modalities¶2 – found on p. 6-7; no masking is performed in transfer learning therefore the loss function is different between pre-training and optimization of output decoders. Bachmann Appendix Section B.3. NYUv2 depth estimation ¶3 and B.4. Taskonomy dense regression tasks ¶2 – found on p. 17; disclose the reverse Huber loss and L1 loss respectively for depth estimation vs principal curvature, z-buffer depth, texture edges, occlusion edges, 2D keypoints, 3D keypoints, surface normal, and reshading. Showing that the loss function is task specific.)
Regarding claims 8, they are the corresponding method claim to claim 20 respectively and is rejected for similar reasons.
Claim(s) 2-3, 10-11, and 14-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bachmann (Bachmann R, Mizrahi D, Atanov A, Zamir A. MultiMAE: Multi-modal Multi-task Masked Autoencoders. arXiv preprint arXiv:2204.01678. 2022 Apr 4.) in view of He (He K, Chen X, Xie S, Li Y, Dollár P, Girshick R. Masked Autoencoders Are Scalable Vision Learners. arXiv e-prints. 2021 Nov:arXiv-2111.) and Arroyo (Pub. No. US20220114805A1).
Regarding claim 14, the combination of Bachmann and He discloses the claim limitations with respect to claim 13, as described above.
The combination of Bachmann and He does not explicitly disclose wherein the multiple distinct sensor data modalities include visual data from a video camera and point cloud data from a LiDAR sensor.
Arroyo, however discloses wherein the multiple distinct sensor data modalities include visual data from a video camera and point cloud data from a LiDAR sensor. (Arroyo ¶21; fine tuning a machine learning model that was pretrained on clear weather conditions on adverse weather conditions. ¶15; input devices include cameras and LIDAR.)
It would have been obvious, before the effective filing date of the claimed invention, to one of ordinary skill in the art to modify the system of the combination of Bachmann and He with the teachings of Arroyo by including visual inputs from video cameras and LIDAR in order to expand the usefulness of the system to other types of visual data.
Regarding claim 15, the combination of Bachmann, He, and Arroyo disclose the claim limitations with respect to claim 14, as described above. They further disclose wherein the visual data includes data from multiple video cameras on a given vehicle. (Arroyo ¶15; image capture devices/ cameras for automated driving perception systems is disclosed. Meaning multiple cameras on a vehicle.)
It would have been obvious, before the effective filing date of the claimed invention, to one of ordinary skill in the art to modify the system of the combination of Bachmann, He, and Arroyo by further including multiple video cameras on a vehicle in order to expand the usefulness of the system to self-driving vehicles.
Regarding claims 2-3 they are the corresponding method claims to claims 14-15 respectively and are rejected for similar reasons.
Regarding claims 10- 11 the combination of Bachmann and He discloses the claim limitations with regards to claim 9 as described above. Additional limitations are similar to claims 2-3 and are rejected for similar reasons.
Claim(s) 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bachmann (Bachmann R, Mizrahi D, Atanov A, Zamir A. MultiMAE: Multi-modal Multi-task Masked Autoencoders. arXiv preprint arXiv:2204.01678. 2022 Apr 4.) in view of He (He K, Chen X, Xie S, Li Y, Dollár P, Girshick R. Masked Autoencoders Are Scalable Vision Learners. arXiv e-prints. 2021 Nov:arXiv-2111.) and Dipple (Dipple J, Lenga M, Goerttler T, Obermayer K, Höhne J. Transfer learning for segmentation problems: choose the right encoder and skip the decoder. arXiv preprint arXiv:2207.14508. 2022 Jul 29.).
Regarding claim 21, the combination of Bachmann and He disclose the claim limitations with regards to claim 1, as described above.
The combination of Bachmann and He does not explicitly disclose wherein parameters of the output decoder are initialized with parameters of the pre-training decoder.
Dipple, however, discloses wherein parameters of the output decoder are initialized with parameters of the pre-training decoder. (Dipple Section 3 Do we need a Pretrained Decoder? – found on p. 5 and Fig 1; initializing a pretrained encoder decoder pair is disclosed. This matches scenario 3 from Fig. 1.)
It would have been obvious, before the effective filing date of the claimed invention, to one of ordinary skill in the art to modify the system of the combination of Bachmann and He with the teachings of Dipple by initializing the output decoder with the parameters of the pre-training decoder in order to yield optimal model performance with less training steps (Dipple Section 3 Do we need a Pretrained Decoder? – found on p. 5).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Yosinski (Yosinski J, Clune J, Bengio Y, Lipson H. How transferable are features in deep neural networks?. Advances in neural information processing systems. 2014;27.) Leaving base layers frozen during fine tuning so as to not propagate errors is explained (see specifically the introduction.)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MEREDITH TAYLOR whose telephone number is (571)270-5805. The examiner can normally be reached M-Th 7:30-5. Examiner’s email is Meredith.taylor@uspto.gov.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent Rudolph can be reached at (571)272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MEREDITH TAYLOR/Examiner, Art Unit 2671
/VINCENT RUDOLPH/Supervisory Patent Examiner, Art Unit 2671
Prosecution Insights