DETAILED ACTION
Notice of AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements submitted on 1/2/2024 has been considered.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claim 6 is rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Claim 6 recites “wherein the one or more processors are configured to freeze a plurality of parameters of the first neural network after training of the first neural network during the first training stage is complete and before the second training stage to train the second neural network is initiated”. In view of this limitation, the freezing of parameters occurs (a) after training of the first neural network, (b) during the first training stage is complete, and (c) before the second training stage. It’s unclear how freezing can be performed “during the first training stage is complete.” The examiner suggests amending this to delete “during the first training stage” consistent with dependent claims 14 and 20.
Appropriate correction is required.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 3-4, 6, 9, 11-12, 14, 17, and 20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by US 20230260521 A1, hereinafter referenced as SLOCUM.
Regarding Claim 1
SLOCUM teaches:
A system comprising: a memory; and one or more processors coupled to the memory, the one or more processors being configured to: (SLOCUM, para. 0008: “Another aspect of the disclosure provides a system including data processing hardware and memory hardware in communication with the data processing hardware and storing instructions executable on the data processing hardware that cause the data processing hardware to perform operations.”;
SLOCUM, para. 0063: “The processor (i.e., data processing hardware) 510 can process instructions stored on the memory hardware (i.e., the memory 520) for execution within the computing device 500, including instructions stored in the memory 520”)
train, using training data, a first neural network during a first training stage; (SLOCUM, para. 0004: “Here, the multi-task neural network model includes the trained audio encoder, the trained SID head overlain on the trained audio encoder, and the trained SSD head overlain on the trained audio encoder.”;
SLOCUM, para. 0032: “Advantageously, the multitask neural network model 130 includes a shared audio encoder 150 that receives the audio data 120 as input, and generates, as output, audio encodings 152 corresponding to a hidden feature representation of the audio data 120 encoded by the audio encoder 150. Each audio encoding 152 output from the audio encoder 150 may be input to respective ones of a SID head 154, a SSD head 160, and a RAD head 161 that branch from the audio encoder 150. Accordingly, the audio encoder 150 may serve as a “body” of the multitask neural network model 130 where the audio encoder 150 and the SID head 154 form a SID branch corresponding to a SID model 151 (FIG. 3A), the audio encoder 150 and the SSD head 160 form a SSD branch corresponding to a SSD model 153 (FIG. 3B), and the audio encoder 150 and the RAD head 161 form a RAD branch corresponding to a RAD model.”;
SLOCUM, para. 0046: “With continued reference to FIG. 3A, the first stage 300a of the training process 300 trains the untrained audio encoder 150 on the training utterances 320, 320a-n to generate, as output from the audio encoder 150, audio encodings 152 for the training utterances 320.”;
Examiner’s Note: training utterances 320, 320a-n correspond to recited “training data”, the audio encoder 150 corresponds to the recited “first neural network”, and the first stage 300a corresponds to the recited “first training stage”)
generate, by the first neural network, one or more outputs after the first training stage and training of the first neural network are complete; (SLOCUM, para. 0049: “Referring to FIG. 3B, the second training stage 300b (i.e., a SSD training stage) receives/obtains the SID model 151 trained by the first training stage 300a and uses the trained audio encoder 150 to train the SSD head 160 to learn how to predict whether input audio corresponds to human-generated bona fide speech or synthetic/synthesized speech. During the second training stage 300b, parameters/weights of the trained audio encoder 150 are frozen and the SSD head 160 is trained (or fine-tuned) to learn how to project SSD classification scores 162 indicating whether or not input audio includes human-generated bona fide speech or synthetic speech. As will become apparent, the SSD head 160 receives, as input, audio encodings 152 generated by the trained audio encoder 150 for SSD training utterances 330 and generates, as output, a respective SSD classification score 162 for each SSD training utterance 330.”;
Examiner’s Note: after the first stage, the audio encoder 150 has been trained, and now audio encodings 152 are output from the audio encoder 150)
based on a determination that the first training stage and training of the first neural network are complete, provide the one or more outputs to a second neural network that has an input data dependency comprising data generated by the first neural network; and (SLOCUM, para. 0004: “The operations also include receiving a plurality of synthetic speech detection (SSD) training utterances that include a set of human-originated speech samples and a set of synthetic speech samples. The operations also include training, using the trained audio encoder, a SSD head on the SSD training utterances to learn to detect the presence of synthetic speech in audio encodings encoded by the trained audio encoder, wherein the SSD head is overlain on the trained audio encoder. ... The operations also include providing, for execution on a computing device, a multitask neural network model for performing both SID tasks and SSD tasks on input audio data in parallel.”;
SLOCUM, para. 0032: “Accordingly, the audio encoder 150 may serve as a “body” of the multitask neural network model 130 where the audio encoder 150 and the SID head 154 form a SID branch corresponding to a SID model 151 (FIG. 3A), the audio encoder 150 and the SSD head 160 form a SSD branch corresponding to a SSD model 153 (FIG. 3B), and the audio encoder 150 and the RAD head 161 form a RAD branch corresponding to a RAD model.”
SLOCUM, para. 0049: “Referring to FIG. 3B, the second training stage 300b (i.e., a SSD training stage) receives/obtains the SID model 151 trained by the first training stage 300a and uses the trained audio encoder 150 to train the SSD head 160 to learn how to predict whether input audio corresponds to human-generated bona fide speech or synthetic/synthesized speech. During the second training stage 300b, parameters/weights of the trained audio encoder 150 are frozen and the SSD head 160 is trained (or fine-tuned) to learn how to project SSD classification scores 162 indicating whether or not input audio includes human-generated bona fide speech or synthetic speech. As will become apparent, the SSD head 160 receives, as input, audio encodings 152 generated by the trained audio encoder 150 for SSD training utterances 330 and generates, as output, a respective SSD classification score 162 for each SSD training utterance 330.”;
Examiner’s Note: the end of the first training stage and beginning of the second training stage corresponds to the recited “based on a determination that the first training stage and training of the first neural network are complete” and the audio encodings 152 are input to the SSD head 160 (corresponding to recited “second neural network”), where as shown in Fig. 3B, there is a direct data dependency between the trained audio encoder 150 and the SSD Head 160)
based on the determination that the first training stage and training of the first neural network are complete, train, using the one or more outputs from the first neural network, the second neural network during a second training stage initiated after completion of the first training stage. (SLOCUM, para. 0049: “Referring to FIG. 3B, the second training stage 300b (i.e., a SSD training stage) receives/obtains the SID model 151 trained by the first training stage 300a and uses the trained audio encoder 150 to train the SSD head 160 to learn how to predict whether input audio corresponds to human-generated bona fide speech or synthetic/synthesized speech. During the second training stage 300b, parameters/weights of the trained audio encoder 150 are frozen and the SSD head 160 is trained (or fine-tuned) to learn how to project SSD classification scores 162 indicating whether or not input audio includes human-generated bona fide speech or synthetic speech. As will become apparent, the SSD head 160 receives, as input, audio encodings 152 generated by the trained audio encoder 150 for SSD training utterances 330 and generates, as output, a respective SSD classification score 162 for each SSD training utterance 330.”;
Examiner’s Note: the end of the first training stage and beginning of the second training stage corresponds to the recited “based on the determination that the first training stage and training of the first neural network are complete” and the audio encodings 152 are input to the SSD head 160 (corresponding to recited “second neural network”), where as shown in Fig. 3B, there is a direct data dependency between the trained audio encoder 150 and the SSD Head 160, during second training stage 300b)
Regarding Claim 3
SLOCUM teaches the system of claim 1. SLOCUM further teaches:
wherein the first neural network and the second neural network comprise a multi-network system and (SLOCUM, para. 0004: “Here, the multi-task neural network model includes the trained audio encoder, the trained SID head overlain on the trained audio encoder, and the trained SSD head overlain on the trained audio encoder.”;
SLOCUM, para. 0032: “Advantageously, the multitask neural network model 130 includes a shared audio encoder 150 that receives the audio data 120 as input, and generates, as output, audio encodings 152 corresponding to a hidden feature representation of the audio data 120 encoded by the audio encoder 150. Each audio encoding 152 output from the audio encoder 150 may be input to respective ones of a SID head 154, a SSD head 160, and a RAD head 161 that branch from the audio encoder 150. Accordingly, the audio encoder 150 may serve as a “body” of the multitask neural network model 130 where the audio encoder 150 and the SID head 154 form a SID branch corresponding to a SID model 151 (FIG. 3A), the audio encoder 150 and the SSD head 160 form a SSD branch corresponding to a SSD model 153 (FIG. 3B), and the audio encoder 150 and the RAD head 161 form a RAD branch corresponding to a RAD model.”;
Examiner’s Note: the audio encoder 150 and SSD Head 160 are both neural network components of multi-task neural network 130)
wherein, during an inference stage of the multi-network system, the second neural network is configured to run at some point sequentially after the first neural network.
(SLOCUM, para. 0032: “Each audio encoding 152 output from the audio encoder 150 may be input to respective ones of a SID head 154, a SSD head 160, and a RAD head 161 that branch from the audio encoder 150.”
PNG
media_image1.png
184
500
media_image1.png
Greyscale
Examiner’s Note: as shown by Fig. 1, the arrow from audio encoder 150 to SSD Head 160 shows that SSD Head 160 is sequentially after the audio encoder 150)
Regarding Claim 4
SLOCUM teaches the system of claim 1. SLOCUM further teaches:
wherein, based on the input data dependency, the second neural network is configured to use an output of the first neural network as an input to the second neural network or receive input data formed at least partly based on the output of the first neural network. (SLOCUM, para. 0032: “Each audio encoding 152 output from the audio encoder 150 may be input to respective ones of a SID head 154, a SSD head 160, and a RAD head 161 that branch from the audio encoder 150.”
PNG
media_image1.png
184
500
media_image1.png
Greyscale
Examiner’s Note: as shown by Fig. 1, the arrow from audio encoder 150 to SSD Head 160 shows that SSD Head 160 is sequentially after the audio encoder 150 and utilizes audio encoded outputs 152)
Regarding Claim 6
SLOCUM teaches the system of claim 1. SLOCUM further teaches:
wherein the one or more processors are configured to freeze a plurality of parameters of the first neural network after training of the first neural network during the first training stage is complete and before the second training stage to train the second neural network is initiated, and
(SLOCUM, para. 0006: “In additional examples, parameters/weights of the trained audio encoder are frozen while training the SSD head on the SSD training utterances. “;
SLOCUM, para. 0049: “Referring to FIG. 3B, the second training stage 300b (i.e., a SSD training stage) receives/obtains the SID model 151 trained by the first training stage 300a and uses the trained audio encoder 150 to train the SSD head 160 to learn how to predict whether input audio corresponds to human-generated bona fide speech or synthetic/synthesized speech. During the second training stage 300b, parameters/weights of the trained audio encoder 150 are frozen and the SSD head 160 is trained (or fine-tuned) to learn how to project SSD classification scores 162 indicating whether or not input audio includes human-generated bona fide speech or synthetic speech. As will become apparent, the SSD head 160 receives, as input, audio encodings 152 generated by the trained audio encoder 150 for SSD training utterances 330 and generates, as output, a respective SSD classification score 162 for each SSD training utterance 330.”;
Examiner’s Note: after the first training session, the parameters/weights of the trained audio encoder 150 are frozen)
wherein freezing the plurality of parameters of the first neural network prevents any updates to the plurality of parameters of the first neural network during the second training stage. (SLOCUM, para. 0006: “In additional examples, parameters/weights of the trained audio encoder are frozen while training the SSD head on the SSD training utterances. “;
SLOCUM, para. 0049: “Referring to FIG. 3B, the second training stage 300b (i.e., a SSD training stage) receives/obtains the SID model 151 trained by the first training stage 300a and uses the trained audio encoder 150 to train the SSD head 160 to learn how to predict whether input audio corresponds to human-generated bona fide speech or synthetic/synthesized speech. During the second training stage 300b, parameters/weights of the trained audio encoder 150 are frozen and the SSD head 160 is trained (or fine-tuned) to learn how to project SSD classification scores 162 indicating whether or not input audio includes human-generated bona fide speech or synthetic speech. As will become apparent, the SSD head 160 receives, as input, audio encodings 152 generated by the trained audio encoder 150 for SSD training utterances 330 and generates, as output, a respective SSD classification score 162 for each SSD training utterance 330.”;
Examiner’s Note: after the first training session, the parameters/weights of the trained audio encoder 150 are frozen so that those parameters/weights are not changed during the second stage of training)
Claim 9 recites a method that corresponds to the system of claim 1 and is therefore rejected for the same reasons explained with respect to claim 1.
Claim 11 depends from claim 9 and recites a method that corresponds to the system of claim 3 and is therefore rejected for the same reasons explained with respect to claims 3 and 9.
Claim 12 depends from claim 9 and recites a method that corresponds to the system of claim 4 and is therefore rejected for the same reasons explained with respect to claims 4 and 9.
Claim 14 depends from claim 9 and recites a method that corresponds to the system of claim 6 and is therefore rejected for the same reasons explained with respect to claims 6 and 9.
Regarding Claim 17
SLOCUM teaches:
A non-transitory computer-readable medium having stored thereon instructions which, when executed by one or more processors, cause the one or more processors to: (SLOCUM, para. 0008: “Another aspect of the disclosure provides a system including data processing hardware and memory hardware in communication with the data processing hardware and storing instructions executable on the data processing hardware that cause the data processing hardware to perform operations.”;
SLOCUM, para. 0064: “The memory 520 stores information non-transitorily within the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit(s), or non-volatile memory unit(s). The non-transitory memory 520 may be physical devices used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by the computing device 500.”)
The remaining limitations correspond to the system of claim 1, and therefore this claim 17 is rejected for the same reasons explained above with respect to claim 1.
Claim 20 depends from claim 17 and recites a non-transitory computer-readable medium that corresponds to the system of claim 6 and is therefore rejected for the same reasons explained with respect to claims 6 and 17.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 2, 7, 10, 15, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over SLOCUM in view of Dijkstra, Klaas, et al. "Centroidnet: A deep neural network for joint object localization and counting." Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Cham: Springer International Publishing, 2019, hereinafter referenced as DJIKSTRA.
Under MPEP 2131.02, the examiner further cites to Ronneberger, Olaf, et al. "U-net: Convolutional networks for biomedical image segmentation." International Conference on Medical image computing and computer-assisted intervention. Cham: Springer international publishing, 2015, hereinafter referenced as RONNEBERGER, to explain the meaning of the “U-Net” term used in the SLOCUM reference, where RONNEBERGER is citation [16] in SLOCUM.
Regarding Claim 2
SLOCUM teaches the system of claim 1. However, SLOCUM fails to explicitly teach:
wherein the first neural network comprises a segmentation network and the second neural network comprises a centroid prediction network.
However, in a related field of endeavor (neural networks, see p. 585, section 1), DIJKSTRA teaches and makes obvious:
wherein the first neural network comprises a segmentation network and the second neural network comprises a centroid prediction network. (DIJKSTRA, p. 586, section 1: “This paper introduces CentroidNet which has been specifically designed for joint object counting and localization. CentroidNet produces centroids of image objects rather than bounding boxes. The key idea behind CentroidNet is to combine image segmentation and centroid majority voting to regress a vector field with the same resolution as the input image. ... CentroidNet is a fully-convolutional one-stage detector which can adopt an FCN as a backbone. We choose a U-Net segmentation network as a basis because of its good performance [16]. The output of U-Net is adapted to accommodate CentroidNet.”;
DIJKSTRA, p. 589, section 3: “The input image to CentroidNet can be of any size and can have an arbitrary number of color channels.”;
DIJKSTRA, p. 590, section 3: “Theoretically any fully-convolutional network can be used as a basis for CentroidNet as long as the spatial dimensions of the input and the output are identical. However there are certain advantages to employing this specific CNN architecture.”
RONNENBERGER, p. 2, section 1: “In this paper, we build upon a more elegant architecture, the so-called ‘fully conventional network.’”;
Examiner’s Note: DJIKSTRA teaches that CentroidNet (a CNN) predicts a centroid, and is build on an image segmenter called U-Net, which is a fully convolutional network that inputs a segmented image into CentroidNet; the SLOCUM-DIJKSTRA combination now takes the 2-stage training techniques of SLOCUM and applies them to the U-Net and CentroidNet neural networks of DIJKSTRA)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of SLOCUM with DIJKSTRA. As disclosed by DIJKSTRA, one of ordinary skill would have been motivated to do so in order to perform “joint object counting and localization.” (p. 586, section 1).
Regarding Claim 7
SLOCUM teaches the system of claim 1. However, SLOCUM fails to explicitly teach:
wherein the first neural network comprises a segmentation network and the second neural network comprises a centroid prediction network.
wherein the training data comprises sensor data having a bounding box that encloses a portion of the sensor data corresponding to a target in a scene, and wherein the sensor data comprises data from at least one of a light detection and ranging sensor and a camera sensor.
However, in a related field of endeavor (neural networks, see p. 585, section 1), DIJKSTRA teaches and makes obvious:
wherein the training data comprises sensor data having a bounding box that encloses a portion of the sensor data corresponding to a target in a scene, and wherein the sensor data comprises data from at least one of a light detection and ranging sensor and a camera sensor. (DIJKSTRA, p. 587, section 2.1: “Two domain experts annotated bounding boxes of potato plants in each of the images. This set is split into a training and validation set each containing 5 images and over 3000 annotated potato plant locations.”;
Examiner’s Note: DIJKSTRA teaches that the training data includes images (from a camera sensor) that enclose a scene of crops; the SLOCUM-DIJKSTRA combination now takes the 2-stage training techniques of SLOCUM and applies such techniques to the U-Net and CentroidNet neural networks of DIJKSTRA, where CentroidNET is trained using the crops training data of DIJKSTRA)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of SLOCUM with DIJKSTRA. As disclosed by DIJKSTRA, one of ordinary skill would have been motivated to do so in order to perform “joint object counting and localization.” (p. 586, section 1).
Claim 10 depends from claim 9 and recites a method that corresponds to the system of claim 2 and is therefore rejected for the same reasons explained with respect to claims 2 and 9.
Claim 15 depends from claim 9 and recites a method that corresponds to the system of claim 7 and is therefore rejected for the same reasons explained with respect to claims 7 and 9.
Claim 18 depends from claim 17 and recites a non-transitory computer-readable medium that corresponds to the system of claim 2 and is therefore rejected for the same reasons explained with respect to claims 2 and 17.
Claims 5, 13, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over SLOCUM in view of US 20230121711 A1, hereinafter referenced as CHHAYA.
Regarding Claim 5
SLOCUM teaches the system of claim 1. SLOCUM further teaches:
wherein training the second neural network comprises updating one or more parameters of the second neural network during one or more training iterations of the second training stage based on another respective error or loss function value calculated during each training iteration of the second training stage. (SLOCUM, para. 0052: “In some implementations, the second training stage 300b trains the SSD head 160 using supervised learning where each SSD training utterance 330 is paired with a corresponding ground-truth target 165 indicating whether the SSD training utterance 330 includes a human-generated speech sample 330a or a synthetic speech sample 330b. Here, a SSD loss module 168 may apply a loss function on the classification score 162 predicted by the SSD head 160 to generate a corresponding SSD loss 169 based on the ground-truth target 165. In some examples, the loss function includes a softmax cross-entropy loss where a softmax layer applies a softmax on the classification score 162 (e.g., the Boolean values) output from the SSD head 160 that is fed to the cross-entropy loss function to generate the SSD loss 169.”;
Examiner’s Note: SLOCUM teaches that the SSD Head 160 uses a loss function including a softmax cross-entropy loss during a training iteration)
However, SLOCUM fails to explicitly teach:
wherein training the first neural network comprises updating one or more parameters of the first neural network during one or more training iterations of the first training stage based on a respective error or loss function value calculated during each training iteration of the first training stage
However, in a related field of endeavor (training neural networks, see para. 0030), CHAAYA teaches and makes obvious:
wherein training the first neural network comprises updating one or more parameters of the first neural network during one or more training iterations of the first training stage based on a respective error or loss function value calculated during each training iteration of the first training stage (CHAAYA, para. 0078: “During the training period, the computing system 200 may provide a training information that is used to train the neural network 121. As part of the processing in 440, one or more optimization techniques (e.g., back propagation techniques) may be used to iteratively train the neural network 121 while optimizing one or more of the combined loss functions, the causal modelling loss function, the causal loss function, and the metric loss function. As part of the optimization, weights and biases associated with different layers of the neural network 121 may be changed such that the errors in the prediction of the output text and the predicted stage identifier for the output text is minimized.”;
Examiner’s Note: the SLOCUM-CHAAYA combination now modifies SLOCUM to iteratively train the audio encoder 150 of SLOCUM using the teachings of CHAAYA, where a loss function is used to backpropagate changes to the weights and biases of different layers of the neural network)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of SLOCUM with CHAAYA. As disclosed by CHAAYA, one of ordinary skill would have been motivated to do so in order to utilize known training techniques, such as back propagation, to optimize the neural network models. (para. 0070). One of ordinary skill would further have been motivated to continue to iterate until a loss metric converges below a threshold to ensure a certain level of accuracy with respect to the neural network model).
Claim 13 depends from claim 9 and recites a method that corresponds to the system of claim 5 and is therefore rejected for the same reasons explained with respect to claims 5 and 9.
Claim 19 depends from claim 17 and recites a non-transitory computer-readable medium that corresponds to the system of claim 5 and is therefore rejected for the same reasons explained with respect to claims 5 and 17.
Claims 8 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over SLOCUM in view of DIJKSTRA and further in view of US 20240032856 A1, hereinafter referenced as PARK, and further in view of US 20230252443 A1, hereinafter referenced as MCDANIEL.
Regarding Claim 8
SLOCUM and DIJKSTRA teach the system of claim 7. However, SLOCUM fails to explicitly teach:
wherein the first neural network comprises a segmentation network and the second neural network comprises a centroid prediction network,
wherein the one or more processors are configured to apply one or more operations to the one or more outputs from the first neural network before providing the one or more outputs to the second neural network,
wherein the one or more operations comprises a point masking operation configured to mask or remove one or more datapoints from the portion of the sensor data enclosed within the bounding box based on a determination that the one or more datapoints do not correspond to the target in the scene.
However, in a related field of endeavor (neural networks, see p. 585, section 1), DIJKSTRA teaches and makes obvious:
wherein the first neural network comprises a segmentation network and the second neural network comprises a centroid prediction network, (DIJKSTRA, p. 586, section 1: “This paper introduces CentroidNet which has been specifically designed for joint object counting and localization. CentroidNet produces centroids of image objects rather than bounding boxes. The key idea behind CentroidNet is to combine image segmentation and centroid majority voting to regress a vector field with the same resolution as the input image. ... CentroidNet is a fully-convolutional one-stage detector which can adopt an FCN as a backbone. We choose a U-Net segmentation network as a basis because of its good performance [16]. The output of U-Net is adapted to accommodate CentroidNet.”;
DIJKSTRA, p. 589, section 3: “The input image to CentroidNet can be of any size and can have an arbitrary number of color channels.”;
DIJKSTRA, p. 590, section 3: “Theoretically any fully-convolutional network can be used as a basis for CentroidNet as long as the spatial dimensions of the input and the output are identical. However there are certain advantages to employing this specific CNN architecture.”
RONNENBERGER, p. 2, section 1: “In this paper, we build upon a more elegant architecture, the so-called ‘fully conventional network.’”;
Examiner’s Note: DJIKSTRA teaches that CentroidNet (a CNN) predicts a centroid, and is built on an image segmenter called U-Net, which is a fully convolutional network that inputs a segmented image into CentroidNet; the SLOCUM-DIJKSTRA combination now takes the 2-stage training techniques of SLOCUM and applies them to the U-Net and CentroidNet neural networks of DIJKSTRA)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of SLOCUM with DIJKSTRA. As disclosed by DIJKSTRA, one of ordinary skill would have been motivated to do so in order to perform “joint object counting and localization.” (p. 586, section 1).
However, SLOCUM and DIJKSTRA fail to explicitly teach:
wherein the one or more processors are configured to apply one or more operations to the one or more outputs from the first neural network before providing the one or more outputs to the second neural network,
wherein the one or more operations comprises a point masking operation configured to mask or remove one or more datapoints from the portion of the sensor data enclosed within the bounding box based on a determination that the one or more datapoints do not correspond to the target in the scene.
However, in a related field of endeavor (neural network image processing, see para. 0103), PARK teaches and makes obvious:
wherein the one or more processors are configured to apply one or more operations to the one or more outputs from the first neural network before providing the one or more outputs to the second neural network (PARK, para. 0514: “Meanwhile, in order to calculate the hair loss diagnosis assistance information, as described above, a pre-processing operation of the target image or a post-processing operation of correcting the initial pore region information output through the neural network model may be performed before being input to the second neural network model.”;
Examiner’s Note: the SLOCUM-DIJKSTRA-PARK combination now takes the 2-stage training techniques of SLOCUM and applies them to the U-Net and CentroidNet neural networks of DIJKSTRA, where as taught in PARK, the output image of U-Net is pre-processed before being input into CentroidNET)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of SLOCUM with DIJKSTRA and PARK. As disclosed by PARK, one of ordinary skill would have been motivated to do so to remove noise from the image. (para. 0428). One of ordinary skill would further have been motivated to do so, for example, to re-size the image output from the first neural network into an acceptable range for the second neural network.
However, SLOCUM, DIJKSTRA, and MCDANIEL fail to explicitly teach:
wherein the one or more operations comprises a point masking operation configured to mask or remove one or more datapoints from the portion of the sensor data enclosed within the bounding box based on a determination that the one or more datapoints do not correspond to the target in the scene.
However, in a related field of endeavor (neural network image operations, see para. 0008), MCDANIEL teaches and makes obvious:
wherein the one or more operations comprises a point masking operation configured to mask or remove one or more datapoints from the portion of the sensor data enclosed within the bounding box based on a determination that the one or more datapoints do not correspond to the target in the scene. (MCDANIEL, para. 0071: “Note that the mashing and segmentation first MLM 118 removed background pixel data from the images of the scene. Bounding box manager 117 or the masks can be used to uniquely identify each unique item in each image of the scene.”
MCDANIEL, para.0073: “Bounding box manager 117 performs a clustering algorithm on the remaining depth information and RGB data for the scene in the single point cloud or utilizes the masks generated by the first MLM 118. This associates the component point cloud points that each individual camera 120 and/or 130 was contributing. The bounding box manager 117 creates a bounding box (using the mask in some cases) around each cluster, resulting in a single bounding box per item in the scene of the designated area. Each item’s 3D bounding box can be used to create 2D bounding boxes in each 2D RGB image where the item is visible.”;
Examiner’s Note: MCDANIEL discloses removing background pixel data from images of the scene; the SLOCUM-DIJKSTRA-PARK-MCDANIEL combination now takes the 2-stage training techniques of SLOCUM and applies them to the U-Net and CentroidNet neural networks of DIJKSTRA, where as taught in PARK, the output image of U-Net is pre-processed before being input into CentroidNET, where the processing between the networks includes removing background pixels from a bounding box of an image as in MCDANIEL)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of SLOCUM with DIJKSTRA, PARK, and MCDANIEL. As disclosed by MCDANIEL, one of ordinary skill would have been motivated to do so to remove background pixels from a scene to highlight only the pixels that are desired. (para. 0042).
Claim 16 depends from claim 15 and recites a method that corresponds to the system of claim 8 and is therefore rejected for the same reasons explained with respect to claims 8 and 15.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20240202325 A1 (Kim). “During the second training stage 200B, partial networks may be freezed to achieve the best performance on testing datasets and strengthen robustness of entire model (i.e., the first neural network 111 and the second neural network 115). For instance, the encoder and the decoder of the first neural network 111 may be freezed and only the second neural network 115 is trained with the labeled data samples 203.” (para. 0049). “In some embodiments, partial networks may be freezed during the third training stage 200C. For example, the encoder and the decoder of the first neural network 111 may be freezed and only the second neural network 115 is trained with the labeled samples 205 of the new domain. In another example, only the encoder is freezed, and the decoder and the second neural network 115 are trained with the labeled samples 205 of the new domain.” (para. 0052).
US 20230177334 A1 (Borgaud Dit Avocat). “The neural network system 100 can be trained jointly (e.g., where both the embedding network 104 and decoder neural network 200 are trained concurrently at each training iteration), or trained in stages using pre-training (e.g., where the embedding network 104 is pre-trained and the decoder neural network is trained over a series of training iterations with the parameter values of the pre-trained embedding neural network 104 being frozen), as is described below with reference to FIG. 5.” (para. 0074).
US 20250140012 A1 (Rimchala). “In the method of FIG. 3A, the visual encoder model, the optical character recognition model, and the large language model may be frozen. The optical character recognition model also may be frozen. The term “frozen,” as used herein, means that the parameters of the frozen machine learning model are not modified during a training operation of some other machine learning model, even when both models are involved in the training of the model that is not frozen. Thus, in this example, it is possible during stage 1 of training that only the projection network model is trained, even though the large language model and other models of the multi-model model ensemble are used during the training of the projection network model. In other words, the visual encoder model and the large language are not trained or modified in the method of FIG. 3A.” (para. 0115).
US 20240312177 A1 (Redford). “In a conventional pre-training/fine-tuning approach, encoder weights/parameters would be learned initially entirely via self-supervised training, independently of the ultimately desired task. In the fine-tuning phase, the encoder weights would be frozen. The perception component would receive and process features from the frozen encoder, and be trained independently via conventional supervised training.” (para. 0022).
US 20240193887 A1 (Hao). “In an embodiment, the vector field decoder 220 is combined with an encoder to implement a vector quantized-variational autoencoder (VQ-VAE) for encoding vector fields and reconstruct shapes. Training may be performed in two phases, with the vector field decoder 220 and the encoder being trained in the first phase. In the second stage, the learned parameters used by the vector field decoder 220 and encoder are not updated (are frozen) and, the diffusion model 225 is trained to map input text to corresponding 3D shape latent representations. Parameters used by the diffusion model 225 are learned during the second stage. The two-stage training produces a vector field decoder 220 and a diffusion model 225 that are able to synthesize high-quality 3D shapes with open surfaces and generalizes to zero-shot synthesis.” (para. 0040).
US 11404087 B1 (Khalilia). “Referring now to FIG. 2, an example process for training of the content-extraction encoder 121, the style-extraction encoder 122, and the decoder 140 will now be described in detail. In the example, of FIG. 2 a three phrase training process is employed. At operation 201, training of the style-extraction encoder 122 is performed as a first training phase. The first training phase is described in detail below with reference to FIG. 3. At operation 202, training of the content-extraction encoder 121 is performed as a second training phase. The second training phase is described in detail below with reference to FIG. 4. In some examples, the first and second training phases may be performed wholly or partially simultaneously with one another or in any order with respect to one another. At operation 203, the encoder parameters for the content-extraction encoder 121 and the style-extraction encoder 122 are frozen, meaning that they are not updated during the training of the decoder 140. The encoding parameters are the encoding instructions that are learned by the content-extraction encoder 121 and the style-extraction encoder 122 during the first and second training phases, respectively, as described in detail below with reference to FIGS. 3 and 4. At operation 204, training of the decoder 140 is performed as a third training phase. The third training phase is described in detail below with reference to FIG. 5. The third training phase may be performed after completion of both the first and the second training phases. Performing of decoder training after encoder training may allow encoding parameters for the content-extraction encoder 121 and the style-extraction encoder 122 and their connections to the decoder to be frozen (i.e., not updated) during the decoder training. This helps to prevent unlearning of weights for the style-extraction encoding and the content-extraction encoding, so that the separation of content and style are not lost.” (col. 6, line 65 – col. 7, line 31).
US 20220067983 A1 (Fidler). “At 208, said system learns a new encoder based on data from a synthetic dataset generated in relation to operation 204. In at least one embodiment, said system trains said new encoder while its corresponding decoder, trained at 202, is frozen.” (para. 0093).
US 20200104707 A1 (Azadi). “ Optionally, after jointly training the generator neural network 104 and the discriminator neural network 106, the training engine 108 may freeze the parameter values of the generator neural network 104 and subsequently continue to train the discriminator neural network 106 (i.e., for one or more additional training iterations). For example, the training engine 108 may continue training the discriminator neural network 106 until its performance, evaluated on a set of validation set of real data samples and data samples generated using the frozen parameter values of the generator neural network 104, achieves a maximum value.” (para. 0033).
US 20230401837 A1 (Yan). “A shadow (i.e., a projection) of dynamic objects is not considered when determining the pixels of the image which are associated with static objects through semantic recognition as described above. Generally, semantic recognition does not label a shadow of an object. Therefore, in some embodiments, to determine the content associated with the static objects (i.e., the background) of the scene in the image, the computing device 120 can perform semantic recognition on each frame of image acquired in step 401, and determine the content associated with the dynamic objects (e.g., the vehicle 331 and vehicle 332). Then, the computing device 120 determines the content associated with the shadow (i.e., the projection) of the dynamic objects in the image, and removes the content associated with the shadow of the dynamic objects and the content associated with the dynamic objects from the image to obtain the content associated with the static objects.” (para. 0057).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL C LEE whose telephone number is (571)272-4933. The examiner can normally be reached M-F 12:00 pm - 8:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Omar Fernandez Rivas can be reached at 571-272-2589. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL C. LEE/Examiner, Art Unit 2128
Prosecution Insights