Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Preliminary Amendments
This action is in response to preliminary amendments filed January 19th, 2024, in which Claims 1-15 are cancelled and Claims 16-35 are added. Claims 16-35 are currently pending and have been examined.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 30-33 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 30 recites the limitations a second value and a second objective function without having previously recited any first value nor any first objective function. This renders the claim indefinite, because it is unclear whether only one, or whether two, values and objective functions are required based on the claim language. For the purpose of examination, the claim will be interpreted using second as a label, and only requiring an invention to have a single value and objective function as recited, to be within the claim scope.
Claim 31 recites the limitations a fifth feature embedding and a sixth feature embedding without having previously recited any fourth feature embedding. This renders the claim indefinite, because it is unclear exactly how many feature embeddings are necessary (six, as per sixth feature embedding, or five, as per the recited first, second third, fifth, and sixth). For the purpose of examination, the claim will be interpreted using all ordinal adjectives merely as labels, and only requiring an invention to have the recited first, second third, fifth, and sixth embeddings to be within the claim scope.
Claim 32 recites the limitations a third global representation without having previously recited any second global representation. This renders the claim indefinite, because it is unclear exactly how many global representations are necessary (three as per third global representation or two, as per the recited first and third). For the purpose of examination, the claim will be interpreted using all ordinal adjectives merely as labels, and only requiring an invention to have the recited first and third global representations to be within the claim scope.
Dependent claims are rejected for inheriting the indefiniteness of a parent claim.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 16-19 and 30-35 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Georgiou et al., “M3: MultiModal Masking applied to sentiment analysis.”
Regarding Claim 16, Georgiou teaches an apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least (Georgiou, pg. 2876, 2nd column, 2nd-to-last paragraph, “Our code is available as open-source” denotes that they perform their method on a computer, in which processor and memory are inherent) to: obtain a first data sample and a second data sample (Georgiou, pg. 2877, Fig. 1,
X
A
and
X
V
are first and second data samples of audio and video); transform the first data sample into a first feature embedding using a first machine learning model (Georgiou, pg. 2877, Fig. 1,
m
A
is an embedding generated though a Bi-LSTM by transforming
X
A
); transform the second data sample into a second feature embedding using a second machine learning model (Georgiou, pg. 2877, Fig. 1,
m
V
is an embedding generated though a Bi-LSTM by transforming
X
V
); generate a first global representation by masking at least one of: the first feature embedding or the second feature embedding (Georgiou, pg. 2877, 2nd column, last paragraph, “decide whether to mask one of the given modalities or leave them unaffected, based on a masking probability” see pg. 2878, Fig. 2 & pg. 2877, Fig. 1, where the masked input to DNN-II is a first global representation); transform the first global representation into a third feature embedding using a third machine learning model (Georgiou, pg. 2877, Fig. 1,
m
O
from another Bi-LSTM & 2nd column, 3rd paragraph “a single representation which is denoted as
m
O
. This final representation is fed to a linear layer which performs regression”); and train at least the third machine learning model based on the third feature embedding (Georgiou, pg. 2877, Fig 1 & pg. 2878, 2nd column, 1st paragraph, “Models are trained for regression on sentiment values using Mean Absolute Error loss”).
Regarding Claim 17, Georgiou teaches the apparatus according to Claim 16 (and thus the rejection of Claim 16 is incorporated). Georgiou has already been shown to teach to train the first machine learning model, the second machine learning model, and the third machine learning model based on the third feature embedding (Georgiou, pg. 2877, Fig 1 & pg. 2878, 2nd column, 1st paragraph, “Models are trained for regression on sentiment values using Mean Absolute Error loss”).
Regarding Claim 18, Georgiou teaches the apparatus according to Claim 16 (and thus the rejection of Claim 16 is incorporated). Georgiou has already been shown to teach wherein the first data sample is associated with a first sensor and the second data sample is associated with a second sensor (Georgiou, pg. 2877, Fig. 1, where
X
A
and
X
V
are first and second data samples from audio and video sensors).
Regarding Claim 19, Georgiou teaches the apparatus according to Claim 16 (and thus the rejection of Claim 16 is incorporated). Georgiou further teaches wherein the first data sample comprises a plurality of data samples, the second data sample comprises a second plurality of data samples, the first feature embedding comprises a plurality of first feature embeddings, the second feature embedding comprises a second plurality of feature embeddings (Georgiou, pg. 2878, Fig. 2, where the pluralities of data samples and embeddings are from different timestamps, see pg. 2877, 2nd column, 2nd-to-last paragraph, “at every time step
i
in the multimodal sequence of length N”) and wherein: the generating of the first global representation by masking at least one of the first feature embedding or the second feature embedding, further comprises: mask at least one feature embedding in the combination of the first plurality of feature embeddings and the second plurality of feature embeddings (Georgiou, pg. 2878, Fig. 2, where different timestep embeddings of the different modalities are masked).
Regarding Claim 30, Georgiou teaches the apparatus according to Claim 16 (and thus the rejection of Claim 16 is incorporated). Georgiou further teaches to generate a first prediction using a fourth machine learning model and the first global representation (Georgiou, pg. 2877, Fig. 1 & 2nd column, 3rd paragraph “This final representation is fed to a linear layer which performs regression” to predict the sentiment); obtain a second value associated with the first data sample and the second data sample (Georgiou, pg. 2878, 1st column, last paragraph, “video clips of movie reviews annotated at the vide level for sentiment … sentiment scores range from -3 (strongly negative) to 3 (strongly positive)”); determine a value of a second objective function based on the first prediction and the second value; and train at least the third machine learning model based on the value of the second objective function (Georgiou, pg. 2877, Fig 1 & pg. 2878, 2nd column, 1st paragraph, “Models are trained for regression on sentiment values using Mean Absolute Error loss” where the error is the difference between labeled and predicted sentiment).
Regarding Claim 31, Georgiou teaches the apparatus according to Claim 30 (and thus the rejection of Claim 30 is incorporated). Georgiou further teaches to train the first machine learning model, the second machine learning model, the third machine learning model, and the fourth machine learning model based on the value of the second objective function (Georgiou, pg. 2877, Fig 1 & pg. 2878, 2nd column, 1st paragraph, “Models are trained for regression on sentiment values using Mean Absolute Error loss” where the MAE loss was previously identified as the value of the second objective function).
Regarding Claim 32, Georgiou teaches the apparatus according to Claim 30 (and thus the rejection of Claim 30 is incorporated). Georgiou further teaches to obtain a third data sample and a fourth data sample; transform the third data sample into a fifth feature embedding using the first machine learning model; transform the fourth data sample into a sixth feature embedding using the second machine learning model; generate a third global representation by combining the fifth feature embedding and the sixth feature embedding; and transform the third global representation into a seventh feature embedding using the third machine learning model (Georgiou, pg. 2877, Fig. 1, but applied to a second video clip input into the model, with new video and audio data samples -> embeddings, using the same machine learning models, see pg. 2878, 1st column, last paragraph, “The dataset contains 23,454 YouTube video clips of movie reviews” & 2nd column, 1st paragraph, “models are trained using batch size
B
=
32
”).
Regarding Claim 33, Georgiou teaches the apparatus according to Claim 32 (and thus the rejection of Claim 32 is incorporated). Georgiou further teaches to generate a second prediction using the fourth machine learning model and the third global representation (Georgiou, pg. 2877, Fig. 1, again applied to a second video clip to make a second sentiment prediction with the linear model).
Claim 34 recites precisely the method performed by the apparatus of Claim 16, and is thus rejected for reasons set forth in the rejection of Claim 16. Similarly, Claim 35 recites a non-transitory computer readable storage medium comprising program instructions that are the instructions of Claim 16, and is thus also rejected for reasons set forth in the rejection of Claim 16.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Georgiou et al., “M3: MultiModal Masking applied to sentiment analysis,” in view of Grigore, an answer to the question “random numbers with different probabilities” on the website stackoverflow.com.
Regarding Claim 20, Georgiou teaches the apparatus according to Claim 19 (and thus the rejection of Claim 19 is incorporated). Georgiou further teaches to obtain a threshold value and to mask a first embedding in the first plurality in response to determining a random variable based on the threshold value (Georgiou, pg. 2877, 2nd column, last paragraph, “it decides whether to mask one of the given modalities or leave them unaffected, based on a masking probability denotes as
p
M
. … Formally this is described as sampling from a Bernoulli distribution … the probability which decides whether to mask or not”). A Bernoulli random variable is a variable which takes the value 1 or 0 given a certain probability, and as such, Georgiou does not teach to generate a random number; determine if the random number is greater than the threshold value; and mask … in response to determining that the random number is less than the threshold value.
However, Grigore teaches a method to generate a Boolean 1-0 random variable using these steps:
Bool TrueFalse = (rand() % 100) < 75;
“The rand() % 100 will give you a random number between 0 and 100, and the probability of it being under 75 is, well, 75%. You can substitute the 75 for any probability you want”.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the method of Grigore to generate the Bernoulli Boolean random variable of Georgiou. The motivation to do so is that Georgiou is silent on their particular method for generating the random variable, and Grigore gives us code which can do so.
Conclusion
Claims 21-29 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. Claim 21 was searched, but no prior art teaching the limitations of the claim was uncovered.
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure:
He et al., “MF-BERT: Multimodal Fusion in Pre-Trained BERT for Sentiment Analysis” teaches an objective function related to similarity, relevant to Claim 29.
Zhao et al., “Towards Effective Multi-Modal Interchanges in Zero-Resource Sounding Object Localization,” teaches determining a pivot location in order to align multi-modal data sequences, but fails to do so in the manner recited in Claim 21, i.e. to determine a position value by sampling from a probability distribution, where the mean of the probability distribution is the pivot location.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRIAN M SMITH whose telephone number is (469)295-9104. The examiner can normally be reached Monday - Friday, 8:00am - 4pm Pacific.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BRIAN M SMITH/Primary Examiner, Art Unit 2122
Prosecution Insights