DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the original application filed on November, 8th, 2022.
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. CN202211396127.0, filed on November 8th, 2022.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 8 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 8 recites the limitation, “The method of claim 7, wherein generating the noise data frame comprises: generating an intermediate data frame by adjusting a dimension of the data frame based on a predetermined ratio. generating a plurality of copied intermediate data frames by copying the intermediate data frame; and generating the noise data by joining the plurality of copied intermediate data frames.” (Emphasis added). Claims are required to disclose the claimed subject matter in a single sentence. This claim contains a two sentences. Therefore, Because the claim contains more than one sentence, this claim is rejected under 35 U.S.C. 112(b) for being indefinite. For examination purposes the claim will be interpreted to have the period replaced with a semi-colon, “… predetermined ratio;[[.]] generating a …”. Appropriate correction is required.
Claim Rejections - 35 USC § 101 – Software per se.
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 13-19 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. Claims 13-19 do not fall within at least one of the four categories of patent eligible subject matter because it recites claim 13 recites, "An electronic device, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions to be executed by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform a method for managing a model based on a distance between samples, the method comprising:" (emphasis added). The submitted specification recites "The processing unit 1010 may be an actual or virtual processor and can execute various processes according to the programs stored in the memory 1020. In a multiprocessor system, a plurality of processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 1000." (Example Apparatus and Equipment, pp. 24, [00113]), (emphasis added), which states this system may be virtual only and therefore would be considered software per se. This claim does not fall under the four categories of patent eligible subject matter per MPEP 2106.06(I): "Even when a product has a physical or tangible form, it may not fall within a statutory category. For instance, a transitory signal, while physical and real, does not possess concrete structure that would qualify as a device or part under the definition of a machine, is not a tangible article or commodity under the definition of a manufacture (even though it is man-made and physical in that it exists in the real world and has tangible causes and effects), and is not composed of matter such that it would qualify as a composition of matter.".
Claim 13
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
This claim is not directed to one of the four categories of statutory subject matter per MPEP 2106.03(I). However, for compact prosecution, the examiner will interpret this claim as falling under one of the four categories to further evaluate the claim using the Alice/Mayo test.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites, inter alia:
“generating a sequence of the plurality of negative samples based on distances between the plurality of negative samples and the basic sample;” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to evaluate data and identify and generate negative samples based on distance or similarity values. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
“dividing the sequence of the plurality of negative samples into a first set of negative samples and a second set of negative samples, a first distance between a first negative sample in the first set of negative samples and the basic sample being less than a second distance between a second negative sample in the second set of negative samples and the basic sample; and” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to evaluate a set and divide it in a given way. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
“determining an update parameter for updating the contrastive learning model based on the basic sample, the first set of negative samples and a first weight of the first set of negative samples, and the second set of negative samples and a second weight of the second set of negative samples, the first weight being greater than the second weight.” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to evaluate data and determine a value to update a machine learning model based on the evaluation. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions to be executed by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform a method for managing a model based on a distance between samples, the method comprising:” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
“obtaining a basic sample for training a contrastive learning model and a plurality of negative samples associated with the basic sample;” is an insignificant extra-solution activity required for any uses of the mental processes (see MPEP § 2106.05(g)) As such, the claim is ineligible.
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions to be executed by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform a method for managing a model based on a distance between samples, the method comprising:” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
“obtaining a basic sample for training a contrastive learning model and a plurality of negative samples associated with the basic sample;” is an insignificant extra-solution activity required for any uses of abstract ideas (see MPEP § 2106.05(g)), and is a well-understood, routine, conventional activity (see MPEP § 2106.05(d)(iv); “Storing and retrieving information in memory”.
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 14
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
This claim is not directed to one of the four categories of statutory subject matter per MPEP 2106.03(I). However, for compact prosecution, the examiner will interpret this claim as falling under one of the four categories to further evaluate the claim using the Alice/Mayo test.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites, inter alia:
“selecting, from a first data sequence of a plurality of data sequences for training the contrastive learning model, a first data segment as the basic sample; and” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to evaluate a set of data and select a basic sample. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
“selecting, from a second sequence of the plurality of data sequences, a second data segment as a negative sample of the plurality of negative samples.” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to evaluate a set of data and identify a sequence of negative samples. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
This claim does not recite any additional limitations which integrate the abstract idea into a practical application.
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea and thus the claim is subject-matter ineligible.
Claim 15
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
This claim is not directed to one of the four categories of statutory subject matter per MPEP 2106.03(I). However, for compact prosecution, the examiner will interpret this claim as falling under one of the four categories to further evaluate the claim using the Alice/Mayo test.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites, inter alia:
“selecting a third data segment from the first data sequence as a positive sample associated with the basic sample; and” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to identify different segments of data which is similar to target item. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
“wherein determining the update parameter further comprises: determining the update parameter based on the basic sample and the positive sample.” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to evaluate a model and determine a value to update that model. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
This claim does not recite any additional limitations which integrate the abstract idea into a practical application.
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea and thus the claim is subject-matter ineligible.
Claim 16
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
This claim is not directed to one of the four categories of statutory subject matter per MPEP 2106.03(I). However, for compact prosecution, the examiner will interpret this claim as falling under one of the four categories to further evaluate the claim using the Alice/Mayo test.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites, inter alia:
“selecting a fourth data segment from the first data sequence;” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to select data items in a sequence of data items. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
“adjusting a sampling frequency of a plurality of data frames in the fourth data segment to generate a fifth data segment;” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to adjust values of a function to produce different segments of data. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
“generating a third set of negative samples for training the contrastive learning model based on the fifth data segment; and” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to evaluate a set of data and identify different negative samples compared to a target item. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
“wherein determining the update parameter further comprises: determining the update parameter based on the third set of negative samples and a third weight of the third set of negative samples, the third weight being greater than the second weight.” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human can evaluate data and determine a value to update a model. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
This claim does not recite any additional limitations which integrate the abstract idea into a practical application.
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea and thus the claim is subject-matter ineligible.
Claim 17
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
This claim is not directed to one of the four categories of statutory subject matter per MPEP 2106.03(I). However, for compact prosecution, the examiner will interpret this claim as falling under one of the four categories to further evaluate the claim using the Alice/Mayo test.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “generating the negative sample by updating at least one of an appearance and a sampling frequency of a plurality of data frames of the second data segment.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “generating the negative sample by updating at least one of an appearance and a sampling frequency of a plurality of data frames of the second data segment.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 18
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
This claim is not directed to one of the four categories of statutory subject matter per MPEP 2106.03(I). However, for compact prosecution, the examiner will interpret this claim as falling under one of the four categories to further evaluate the claim using the Alice/Mayo test.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites, inter alia:
“selecting a data frame from a data sequence other than the second data sequence in the plurality of data sequences; and” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to evaluate and select data in a set of data frames. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “generating the negative sample by updating an appearance of the second data segment with the data frame.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “generating the negative sample by updating an appearance of the second data segment with the data frame.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 19
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
This claim is not directed to one of the four categories of statutory subject matter per MPEP 2106.03(I). However, for compact prosecution, the examiner will interpret this claim as falling under one of the four categories to further evaluate the claim using the Alice/Mayo test.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “generating a noise data frame based on the data frame; and” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
“updating a data frame of the second data segment with the noise data frame.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “generating a noise data frame based on the data frame; and” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
“updating a data frame of the second data segment with the noise data frame.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim Rejections - 35 USC § 101 – Abstract idea.
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-12, and 20 are rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The analysis of the claims will follow the 2019 Revised Patent Subject Matter Eligibility Guidance, 84 Fed. Reg. 50 (“2019 PEG”).
Claim 1
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Claim 1 recites, “A method for managing a model based on a distance between samples, comprising:” therefore it is directed to the statutory category of a process.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites, inter alia:
“generating a sequence of the plurality of negative samples based on distances between the plurality of negative samples and the basic sample;” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to evaluate data and identify and generate negative samples based on distance or similarity values. The limitation recites a mental process including observation, evaluation, judgment and/or opinion, see MPEP 2106.04(a).
“dividing the sequence of the plurality of negative samples into a first set of negative samples and a second set of negative samples, a first distance between a first negative sample in the first set of negative samples and the basic sample being less than a second distance between a second negative sample in the second set of negative samples and the basic sample; and” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to evaluate a set and divide it in a given way. The limitation recites a mental process including observation, evaluation, judgment and/or opinion, see MPEP 2106.04(a).
“determining an update parameter for updating the contrastive learning model based on the basic sample, the first set of negative samples and a first weight of the first set of negative samples, and the second set of negative samples and a second weight of the second set of negative samples, the first weight being greater than the second weight.” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to evaluate data and determine a value to update a machine learning model based on the evaluation. The limitation recites a mental process including observation, evaluation, judgment and/or opinion, see MPEP 2106.04(a).).
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “A method for managing a model based on a distance between samples, comprising:” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
“obtaining a basic sample for training a contrastive learning model and a plurality of negative samples associated with the basic sample;” is an insignificant extra-solution activity required for any uses of the mental processes (see MPEP § 2106.05(g)) As such, the claim is ineligible.
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, A method for managing a model based on a distance between samples, comprising:” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
“obtaining a basic sample for training a contrastive learning model and a plurality of negative samples associated with the basic sample;” is an insignificant extra-solution activity required for any uses of abstract ideas (see MPEP § 2106.05(g)), and is a well-understood, routine, conventional activity (see MPEP § 2106.05(d)(iv); “Storing and retrieving information in memory”.
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 2
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A process, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites, inter alia:
“selecting, from a first data sequence of a plurality of data sequences for training the contrastive learning model, a first data segment as the basic sample; and” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to evaluate a set of data and select a basic sample. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
“selecting, from a second sequence of the plurality of data sequences, a second data segment as a negative sample of the plurality of negative samples.” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to evaluate a set of data and identify a sequence of negative samples. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
This claim does not recite any additional limitations which integrate the abstract idea into a practical application.
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea and thus the claim is subject-matter ineligible.
Claim 3
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A process, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites, inter alia:
“selecting a third data segment from the first data sequence as a positive sample associated with the basic sample; and” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to identify different segments of data which is similar to target item. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
“wherein determining the update parameter further comprises: determining the update parameter based on the basic sample and the positive sample.” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to evaluate a model and determine a value to update that model. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
This claim does not recite any additional limitations which integrate the abstract idea into a practical application.
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea and thus the claim is subject-matter ineligible.
Claim 4
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A process, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites, inter alia:
“selecting a fourth data segment from the first data sequence;” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to select data items in a sequence of data items. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
“adjusting a sampling frequency of a plurality of data frames in the fourth data segment to generate a fifth data segment;” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to adjust values of a function to produce different segments of data. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
“generating a third set of negative samples for training the contrastive learning model based on the fifth data segment; and” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to evaluate a set of data and identify different negative samples compared to a target item. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
“wherein determining the update parameter further comprises: determining the update parameter based on the third set of negative samples and a third weight of the third set of negative samples, the third weight being greater than the second weight.” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human can evaluate data and determine a value to update a model. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
This claim does not recite any additional limitations which integrate the abstract idea into a practical application.
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea and thus the claim is subject-matter ineligible.
Claim 5
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A process, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “generating the negative sample by updating at least one of an appearance and a sampling frequency of a plurality of data frames of the second data segment.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “generating the negative sample by updating at least one of an appearance and a sampling frequency of a plurality of data frames of the second data segment.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 6
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A process, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites, inter alia:
“selecting a data frame from a data sequence other than the second data sequence in the plurality of data sequences; and” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to evaluate and select data in a set of data frames. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “generating the negative sample by updating an appearance of the second data segment with the data frame.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “generating the negative sample by updating an appearance of the second data segment with the data frame.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 7
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A process, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “generating a noise data frame based on the data frame; and” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
“updating a data frame of the second data segment with the noise data frame.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “generating a noise data frame based on the data frame; and” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
“updating a data frame of the second data segment with the noise data frame.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 8
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A process, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “generating an intermediate data frame by adjusting a dimension of the data frame based on a predetermined ratio.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
“generating a plurality of copied intermediate data frames by copying the intermediate data frame; and” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
“generating the noise data by joining the plurality of copied intermediate data frames.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “generating an intermediate data frame by adjusting a dimension of the data frame based on a predetermined ratio.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
“generating a plurality of copied intermediate data frames by copying the intermediate data frame; and” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
“generating the noise data by joining the plurality of copied intermediate data frames.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 9
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A process, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “generating the negative sample by adjusting the sampling frequency of the plurality of data frames of the second data segment.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “generating the negative sample by adjusting the sampling frequency of the plurality of data frames of the second data segment.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 10
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A process, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “the number of the first set of negative samples, a ratio of the first set of negative samples to the plurality of negative samples.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “the number of the first set of negative samples, a ratio of the first set of negative samples to the plurality of negative samples.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 11
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A process, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “updating the contrastive learning model with the update function.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “updating the contrastive learning model with the update function.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 12
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A process, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “determining an association between a first data segment and a second data segment of a pair of samples to be processed with an updated contrastive learning model.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “determining an association between a first data segment and a second data segment of a pair of samples to be processed with an updated contrastive learning model.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 20
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Claim 20 recites, “A non-transitory computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing a method for managing a model based on a distance between samples, the method comprising:” therefore it is directed to the statutory category of a machine.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites, inter alia:
“generating a sequence of the plurality of negative samples based on distances between the plurality of negative samples and the basic sample;” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to evaluate data and identify and generate negative samples based on distance or similarity values. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
“dividing the sequence of the plurality of negative samples into a first set of negative samples and a second set of negative samples, a first distance between a first negative sample in the first set of negative samples and the basic sample being less than a second distance between a second negative sample in the second set of negative samples and the basic sample; and” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to evaluate a set and divide it in a given way. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
“determining an update parameter for updating the contrastive learning model based on the basic sample, the first set of negative samples and a first weight of the first set of negative samples, and the second set of negative samples and a second weight of the second set of negative samples, the first weight being greater than the second weight.” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to evaluate data and determine a value to update a machine learning model based on the evaluation. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “A non-transitory computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing a method for managing a model based on a distance between samples, the method comprising:” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
“obtaining a basic sample for training a contrastive learning model and a plurality of negative samples associated with the basic sample;” is an insignificant extra-solution activity required for any uses of the mental processes (see MPEP § 2106.05(g)) As such, the claim is ineligible.
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “A non-transitory computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing a method for managing a model based on a distance between samples, the method comprising:” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
“obtaining a basic sample for training a contrastive learning model and a plurality of negative samples associated with the basic sample;” is an insignificant extra-solution activity required for any uses of abstract ideas (see MPEP § 2106.05(g)), and is a well-understood, routine, conventional activity (see MPEP § 2106.05(d)(iv); “Storing and retrieving information in memory”.
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5-8, 10-15, and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Tao et al, (Tao et al, “An Improved Inter-Intra Contrastive Learning Framework on Self-Supervised Video Representation”, 2022, hereinafter “Tao”) in view of Kalantidis et al, (Kalantidis et al, “Hard Negative Mixing for Contrastive Learning”, 2020, hereinafter “Kalantidis”).
Regarding claim 1, Tao discloses, “A method for managing a model based on a distance between samples, comprising:” (Methods, pp. 5269; “Our goal is to learn discriminative feature representations from videos, not only for distinguishing one action from another, but also for capturing rich temporal information. The framework of our IICv2 is shown in Fig. 2. In this section, we start from the novel input part and then elaborate on contrastive learning with these inputs.”)
“obtaining a basic sample for training a contrastive learning model and a plurality of negative samples associated with the basic sample;” (Sample Selection, pp. 5269; “Our proposed method is closely related to contrastive learning, using anchor, positive, and negative samples to train.” This model uses contrastive learning and it trains models using a video data which is segmented.) and (figure 2, pp. 5269; This figure shows that a video i is a set of frames. Within these frames is a subset of frames called the anchor which is interpreted to be the basic sample. Frames connected to the anchor can be considered to be the intra negative frames and is interpreted to be the plurality of negative samples.)
PNG
media_image1.png
299
696
media_image1.png
Greyscale
“generating a sequence of the plurality of negative samples based on distances between the plurality of negative samples and the basic sample;” (Inter and Intra Inputs, pp. 5270; “Then
x
n
e
g
(
e
i
t
h
e
r
x
r
e
p
e
a
t
,
x
s
h
u
f
f
l
e
,
x
r
o
t
a
t
i
o
n
)
is used to represent an intra-negative sample from
x
1
. We also want to address that the generated intra-negative samples share similar pixel value distributions with the original one (Fig. 4). From the figure, we can find that the pixel value distributions for the anchor (Video1: view1), the positive (Video1: view2), and intra-negative samples (Video1: repeat, Video1: shuffle, Video1: rotation) are close to each other, constraining the model to learn more discriminative temporal information from video clips.” The model in this article creates new negative sample frames which are used to train the model. The negatives frames have a value denoting their distance or similarity to the anchor and positive frames.)
“dividing the sequence of the plurality of negative samples into a first set of negative samples and a second set of negative samples, a first distance between a first negative sample in the first set of negative samples and the basic sample being less than a second distance between a second negative sample in the second set of negative samples and the basic sample; and” (Figure 8, pp. 5277; “Feature distance distribution. The feature L2 distances are calculated using samples pairs from UCF101 split 1. For each sample pair, one is the anchor, and the other one could be intra-positive, inter-negative, or intra negative sample. The parameters of the network are randomly initialized without optimization. Curves are obtained using kernel density estimation (KDE).” Each of the samples are divided and a distance is calculated between the different samples. The arrows are added by the examiner to point to different examples since the image will show in black and white.)
PNG
media_image2.png
256
355
media_image2.png
Greyscale
Tao fails to explicitly disclose, “determining an update parameter for updating the contrastive learning model based on the basic sample, the first set of negative samples and a first weight of the first set of negative samples, and the second set of negative samples and a second weight of the second set of negative samples, the first weight being greater than the second weight.”
However, Kalantidis discloses, “determining an update parameter for updating the contrastive learning model based on the basic sample, the first set of negative samples and a first weight of the first set of negative samples, and the second set of negative samples and a second weight of the second set of negative samples, the first weight being greater than the second weight.” (Mixing the hardest negatives, pp. 5; “Given a query q, its key k and negative/queue features
n
∈
Q
from a queue of size K, the loss for the query is composed of logits
l
z
i
=
q
T
z
i
/
τ
fed into a softmax function. Let
Q
~
=
{
n
1
,
…
n
k
}
be the ordered set of all negative features, such that:
l
n
i
>
l
n
i
,
∀
i
<
j
, i.e. the set of negative features sorted by decreasing similarity to that particular query feature. For each query, we propose to synthesize s hard negative features, by creating convex linear combinations of pairs of its “hardest” existing negatives. We define the hardest negatives by truncating the ordered set
Q
~
, i.e. only keeping the first
N
<
K
items. Formally, let
H
=
{
h
1
,
…
h
s
}
be the set of synthetic points to be generated. Then, a synthetic point
h
k
∈
H
, would be given by: [See Equation (3)]
n
i
,
n
j
∈
Q
~
N
are randomly chosen negative features from the set
Q
~
N
=
{
n
1
,
…
n
N
}
of the closest N negatives,
α
k
∈
(
0,1
)
is a randomly chosen mixing coefficient and
∙
2
is the
l
2
-norm. After mixing, the logits
l
(
h
k
)
are computed and appended as further negative logits for query q. The process repeats for each query in the batch.” The process in this article will store the negative samples in an ordered list. Meaning that during updating the model the samples with the most negative features, or the samples that would induce the highest weight parameters changes, would be evaluated first.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Tao and Kalantidis. Tao teaches contrastive learning methods that generate and relies on negative samples to train models. Kalantidis teaches contrastive learning methods that use “hard negative” samples in training contrastive learning models and emphasizes the importance of negative samples. One of ordinary skill would have motivation to combine a machine learning method that uses contrastive learning which generates negative samples to further improve machine learning models with an article that emphasizes the importance of using hard negative samples while training contrastive learning modes, “In Table 2 we present results obtained after training on the ImageNet-1K dataset. Looking at the average negative logits plot and because both the queue and the dataset are about an order of magnitude larger for this training dataset, we mostly experiment with smaller values for N than in ImageNet-100. Our main observations are the following: a) MoCHi does not show performance gains over MoCo-v2 for linear classification on ImageNet-1K. We attribute this to the biases induced by training with hard negatives on the same dataset as the downstream task: Figures 3c and 2c show how hard negative mixing reduces alignment and increases uniformity for the dataset that is used during training. MoCHi still retains state-of-the-art performance. b) MoCHi helps the model learn faster and achieves high performance gains over MoCo-v2 for transfer learning after only 100 epochs of training. c) The harder negative strategy presented in Section 4.2 helps a lot for shorter training. d) In 200 epochs MoCHi can achieve performance similar to MoCo-v2 after 800 epochs on PASCAL VOC. e) From all the MoCHi runs reported in Table 2 as well as in the Appendix, we see that performance gains are consistent across multiple hyperparameter configurations.” (Kalantidis, Comparison with the state of the art on ImageNet-1K, PASCAL VOC and COCO., pp. 8).
Regarding claim 2, Tao discloses, “selecting, from a first data sequence of a plurality of data sequences for training the contrastive learning model, a first data segment as the basic sample; and” (Figure 1, pp. 5266; “Given video i and video j, two sampled video clips from video i are treated as the anchor and intra-positive samples, whose features are constrained to be similar to each other.” As seen in the figure, a first data item is selected to be the anchor frames. This is interpreted to be the basic samples)
“selecting, from a second sequence of the plurality of data sequences, a second data segment as a negative sample of the plurality of negative samples.” (Figure 1, pp. 5266; “Data sampled from video j is treated as the negative sample. We generated intra-negative samples from the anchor sample by breaking its temporal relations, which can be treated as hard-negatives because they share similar spatial information but different motion features, and can force the model to learn better more discriminative temporal information.” As seen in the figure, there are negative samples drawn from the same and video i and different video j. This image shows the sets of frames.)
Regarding claim 3, Tao discloses, “selecting a third data segment from the first data sequence as a positive sample associated with the basic sample; and” (Contrastive Learning, pp. 5270; “Contrastive learning uses anchor, positive, and negative samples and aims to extract discriminative features from the anchor and negative samples while maintaining the similarity between the anchor and positive samples. In traditional contrastive learning methods (e.g., CMC [27]), the sample pairs
{
x
i
1
,
x
i
2
}
are positives while
{
x
i
1
,
x
i
2
}
(
i
≠
j
)
are negatives. Because intra-negative samples are used in our approach, the negative pairs are extended by adding
{
x
i
1
,
x
j
n
e
g
}
, where j can be equal to i.” The model in this article will use positive data items in the training as well. As seen in figure 2, the positive frames come from the original set of frames or video i. The first sequence of data is interpreted to be the whole video i)
“wherein determining the update parameter further comprises: determining the update parameter based on the basic sample and the positive sample.” (Contrastive Learning, pp. 5271; “The function is trained by selecting a single positive sample from a set of data. After feature
v
i
1
has been extracted, traditional contrastive learning methods train this function to correctly select a positive sample out of a set
S
2
=
{
v
1
2
,
…
,
v
i
2
,
…
,
v
k
+
1
2
}
, which contains one positive sample
v
i
2
and k negative samples. In our proposed method, another set
S
2
=
{
v
1
n
e
g
,
…
,
v
k
+
1
n
e
g
}
is also used that only contains negative samples. The loss function is similar to recent works for contrastive learning [24], [27], [86]: [see equation (4)] Here, k is the number of negative samples, which can be equal to N − 1, where N is the total number of training samples. We randomly select k samples from N where k
≪
N to accelerate training.” This model is trained using loss values. This would be interpreted to be the update parameter.)
Regarding claim 5, Tao discloses, “generating the negative sample by updating at least one of an appearance and a sampling frequency of a plurality of data frames of the second data segment.” (Inter and Intra inputs, pp. 5270; “To simplify, we use
{
f
r
a
m
e
1
,
…
,
f
r
a
m
e
T
}
to represent a set of temporally-ordered frames. Three different intra-negative generation methods, frame repeating, temporal shuffling, and clip rotation, are proposed to break the temporal relationship and generate intra-negative video clips (Fig. 3).” This article discloses different methods of altering video frames. These methods include adjusting the appearance, 3) Rotation, and sampling frequency, 1) Repeat. Repeat changes the sampling from multiple frames to a repeating a single frame.)
Regarding claim 6, Tao discloses, “selecting a data frame from a data sequence other than the second data sequence in the plurality of data sequences; and” (Inter and Intra Inputs, pp. 5270; “One frame that is randomly selected from the video clip is repeated T times to generate intra-negative samples (eq. 1).” This article discloses different methods to alter frames of data. One process disclosed is called Repeat. This will select a frame to be repeated. The frame can come from a frame in the sets of frames.)
“generating the negative sample by updating an appearance of the second data segment with the data frame.” (Inter and Intra Inputs, pp. 5270; “Then no frame changes exist in this video clip and the corresponding temporal information should have been broken, even though the spatial information is almost the same as its source. [See Equation (1)].” The Repeat method is used to generate a set of frames or samples that would appear different to the anchor. The anchor consists of a set of consecutive frames would not be similar to a negative sample which is generated by repeating a single frame.)
Regarding claim 7, Tao discloses, “generating a noise data frame based on the data frame; and” (Inter and Intra Inputs, pp. 5270; “Clip Rotation: Rotation is one pretext task that is used in self-supervised learning [41], [85]. In videos, when one video clip is rotated using eq. 3, where the angle is large, the movement direction is changed. In such cases, the rotated video clip should represent a different motion from the original one. [See equation (3)]” A third method is disclosed in this article. This will alter images by rotating them. This generates new negative samples which is interpreted to be noise because a function is applied to alter the frames.)
“updating a data frame of the second data segment with the noise data frame.” (Inter and Intra Inputs, pp. 5270; “Here, we introduce intra-negative samples in contrastive learning for videos by breaking the temporal relationship. For one video clip, the data
x
i
1
is a set of frames. To simplify, we use
{
f
r
a
m
e
1
,
…
,
f
r
a
m
e
T
}
to represent a set of temporally-ordered frames. Three different intra-negative generation methods, frame repeating, temporal shuffling, and clip rotation, are proposed to break the temporal relationship and generate intra-negative video clips (Fig. 3). Though similar transformation might have been used in other works, we are the first to use them to generate negative samples in video self-supervised learning.” This article discloses different methods to generate negative samples. This will take frames and alter them and then store them as negative samples. This teaches the use of updating data with generated negative samples.)
Regarding claim 8, Tao discloses, “generating an intermediate data frame by adjusting a dimension of the data frame based on a predetermined ratio.” (Inter and Intra Inputs, pp. 5270; “3) Clip Rotation: Rotation is one pretext task that is used in self-supervised learning [41], [85]. In videos, when one video clip is rotated using eq. 3, where the angle θ is large, the movement direction is changed. In such cases, the rotated video clip should represent a different motion from the original one. [See equation (3)]” The rotation method disclosed in this article can change the appearance of the images in the set. This method would also change the dimension of the data by changing the orientation of the frames. This applies a function to a set of frames and will change the rotational dimension of these frames by an angle θ.)
“generating a plurality of copied intermediate data frames by copying the intermediate data frame; and” (Inter and Intra Inputs, pp. 5270; “1) Frame Repeating: One frame that is randomly selected from the video clip is repeated T times to generate intra-negative samples (eq. 1). Then no frame changes exist in this video clip and the corresponding temporal information should have been broken, even though the spatial information is almost the same as its source. [See equation (1)].” One of the methods used to generate negative samples is the repeat method. This method will select a frame and copy and repeat it n number of times. The article discloses that they did test using all 3 of these methods, however it is noted that they only select altering one method per experiment.)
“generating the noise data by joining the plurality of copied intermediate data frames.” (Inter and Intra Inputs, pp. 5207; “(1) Frame Repeating: One frame that is randomly selected from the video clip is repeated T times to generate intra-negative samples (eq. 1). Then no frame changes exist in this video clip and the corresponding temporal information should have been broken, even though the spatial information is almost the same as its source. [See equation (1)].” This method will generate a set of copied frames and it will introduce the set back into the negative samples.)
Regarding claim 10, Kalantidis discloses, “the number of the first set of negative samples, a ratio of the first set of negative samples to the plurality of negative samples.” (Mixing the hardest negatives, pp. 5; “Given a query q, its key k and negative/queue features
n
∈
Q
from a queue of size K, the loss for the query is composed of logits
l
z
i
=
q
T
z
i
/
τ
fed into a softmax function. Let
Q
~
=
{
n
1
,
…
,
n
k
}
be the ordered set of all negative features, such that:
l
n
i
>
l
n
j
,
∀
i
<
j
, i.e. the set of negative features sorted by decreasing similarity to that particular query feature.” The negative samples in this article are stored in an ordered set. Then the first set of samples are considered while the other set of samples are considered to be too negative. This teaches a set of negative samples; the first n samples are considered as the top ratio.)
Regarding claim 11, Tao discloses, “updating the contrastive learning model with the update function.” (Contrastive Learning, pp. 5271; “The function is trained by selecting a single positive sample from a set of data. After feature
v
i
1
has been extracted, traditional contrastive learning methods train this function to correctly select a positive sample out of a set
S
2
=
{
v
1
2
,
…
,
v
i
2
,
…
,
v
k
+
1
2
}
, which contains one positive sample
v
i
2
and k negative samples. In our proposed method, another set
S
2
=
{
v
1
n
e
g
,
…
,
v
k
+
1
n
e
g
}
is also used that only contains negative samples. The loss function is similar to recent works for contrastive learning [24], [27], [86]: [see equation (4)] Here, k is the number of negative samples, which can be equal to N − 1, where N is the total number of training samples. We randomly select k samples from N where k
≪
N to accelerate training.” The model in this article and each of the other prior proposed disclose a training method that uses a loss value to update the models. This article discloses an updating function as stated above.)
Regarding claim 12, Kalantidis discloses, “determining an association between a first data segment and a second data segment of a pair of samples to be processed with an updated contrastive learning model.” (Contrastive learning with memory, pp. 3; “Let the query q and key k embeddings form the positive pair, which is contrasted with every feature n in the bank of negatives (Q) also called the queue in [21]. A popular and highly successful loss function for contrastive learning [8, 21, 38] is the following: [See equation (1)] where
τ
is a temperature parameter and all embeddings are
l
2
-normalized. In a number of recent successful approaches [8, 21, 31, 39] the query and key are the embeddings of two augmentations of the same image.” This model will generate a bank of negative samples and perform operation interactively based on a query. This will use an ordered set which is based on similarity to the target feature.)
Regarding claim 13, Tao discloses, “An electronic device, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions to be executed by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform a method for managing a model based on a distance between samples, the method comprising:” (Experiments, pp. 5272; “There are several existing labeled benchmark datasets in video recognition: UCF101 [74], HMDB51 [75], something-something [91], and Kinetics400 [6]. The UCF101 dataset contains 13,320 videos, which consists of 101 different action categories. HMDB51 consists of 6,849 videos containing 51 action classes. Kinetics400 is a much larger dataset, consisting of around 240k videos. For fair comparisons with existing works [17], [27], [49], we followed their settings and used UCF101 dataset to conduct self-supervised learning part and used UCF101 and HMDB51 datasets for evaluation. Both UCF101 and HMDB51 datasets have three data splits. And if not specially declared, results are averaged over three splits. Self-supervised learning part can be treated as the pre-training period. And larger dataset can bring further improvements. Thus, we also used the Kinetics400 dataset to pre-train our network for further improvements.” This article discloses different experiments performed using the methods disclosed in this article. This uses databases that can be accessed on generic computing devices which contains central processors linked to memory which holds machine instructions for the system to perform.)
“obtaining a basic sample for training a contrastive learning model and a plurality of negative samples associated with the basic sample;” (Sample Selection, pp. 5269; “Our proposed method is closely related to contrastive learning, using anchor, positive, and negative samples to train.” This model uses contrastive learning and it trains models using a video data which is segmented.) and (figure 2, pp. 5269; This figure shows that a video i is a set of frames. Within these frames is a subset of frames called the anchor which is interpreted to be the basic sample. Frames outside of the anchor can be considered to be the intra or inter negative frames and is interpreted to be the plurality of negative samples.)
PNG
media_image1.png
299
696
media_image1.png
Greyscale
“generating a sequence of the plurality of negative samples based on distances between the plurality of negative samples and the basic sample;” (Inter and Intra Inputs, pp. 5270; “Then
x
n
e
g
(
e
i
t
h
e
r
x
r
e
p
e
a
t
,
x
s
h
u
f
f
l
e
,
x
r
o
t
a
t
i
o
n
)
is used to represent an intra-negative sample from
x
1
. We also want to address that the generated intra-negative samples share similar pixel value distributions with the original one (Fig. 4). From the figure, we can find that the pixel value distributions for the anchor (Video1: view1), the positive (Video1: view2), and intra-negative samples (Video1: repeat, Video1: shuffle, Video1: rotation) are close to each other, constraining the model to learn more discriminative temporal information from video clips.” The model in this article creates new negative sample frames which are used to train the model. The negatives frames have a value denoting their distance or similarity to the anchor and positive frames.)
“dividing the sequence of the plurality of negative samples into a first set of negative samples and a second set of negative samples, a first distance between a first negative sample in the first set of negative samples and the basic sample being less than a second distance between a second negative sample in the second set of negative samples and the basic sample; and” (Figure 8, pp. 5277; “Feature distance distribution. The feature L2 distances are calculated using samples pairs from UCF101 split 1. For each sample pair, one is the anchor, and the other one could be intra-positive, inter-negative, or intra negative sample. The parameters of the network are randomly initialized without optimization. Curves are obtained using kernel density estimation (KDE).” Each of the samples are divided and a distance is calculated between the different samples. The arrows are added by the examiner to point to different examples since the image will show in black and white.)
PNG
media_image2.png
256
355
media_image2.png
Greyscale
Tao fails to explicitly disclose, “determining an update parameter for updating the contrastive learning model based on the basic sample, the first set of negative samples and a first weight of the first set of negative samples, and the second set of negative samples and a second weight of the second set of negative samples, the first weight being greater than the second weight.”.
However, Kalantidis discloses, “determining an update parameter for updating the contrastive learning model based on the basic sample, the first set of negative samples and a first weight of the first set of negative samples, and the second set of negative samples and a second weight of the second set of negative samples, the first weight being greater than the second weight.” (Mixing the hardest negatives, pp. 5; “Given a query q, its key k and negative/queue features
n
∈
Q
from a queue of size K, the loss for the query is composed of logits
l
z
i
=
q
T
z
i
/
τ
fed into a softmax function. Let
Q
~
=
{
n
1
,
…
n
k
}
be the ordered set of all negative features, such that:
l
n
i
>
l
n
i
,
∀
i
<
j
, i.e. the set of negative features sorted by decreasing similarity to that particular query feature. For each query, we propose to synthesize s hard negative features, by creating convex linear combinations of pairs of its “hardest” existing negatives. We define the hardest negatives by truncating the ordered set
Q
~
, i.e. only keeping the first
N
<
K
items. Formally, let
H
=
{
h
1
,
…
h
s
}
be the set of synthetic points to be generated. Then, a synthetic point
h
k
∈
H
, would be given by: [See Equation (3)]
n
i
,
n
j
∈
Q
~
N
are randomly chosen negative features from the set
Q
~
N
=
{
n
1
,
…
n
N
}
of the closest N negatives,
α
k
∈
(
0,1
)
is a randomly chosen mixing coefficient and
∙
2
is the
l
2
-norm. After mixing, the logits
l
(
h
k
)
are computed and appended as further negative logits for query q. The process repeats for each query in the batch.” The process in this article will store the negative samples in an ordered list. Meaning that during updating the model the samples with the most negative features, or the samples that would induce the highest weight parameters changes, would be evaluated first.)
Regarding claim 14, Tao discloses, “selecting, from a first data sequence of a plurality of data sequences for training the contrastive learning model, a first data segment as the basic sample; and” (Figure 1, pp. 5266; “Given video i and video j, two sampled video clips from video i are treated as the anchor and intra-positive samples, whose features are constrained to be similar to each other.” As seen in the figure, a first data item is selected to be the anchor frames. This is interpreted to be the basic samples)
“selecting, from a second sequence of the plurality of data sequences, a second data segment as a negative sample of the plurality of negative samples.” (Figure 1, pp. 5266; “Data sampled from video j is treated as the negative sample. We generated intra-negative samples from the anchor sample by breaking its temporal relations, which can be treated as hard-negatives because they share similar spatial information but different motion features, and can force the model to learn better more discriminative temporal information.” As seen in the figure, there are negative samples drawn from the same and video i and different video j. This image shows the sets of frames.)
Regarding claim 15, Tao discloses, “selecting a third data segment from the first data sequence as a positive sample associated with the basic sample; and” (Contrastive Learning, pp. 5270; “Contrastive learning uses anchor, positive, and negative samples and aims to extract discriminative features from the anchor and negative samples while maintaining the similarity between the anchor and positive samples. In traditional contrastive learning methods (e.g., CMC [27]), the sample pairs
{
x
i
1
,
x
i
2
}
are positives while
{
x
i
1
,
x
i
2
}
(
i
≠
j
)
are negatives. Because intra-negative samples are used in our approach, the negative pairs are extended by adding
{
x
i
1
,
x
j
n
e
g
}
, where j can be equal to i.” The model in this article will use positive data items in the training as well. As seen in figure 2, the positive frames come from the original set of frames or video i. The first sequence of data is interpreted to be the whole video i)
“wherein determining the update parameter further comprises: determining the update parameter based on the basic sample and the positive sample.” (Contrastive Learning, pp. 5271; “The function is trained by selecting a single positive sample from a set of data. After feature
v
i
1
has been extracted, traditional contrastive learning methods train this function to correctly select a positive sample out of a set
S
2
=
{
v
1
2
,
…
,
v
i
2
,
…
,
v
k
+
1
2
}
, which contains one positive sample
v
i
2
and k negative samples. In our proposed method, another set
S
2
=
{
v
1
n
e
g
,
…
,
v
k
+
1
n
e
g
}
is also used that only contains negative samples. The loss function is similar to recent works for contrastive learning [24], [27], [86]: [see equation (4)] Here, k is the number of negative samples, which can be equal to N − 1, where N is the total number of training samples. We randomly select k samples from N where k
≪
N to accelerate training.” This model is trained using loss values. This would be interpreted to be the update parameter.)
Regarding claim 17, Tao discloses, “generating the negative sample by updating at least one of an appearance and a sampling frequency of a plurality of data frames of the second data segment.” (Inter and Intra inputs, pp. 5270; “To simplify, we use
{
f
r
a
m
e
1
,
…
,
f
r
a
m
e
T
}
to represent a set of temporally-ordered frames. Three different intra-negative generation methods, frame repeating, temporal shuffling, and clip rotation, are proposed to break the temporal relationship and generate intra-negative video clips (Fig. 3).” This article discloses different methods of altering video frames. These methods include adjusting the appearance, 3) Rotation, and sampling frequency, 1) Repeat. Repeat changes the sampling from multiple frames to a repeating a single frame.)
Regarding claim 18, Tao discloses, selecting a data frame from a data sequence other than the second data sequence in the plurality of data sequences; and” (Inter and Intra Inputs, pp. 5270; “One frame that is randomly selected from the video clip is repeated T times to generate intra-negative samples (eq. 1).” This article discloses different methods to alter frames of data. One process disclosed is called Repeat. This will select a frame to be repeated. The frame can come from a frame in the sets of frames.)
“generating the negative sample by updating an appearance of the second data segment with the data frame.” (Inter and Intra Inputs, pp. 5270; “Then no frame changes exist in this video clip and the corresponding temporal information should have been broken, even though the spatial information is almost the same as its source. [See Equation (1)].” The Repeat method is used to generate a set of frames or samples that would appear different to the anchor. The anchor consists of a set of consecutive frames would not be similar to a negative sample which is generated by repeating a single frame.)
Regarding claim 19, Tao discloses, “generating a noise data frame based on the data frame; and” (Inter and Intra Inputs, pp. 5270; “Clip Rotation: Rotation is one pretext task that is used in self-supervised learning [41], [85]. In videos, when one video clip is rotated using eq. 3, where the angle is large, the movement direction is changed. In such cases, the rotated video clip should represent a different motion from the original one. [See equation (3)]” A third method is disclosed in this article. This will alter images by rotating them. This generates new negative samples which is interpreted to be noise because a function is applied to alter the frames.)
“updating a data frame of the second data segment with the noise data frame.” (Inter and Intra Inputs, pp. 5270; “Here, we introduce intra-negative samples in contrastive learning for videos by breaking the temporal relationship. For one video clip, the data
x
i
1
is a set of frames. To simplify, we use
{
f
r
a
m
e
1
,
…
,
f
r
a
m
e
T
}
to represent a set of temporally-ordered frames. Three different intra-negative generation methods, frame repeating, temporal shuffling, and clip rotation, are proposed to break the temporal relationship and generate intra-negative video clips (Fig. 3). Though similar transformation might have been used in other works, we are the first to use them to generate negative samples in video self-supervised learning.” This article discloses different methods to generate negative samples. This will take frames and alter them and then store them as negative samples. This teaches the use of updating data with generated negative samples.)
Regarding claim 20, Tao discloses, “A non-transitory computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing a method for managing a model based on a distance between samples, the method comprising:” (Experiments, pp. 5272; “There are several existing labeled benchmark datasets in video recognition: UCF101 [74], HMDB51 [75], something-something [91], and Kinetics400 [6]. The UCF101 dataset contains 13,320 videos, which consists of 101 different action categories. HMDB51 consists of 6,849 videos containing 51 action classes. Kinetics400 is a much larger dataset, consisting of around 240k videos. For fair comparisons with existing works [17], [27], [49], we followed their settings and used UCF101 dataset to conduct self-supervised learning part and used UCF101 and HMDB51 datasets for evaluation. Both UCF101 and HMDB51 datasets have three data splits. And if not specially declared, results are averaged over three splits. Self-supervised learning part can be treated as the pre-training period. And larger dataset can bring further improvements. Thus, we also used the Kinetics400 dataset to pre-train our network for further improvements.” This article discloses different experiments performed using the methods disclosed in this article. This uses databases that can be accessed on generic computing devices which contains central processors linked to memory which holds machine instructions for the system to perform.)
“obtaining a basic sample for training a contrastive learning model and a plurality of negative samples associated with the basic sample;” (Sample Selection, pp. 5269; “Our proposed method is closely related to contrastive learning, using anchor, positive, and negative samples to train.” This model uses contrastive learning and it trains models using a video data which is segmented.) and (figure 2, pp. 5269; This figure shows that a video i is a set of frames. Within these frames is a subset of frames called the anchor which is interpreted to be the basic sample. Frames outside of the anchor can be considered to be the intra or inter negative frames and is interpreted to be the plurality of negative samples.)
PNG
media_image1.png
299
696
media_image1.png
Greyscale
“generating a sequence of the plurality of negative samples based on distances between the plurality of negative samples and the basic sample;” (Inter and Intra Inputs, pp. 5270; “Then
x
n
e
g
(
e
i
t
h
e
r
x
r
e
p
e
a
t
,
x
s
h
u
f
f
l
e
,
x
r
o
t
a
t
i
o
n
)
is used to represent an intra-negative sample from
x
1
. We also want to address that the generated intra-negative samples share similar pixel value distributions with the original one (Fig. 4). From the figure, we can find that the pixel value distributions for the anchor (Video1: view1), the positive (Video1: view2), and intra-negative samples (Video1: repeat, Video1: shuffle, Video1: rotation) are close to each other, constraining the model to learn more discriminative temporal information from video clips.” The model in this article creates new negative sample frames which are used to train the model. The negatives frames have a value denoting their distance or similarity to the anchor and positive frames.)
“dividing the sequence of the plurality of negative samples into a first set of negative samples and a second set of negative samples, a first distance between a first negative sample in the first set of negative samples and the basic sample being less than a second distance between a second negative sample in the second set of negative samples and the basic sample; and” (Figure 8, pp. 5277; “Feature distance distribution. The feature L2 distances are calculated using samples pairs from UCF101 split 1. For each sample pair, one is the anchor, and the other one could be intra-positive, inter-negative, or intra negative sample. The parameters of the network are randomly initialized without optimization. Curves are obtained using kernel density estimation (KDE).” Each of the samples are divided and a distance is calculated between the different samples. The arrows are added by the examiner to point to different examples since the image will show in black and white.)
PNG
media_image2.png
256
355
media_image2.png
Greyscale
Tao fails to explicitly disclose, “determining an update parameter for updating the contrastive learning model based on the basic sample, the first set of negative samples and a first weight of the first set of negative samples, and the second set of negative samples and a second weight of the second set of negative samples, the first weight being greater than the second weight.”.
However, Kalantidis discloses, “determining an update parameter for updating the contrastive learning model based on the basic sample, the first set of negative samples and a first weight of the first set of negative samples, and the second set of negative samples and a second weight of the second set of negative samples, the first weight being greater than the second weight.” ((Mixing the hardest negatives, pp. 5; “Given a query q, its key k and negative/queue features
n
∈
Q
from a queue of size K, the loss for the query is composed of logits
l
z
i
=
q
T
z
i
/
τ
fed into a softmax function. Let
Q
~
=
{
n
1
,
…
n
k
}
be the ordered set of all negative features, such that:
l
n
i
>
l
n
i
,
∀
i
<
j
, i.e. the set of negative features sorted by decreasing similarity to that particular query feature. For each query, we propose to synthesize s hard negative features, by creating convex linear combinations of pairs of its “hardest” existing negatives. We define the hardest negatives by truncating the ordered set
Q
~
, i.e. only keeping the first
N
<
K
items. Formally, let
H
=
{
h
1
,
…
h
s
}
be the set of synthetic points to be generated. Then, a synthetic point
h
k
∈
H
, would be given by: [See Equation (3)]
n
i
,
n
j
∈
Q
~
N
are randomly chosen negative features from the set
Q
~
N
=
{
n
1
,
…
n
N
}
of the closest N negatives,
α
k
∈
(
0,1
)
is a randomly chosen mixing coefficient and
∙
2
is the
l
2
-norm. After mixing, the logits
l
(
h
k
)
are computed and appended as further negative logits for query q. The process repeats for each query in the batch.” The process in this article will store the negative samples in an ordered list. Meaning that during updating the model the samples with the most negative features, or the samples that would induce the highest weight parameters changes, would be evaluated first.)
Claims 4, 9, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Tao and Kalantidis in view of Wang et al, (Wang et al, “Long-Short Temporal Contrastive Learning of Video Transformers”, 2022, hereinafter “Wang”).
Regarding claim 4, Tao discloses, “wherein determining the update parameter further comprises: determining the update parameter based on the third set of negative samples and a third weight of the third set of negative samples, the third weight being greater than the second weight.” (Contrastive Learning, pp. 5271; “The function is trained by selecting a single positive sample from a set of data. After feature
v
i
1
has been extracted, traditional contrastive learning methods train this function to correctly select a positive sample out of a set
S
2
=
{
v
1
2
,
…
,
v
i
2
,
…
,
v
k
+
1
2
}
, which contains one positive sample
v
i
2
and k negative samples. In our proposed method, another set
S
2
=
{
v
1
n
e
g
,
…
,
v
k
+
1
n
e
g
}
is also used that only contains negative samples. The loss function is similar to recent works for contrastive learning [24], [27], [86]: [see equation (4)] Here, k is the number of negative samples, which can be equal to N − 1, where N is the total number of training samples. We randomly select k samples from N where k
≪
N to accelerate training.” The process of updating the models in this article is iterative meaning it will iterate over the different generated samples. This teaches that this process will occur n number of times with continently altered data.)
Tao and Kalantidis fail to explicitly disclose, “selecting a fourth data segment from the first data sequence;”, “adjusting a sampling frequency of a plurality of data frames in the fourth data segment to generate a fifth data segment;”, and “generating a third set of negative samples for training the contrastive learning model based on the fifth data segment; and”.
However, Wang discloses, “selecting a fourth data segment from the first data sequence;” (Long-Short Temporal Contrastive Learning, pp. 13993; “Our proposed Long-Short Temporal Contrastive Learning (LSTCL) framework takes as input a pair of clips sampled from the same video–a long clip and a short clip.” This article uses video segments of different time lengths added to the negative and positive pairs. This will receive sample clips of video which can be from any video sources or sets of sequences.)
“adjusting a sampling frequency of a plurality of data frames in the fourth data segment to generate a fifth data segment;” (Long-Short Temporal Contrastive Learning, pp. 13993; “Given a batch B of unlabeled training videos, we randomly sample a short clip and a long clip from each of them. While both the long and the short clip include a total of T frames, we use largely different sampling temporal strides τS and τL with τS < τL in order for the long clip to cover a much longer temporal extent than the short clip.” This model will segment clips into short and long segments. The model will also use clips of a designated size. This altering of the video lengths is interpreted as adjusting the sampling frequency based on given parameters.)
“generating a third set of negative samples for training the contrastive learning model based on the fifth data segment; and” (Long-Short Temporal Contrastive Learning, pp. 13993; “The sets of short and long clips in the batch B are denoted as
X
s
=
{
x
S
1
,
x
S
2
,
…
,
x
S
B
}
and
X
L
=
{
x
L
1
,
x
L
2
,
…
,
x
L
B
}
, respectively, where
x
S
i
and
x
L
i
represent the short clip and the long clip sampled from the i-th example in the batch. The set of short clips is processed by an encoder
f
q
to yield a set of “query” examples
Q
=
{
q
1
,
q
2
,
…
,
q
B
}
where
q
i
=
f
q
(
x
S
i
)
∈
R
D
. The set of long clips is processed by a separate encoder
f
q
to produce “key” examples
K
=
{
K
1
,
K
2
,
…
,
K
B
}
.” This model will produce sets of short and long clips which can be used for training a contrastive learning model.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Tao, Kalantidis and Wang. Tao teaches contrastive learning methods that generate and relies on negative samples to train models. Kalantidis teaches contrastive learning methods that use “hard negative” samples in training contrastive learning models and emphasizes the importance of negative samples. Wang teaches methods to encode and evaluate video data using contrastive learning. One of ordinary skill would have motivation to combine a machine learning method that uses contrastive learning which generates negative samples to further improve machine learning models with an article that emphasizes the importance of using hard negative samples while training contrastive learning modes, with a system that is designed to evaluate video data using contrastive learning methods, “HMDB51 & UCF101. Finally, we assess the ability to transfer the unsupervised representation learned by LSTCL from Kinetics-400 to the small-scale datasets of HMDB [37] and UCF101 [56] via supervised finetuning. The results are shown in Table 5 where we include also accuracies obtained via fully-supervised pretraining (using class labels) on IN-1K and K400 and also the two recent self-supervised methods ρBYOL [20] and BraVe [51]. It can be seen that LSTCL out performs both (i) the previous state-of-the-art unsupervised pretraining methods, and (ii) the supervised pretraining baselines on both datasets.” (Wang, Comparison to the State-of-the-Art, pp. 13996)
Regarding claim 9, Wang discloses, “generating the negative sample by adjusting the sampling frequency of the plurality of data frames of the second data segment.” (Long-Short Temporal Contrastive Learning, pp. 13993; “Given a batch B of unlabeled training videos, we randomly sample a short clip and a long clip from each of them. While both the long and the short clip include a total of T frames, we use largely different sampling temporal strides τS and τL with τS < τL in order for the long clip to cover a much longer temporal extent than the short clip.” The model in this article will use video clips of different lengths as training samples. This article discloses that the clips are randomly selected segments. The frequency is interpreted to be the short, regular and long video clips)
Regarding claim 16, Tao discloses, “wherein determining the update parameter further comprises: determining the update parameter based on the third set of negative samples and a third weight of the third set of negative samples, the third weight being greater than the second weight.” (Contrastive Learning, pp. 5271; “The function is trained by selecting a single positive sample from a set of data. After feature
v
i
1
has been extracted, traditional contrastive learning methods train this function to correctly select a positive sample out of a set
S
2
=
{
v
1
2
,
…
,
v
i
2
,
…
,
v
k
+
1
2
}
, which contains one positive sample
v
i
2
and k negative samples. In our proposed method, another set
S
2
=
{
v
1
n
e
g
,
…
,
v
k
+
1
n
e
g
}
is also used that only contains negative samples. The loss function is similar to recent works for contrastive learning [24], [27], [86]: [see equation (4)] Here, k is the number of negative samples, which can be equal to N − 1, where N is the total number of training samples. We randomly select k samples from N where k
≪
N to accelerate training.” The process of updating the models in this article is iterative meaning it will iterate over the different generated samples. This teaches that this process will occur n number of times with continently altered data.)
Tao and Kalantidis fail to explicitly disclose, “selecting a fourth data segment from the first data sequence;”, “adjusting a sampling frequency of a plurality of data frames in the fourth data segment to generate a fifth data segment;”, and “generating a third set of negative samples for training the contrastive learning model based on the fifth data segment; and”.
However, Wang discloses, “selecting a fourth data segment from the first data sequence;” (Long-Short Temporal Contrastive Learning, pp. 13993; “Our proposed Long-Short Temporal Contrastive Learning (LSTCL) framework takes as input a pair of clips sampled from the same video–a long clip and a short clip.” This article uses video segments of different time lengths added to the negative and positive pairs. This will receive sample clips of video which can be from any video sources or sets of sequences.)
“adjusting a sampling frequency of a plurality of data frames in the fourth data segment to generate a fifth data segment;” (Long-Short Temporal Contrastive Learning, pp. 13993; “Given a batch B of unlabeled training videos, we randomly sample a short clip and a long clip from each of them. While both the long and the short clip include a total of T frames, we use largely different sampling temporal strides τS and τL with τS < τL in order for the long clip to cover a much longer temporal extent than the short clip.” This model will segment clips into short and long segments. The model will also use clips of a designated size. This altering of the video lengths is interpreted as adjusting the sampling frequency based on given parameters.)
“generating a third set of negative samples for training the contrastive learning model based on the fifth data segment; and” (Long-Short Temporal Contrastive Learning, pp. 13993; “The sets of short and long clips in the batch B are denoted as
X
s
=
{
x
S
1
,
x
S
2
,
…
,
x
S
B
}
and
X
L
=
{
x
L
1
,
x
L
2
,
…
,
x
L
B
}
, respectively, where
x
S
i
and
x
L
i
represent the short clip and the long clip sampled from the i-th example in the batch. The set of short clips is processed by an encoder
f
q
to yield a set of “query” examples
Q
=
{
q
1
,
q
2
,
…
,
q
B
}
where
q
i
=
f
q
(
x
S
i
)
∈
R
D
. The set of long clips is processed by a separate encoder
f
q
to produce “key” examples
K
=
{
K
1
,
K
2
,
…
,
K
B
}
.” This model will produce sets of short and long clips which can be used for training a contrastive learning model.)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PAUL MICHAEL GALVIN-SIEBENALER whose telephone number is (571)272-1257. The examiner can normally be reached Monday - Friday 8AM to 5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at (571) 270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PAUL M GALVIN-SIEBENALER/Examiner, Art Unit 2147
/VIKER A LAMARDO/Supervisory Patent Examiner, Art Unit 2147
Prosecution Insights