Response to Restriction/Election
This action is in response to the Applicant’s response to restriction/election requirement, dated 2/11/2026. The Applicant has elected Group II corresponding claims 1-10, 13-23 and 25 which includes generic claims of Group 1). The restriction/election requirement is made FINAL.
CLAIM INTERPRETATION
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following claim limitations have been interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because it they use a generic placeholder “module” coupled with functional language and without reciting sufficient structure to achieve the function. Furthermore, the generic placeholder is not preceded by a structural modifier:
Claim 1 (dependent claims 2-10):
An action recognition system comprising:
an action module trained to recognize performance of predetermined actions in videos;
a matrix module configured to determine similarity matrices for a predetermined number of support videos, respectively, based on comparisons of (a) temporally ordered images of a query video with (b) temporally ordered images of the support videos, respectively,
the predetermined number of support videos being less than 100 support videos,
the query video including performance of a new action that is not one of the predetermined actions;
a similarity module including the transformer architecture and configured to determine similarity values for the support videos based on the similarity matrices determined based on the support videos, respectively,
wherein the action module is configured to:
determine which one of the support videos has the highest one of the similarity values; and
set a first indicator of the action in the query video to the same as a second indicator of the new action performed in the one of the support videos having the highest similarity value.
Claim 13:
An action recognition system comprising:
an action module trained to recognize performance of predetermined actions in videos;
a matrix module configured to determine a similarity matrix based on comparisons of (a) temporally ordered images of a query video with (b) temporally ordered images of a support videos,
the query video including performance of a new action that is not one of the predetermined actions, and
the support video including performance of the action; and
a similarity module including the transformer architecture and configured to determine a similarity value for the support video based on the similarity matrix determined based on the query video and the support video,
wherein the action module is configured to set a first indicator of the new action in the query video to the same as a second indicator of the action performed in the one of the support videos.
Claim 25:
An action recognition method comprising:
by an action module trained to recognize performance of predetermined actions in videos, recognizing performance of the predetermined actions in videos;
determining a similarity matrix based on comparisons of (a) temporally ordered images of a query video with (b) temporally ordered images of a support videos,
the query video including performance of a new action that is not one of the predetermined actions, and
the support video including performance of the action;
by a similarity module including the transformer architecture, determining a similarity value for the support video based on the similarity matrix determined based on the query video and the support video; and
by the action module, setting a first indicator of the new action in the query video to the same as a second indicator of the action performed in the one of the support videos.
If Applicant asserts that the claim element “unit” is a limitation that does not invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, 6th paragraph. If applicant does not wish to have the claim limitation treated under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, 6th Paragraph applicant may:
(a) Amend the claim to add structure, material or acts that are sufficient to perform the claimed function; or
(b) Present a sufficient showing that the claim limitation recites sufficient structure, material, or acts for performing the claimed function to preclude application of 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. For more information, see MPEP § 2181.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-10, 13-23 and 25 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The limitations, under their broadest reasonable interpretation, cover mental process i.e. abstract idea (concept performed in a human mind, including as observation, evaluation, judgment, opinion, organizing human activity and mathematically performing data conversion). This judicial exception is not integrated into a practical application because the steps do not add meaningful limitations to be considered specifically applied to a particular technological problem to be solved .The claims 1-10. 13-23 and 25 do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the steps of the claimed invention can be done mentally using paper/pencil, mathematically performing calculations, comparing/matching, obtaining similarity of data and no additional features in the claims would preclude them from being performed as such except for the generic computer elements at high level of generality (i.e., processor, memory) .
According to the USPTO guidelines, a claim is directed to non-statutory subject matter if:
STEP 1: the claim does not fall within one of the four statutory categories of invention (process, machine, manufacture or composition of matter), or
STEP 2: the claim recites a judicial exception, e.g. an abstract idea, without reciting additional elements that amount to significantly more than the judicial exception, as determined using the following analysis:
STEP 2A (PRONG 1): Does the claim recite an abstract idea, law of nature, or natural phenomenon?
STEP 2A (PRONG 2): Does the claim recite additional elements that integrate the judicial exception into a practical application?
STEP 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
Using the two-step inquiry, it is clear that claims 1, are directed to an abstract idea as shown below:
Regarding independent claims 1, 13, 14 and 25
STEP 1: Do the claims fall within one of the statutory categories?
YES.
Claim(s) 1, 13, 14 and 25 are directed to an action recognition system and an action recognition i.e. system and process.
STEP 2A (PRONG 1): Is the claim directed to a law of nature, a natural phenomenon or an abstract idea?
YES.
The claims are directed toward a mental process and solving mathematical problem (i.e. abstract idea).
With regard to STEP 2A (PRONG 1), the guidelines provide three groupings of subject matter that are considered abstract ideas:
Mathematical concepts – mathematical relationships, mathematical formulas or equations, mathematical calculations;
Certain methods of organizing human activity – fundamental economic principles or practices (including hedging, insurance, mitigating risk); commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations); managing personal behavior or relationships or interactions between people (including social activities, teaching, and following rules or instructions); and
Mental processes – concepts that are practicably performed in the human mind (including an observation, evaluation, judgment, opinion).
Claims 1, 13, 14 and 25 comprise a mental process that can be practicably performed in the human mind and solving mathematical problem (or generic computers or components configured to perform the process) and, therefore, an abstract idea.
Regarding claims 1, 13, 14 and 25 (representative claim 1):
An action recognition system comprising:
an action module trained to recognize performance of predetermined actions in videos (generic computing hardware, person or observer visually recognizing movement, motion or activity in the collected video i.e. mental process);
a matrix module configured to determine similarity matrices for a predetermined number of support videos, respectively, based on comparisons of (a) temporally ordered images of a query video with (b) temporally ordered images of the support videos, respectively (generic computing hardware, person based on visual observation obtaining similarity of pair of video set using mathematical relationship by driving time sequence relationship between pair of video set using paper pencil i.e. mental process and solving mathematical relationship),
the predetermined number of support videos being less than 100 support videos (collection of video data is insignificant extra solution activity, collected data of first set of video data is no more than 100),
the query video including performance of a new action that is not one of the predetermined actions (collection of video data is insignificant extra solution activity, collected data of second set of video data);
a similarity module including the transformer architecture and configured to determine similarity values for the support videos based on the similarity matrices determined based on the support videos, respectively (generic computing hardware, determining similarity between pair of video set by solving based on mathematical relationship of function ( transformer and matrices) i.e. mental process and solving mathematical relationship),
the action module is configured to (generic computing hardware, person or observer visually recognizing movement, motion or activity in the collected pair of video set i.e. mental process) :
determine which one of the support videos has the highest one of the similarity values (determining which one of second set of video is most similar to first set of collected i.e. mental process based on solving mathematical relationship); and
set a first indicator of the action in the query video to the same as a second indicator of the new action performed in the one of the support videos having the highest similarity value (labeling the pair of video set most similar based on movement, motion and activity similarity i.e. hand crafting the labels i.e. mental process).
The above limitations, as drafted, is a simple process that, under their broadest reasonable interpretation, covers performance of the limitations in the mind or by a human intelligence and solving mathematical problem. Furthermore limitations, “an action module trained to recognize performance of predetermined actions in videos (generic computing hardware, person or observer visually recognizing movement, motion or activity in the collected video i.e. mental process), a matrix module configured to determine similarity matrices for a predetermined number of support videos, respectively, based on comparisons of (a) temporally ordered images of a query video with (b) temporally ordered images of the support videos, respectively (generic computing hardware, person based on visual observation obtaining similarity of pair of video set using mathematical relationship by driving time sequence relationship between pair of video set using paper pencil i.e. mental process and solving mathematical relationship), the predetermined number of support videos being less than 100 support videos (collection of video data is insignificant extra solution activity, collected data of first set of video data is no more than 100), the query video including performance of a new action that is not one of the predetermined actions (collection of video data is insignificant extra solution activity, collected data of second set of video data); a similarity module including the transformer architecture and configured to determine similarity values for the support videos based on the similarity matrices determined based on the support videos, respectively (generic computing hardware, determining similarity between pair of video set by solving based on mathematical relationship of function ( transformer and matrices) i.e. mental process and solving mathematical relationship), the action module is configured to (generic computing hardware, person or observer visually recognizing movement, motion or activity in the collected pair of video set i.e. mental process) : determine which one of the support videos has the highest one of the similarity values (determining which one of second set of video is most similar to first set of collected i.e. mental process based on solving mathematical relationship) and set a first indicator of the action in the query video to the same as a second indicator of the new action performed in the one of the support videos having the highest similarity value (labeling the pair of video set most similar based on movement, motion and activity similarity i.e. hand crafting the labels i.e. mental process)” are insignificant.
The Examiner notes that under MPEP 2106.04(A) (2) (III), the courts consider a mental process (thinking, human intelligence) that can be performed in the mind/intelligence using a paper and pencil to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). As the Federal Circuit explained, "methods which can be performed mentally, or which are the equivalent of human mental work, are unpatentable abstract ideas the ‘basic tools of scientific and technological work’ that are open to all.’" 654 F.3d at 1371, 99 USPQ2d at 1694 (citing Gottschalk v. Benson, 409 U.S. 63, 175 USPQ 673 (1972)). See also Mayo Collaborative Servs. v. Prometheus Labs. Inc., 566 U.S. 66, 71, 101 USPQ2d 1961, 1965 ("‘[Mental processes and abstract intellectual concepts are not patentable, as they are the basic tools of scientific and technological work’" (quoting Benson, 409 U.S. at 67, 175 USPQ at 675)); Parker v. Flook, 437 U.S. 584, 589, 198 USPQ 193, 197 (1978).
Furthermore the Examiner also notes that even if you combined the math with the mental process, a combination of abstract ideas don't make a claim eligible. See MPEP 2106.04(II)(A)(2): Because a judicial exception is not eligible subject matter, Bilski, 561 U.S. at 601, 95 USPQ2d at 1005-06 (quoting Chakrabarty, 447 U.S. at 309, 206 USPQ at 197 (1980)), if there are no additional claim elements besides the judicial exception, or if the additional claim elements merely recite another judicial exception, that is insufficient to integrate the judicial exception into a practical application. See, e.g., RecogniCorp, LLC v. Nintendo Co., 855 F.3d 1322, 1327, 122 USPQ2d 1377 (Fed. Cir. 2017) ("Adding one abstract idea (math) to another abstract idea (encoding and decoding) does not render the claim non-abstract").
Other than generic and well-known computer components recited in the independent claims 1 , 13, 14 and 25 i.e. modules which are generic computer hardware as disclosed in the specification, nothing in the claims 1, 13. 14 and 25 elements preclude the processing from being performed as mental process, or merely based on the observations, evaluation, judgement, thought process and solving mathematical problem solving. An action module trained to recognize performance of predetermined actions in videos, a matrix module configured to determine similarity matrices for a predetermined number of support videos, respectively, based on comparisons of (a) temporally ordered images of a query video with (b) temporally ordered images of the support videos, respectively, and the action module is configured to: determine which one of the support videos has the highest one of the similarity values and set a first indicator of the action in the query video to the same as a second indicator of the new action performed in the one of the support videos having the highest similarity value as recited in independent claims 1, 13 14 and 25 is a mere idea of a solution without details per MPEP 2106.05( f ) or the idea of a technological environment without detail per MPEP 2106.05 ( h ). The generic computing hardware (modules) and machine learning and training are recited in the claims as just to automate the mental process of observation, judgement and mathematical problem solving (Step 2A, prong 2 Test Abstract idea = Yes).
STEP 2A (PRONG 2): Does the claim recite additional elements that integrate the judicial exception into a practical application?
[YES/NO].
The claims do not recite additional elements that integrate the judicial exception into a practical application.
With regard to STEP 2A (prong 2), whether the claim recites additional elements that integrate the judicial exception into a practical application, the guidelines provide the following exemplary considerations that are indicative that an additional element (or combination of elements) may have integrated the judicial exception into a practical application:
an additional element reflects an improvement in the functioning of a computer, or an improvement to other technology or technical field;
an additional element that applies or uses a judicial exception to affect a particular treatment or prophylaxis for a disease or medical condition;
an additional element implements a judicial exception with, or uses a judicial exception in conjunction with, a particular machine or manufacture that is integral to the claim;
an additional element effects a transformation or reduction of a particular article to a different state or thing; and
an additional element applies or uses the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception.
While the guidelines further state that the exemplary considerations are not an exhaustive list and that there may be other examples of integrating the exception into a practical application, the guidelines also list examples in which a judicial exception has not been integrated into a practical application:
an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea;
an additional element adds insignificant extra-solution activity to the judicial exception; and
an additional element does no more than generally link the use of a judicial exception to a particular technological environment or field of use.
Claim(s) 1, 13, 14 and 25 do not recite any of the exemplary considerations that are indicative of an abstract idea having been integrated into a practical application. Claim(s) 1, 13, 14 and 25 recite(s) the limitations of:
Regarding claims 1, 13, 14 and 25 (representative claim 1):
An action recognition system comprising:
an action module trained to recognize performance of predetermined actions in videos (generic computing hardware, person or observer visually recognizing movement, motion or activity in the collected video i.e. mental process);
a matrix module configured to determine similarity matrices for a predetermined number of support videos, respectively, based on comparisons of (a) temporally ordered images of a query video with (b) temporally ordered images of the support videos, respectively (generic computing hardware, person based on visual observation obtaining similarity of pair of video set using mathematical relationship by driving time sequence relationship between pair of video set using paper pencil i.e. mental process and solving mathematical relationship),
the predetermined number of support videos being less than 100 support videos (collection of video data is insignificant extra solution activity, collected data of first set of video data is no more than 100),
the query video including performance of a new action that is not one of the predetermined actions (collection of video data is insignificant extra solution activity, collected data of second set of video data);
a similarity module including the transformer architecture and configured to determine similarity values for the support videos based on the similarity matrices determined based on the support videos, respectively (generic computing hardware, determining similarity between pair of video set by solving based on mathematical relationship of function ( transformer and matrices) i.e. mental process and solving mathematical relationship),
the action module is configured to (generic computing hardware, person or observer visually recognizing movement, motion or activity in the collected pair of video set i.e. mental process) :
determine which one of the support videos has the highest one of the similarity values (determining which one of second set of video is most similar to first set of collected i.e. mental process based on solving mathematical relationship); and
set a first indicator of the action in the query video to the same as a second indicator of the new action performed in the one of the support videos having the highest similarity value (labeling the pair of video set most similar based on movement, motion and activity similarity i.e. hand crafting the labels i.e. mental process).
These limitations are recited at a high level of generality (i.e. as a general action or calculation being taken based on the results of the acquiring steps) and amounts to mere post solution actions, which is a form of insignificant extra-solution activity without further detail. Furthermore, claims 1, 13, 14 and 25 are claimed generically and are operating in their ordinary capacity such that they do not use the judicial exception in a manner that imposes a meaningful limitation on the judicial exception. Accordingly, even in combination, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limitations on practicing the abstract idea.
Other than generic and well-known computer components recited in the independent claims 1 , 13, 14 and 25 i.e. modules which are generic computer hardware as disclosed in the specification, nothing in the claims 1, 13. 14 and 25 elements preclude the processing from being performed as mental process, or merely based on the observations, evaluation, judgement, thought process and solving mathematical problem solving. An action module trained to recognize performance of predetermined actions in videos, a matrix module configured to determine similarity matrices for a predetermined number of support videos, respectively, based on comparisons of (a) temporally ordered images of a query video with (b) temporally ordered images of the support videos, respectively, and the action module is configured to: determine which one of the support videos has the highest one of the similarity values and set a first indicator of the action in the query video to the same as a second indicator of the new action performed in the one of the support videos having the highest similarity value as recited in independent claims 1, 13 14 and 25 is a mere idea of a solution without details per MPEP 2106.05( f ) or the idea of a technological environment without detail per MPEP 2106.05 ( h ). The generic computing hardware (modules) and machine learning and training are recited in the claims as just to automate the mental process of observation, judgement and mathematical problem solving (Step 2A, prong 2 Test Abstract idea = Yes).
STEP 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
NO.
The claims 1, 17 and 18 do not recite additional elements that amount to significantly more than the judicial exception.
With regard to STEP 2B, whether the claims recite additional elements that provide significantly more than the recited judicial exception, the guidelines specify that the pre-guideline procedure is still in effect. Specifically, that examiners should continue to consider whether an additional element or combination of elements:
adds a specific limitation or combination of limitations that are not well-understood, routine, conventional activity in the field, which is indicative that an inventive concept may be present; or
simply appends well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception, which is indicative that an inventive concept may not be present.
As noted above other than generic and well-known computer components recited in the independent claims 1 , 13, 14 and 25 i.e. modules which are generic computer hardware as disclosed in the specification, nothing in the claims 1, 13. 14 and 25 elements preclude the processing from being performed as mental process, or merely based on the observations, evaluation, judgement, thought process and solving mathematical problem solving. An action module trained to recognize performance of predetermined actions in videos, a matrix module configured to determine similarity matrices for a predetermined number of support videos, respectively, based on comparisons of (a) temporally ordered images of a query video with (b) temporally ordered images of the support videos, respectively, and the action module is configured to: determine which one of the support videos has the highest one of the similarity values and set a first indicator of the action in the query video to the same as a second indicator of the new action performed in the one of the support videos having the highest similarity value as recited in independent claims 1, 13 14 and 25 is a mere idea of a solution without details per MPEP 2106.05( f ) or the idea of a technological environment without detail per MPEP 2106.05 ( h ). The generic computing hardware (modules) and machine learning and training are recited in the claims as just to automate the mental process of observation, judgement and mathematical problem solving.
Thus, since Claim(s) 1, 13, 14 and 25 are: (a) directed toward an abstract idea, (b) do not recite additional elements that integrate the judicial exception into a practical application, and (c) do not recite additional elements that amount to significantly more than the judicial exception, it is clear that Claim(s) 1, 17 and 18 are not eligible subject matter under 35 U.S.C 101 (Step 2B, Test Abstract idea = Yes).
Regarding dependent claims 2-10 and 15-23 : the additional limitations of dependent claims 2-10 and 15-21, do not integrate the mental process into practical application or add significantly more to the abstract idea of mental process of observation, judgement and solving mathematical relationship. Claims 2-10 and 15-21 further limit the abstract idea of independent claims 1, 13, 14 and 25. The limitations of these dependent claims fall under (mental process including observation and evaluation, and judgement and mathematical problem solving of mathematical relation-ship which can be done mentally in the human mind based on human intelligence using paper and pencil ) OR (insignificant pre/post-solution extra activity of generating/gathering data, performing mathematical calculation) OR (generic computers or components (modules) configured to perform the process and machine training and learning), An action module trained to recognize performance of predetermined actions in videos, a matrix module configured to determine similarity matrices for a predetermined number of support videos, respectively, based on comparisons of (a) temporally ordered images of a query video with (b) temporally ordered images of the support videos, respectively, and the action module is configured to: determine which one of the support videos has the highest one of the similarity values and set a first indicator of the action in the query video to the same as a second indicator of the new action performed in the one of the support videos having the highest similarity value is a mere idea of a solution without details per MPEP 2106.05( f ) or the idea of a technological environment without detail per MPEP 2106.05 ( h ). The generic computing hardware, machine learning/training and robot are recited as just to automate the mental process of observation, judgement and mathematical problem solving
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-8, 13-21 and 25 are rejected under 35 U.S.C. 103 as being unpatentable over Bishay et al. (TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognition, arXiv:1907.09021 v1 cs.CV1, 21 Jul 2019, pages 1-14, USPTO-892) in view of Debnath et al. (Cosine Similarity based Few Shot Video Classifier with Attention-based Aggregation, 2022 IEEE 978-16654-9062-7/22, 2022 26th Internation Conference on Pattern Recognition (ICPR), pages 173-1279, USPTO-892)
Regarding claims 1, 13, 14 and 25 Bishay disclose an action recognition system/method (Bishay Title and page 1, Abstract lines 1-5, disclose in this paper we propose a novel Temporal Attentive Relation Network (TARN) for the problems of few-shot and zero-shot action recognition. At the heart of our network is a meta-learning approach that learns to compare representations of variable temporal length, that is, either two videos of different length (in the case of few-shot action recognition) or a video and a semantic representation such as word vector. This corresponds to action recognition system/process ): comprising:
an action module trained to recognize performance of predetermined actions in videos (Bishay page 1, Abstract lines 1-5, disclose in this paper we propose a novel Temporal Attentive Relation Network (TARN) for the problems of few-shot and zero-shot action recognition. At the heart of our network is a meta-learning approach that learns to compare representations of variable temporal length and section 1 Introduction, page 2, third-paragraph Bishay disclose network (TARN) addresses the few-shot problem by working at video-segment level to calculate the relation scores between a query video and other sample videos– the query video is then assigned with the label of the most related video in the sample set. The relation/similarity is calculated in two stages: the embedding stage and the relation stage and page 4, section 3. Proposed Architecture disclose in this section we introduce a novel deep architecture, named Temporal Attentive Relation Network(TARN) for the problems of Few-Shot Learning(FSL)and Zero-Shot Learning (ZSL) for video-based tasks. Figure1 shows an overview of the network. TARN learns to compare a query video against a sample set of videos in FSL, or semantic attributes in ZSL, representing a group of actions. This action recognition based on temporal relationship i.e. similarity obviously corresponds action recognition trained to recognize performance of predetermined actions in videos based on similarity of video segments between query video and sample videos);
a matrix module configured to determine similarity matrices for a predetermined number of support videos, respectively, based on comparisons of (a) temporally ordered images of a query video with (b) temporally ordered images of the support videos, respectively (Bishay section 1 Introduction, page 2, third-paragraph Bishay disclose network (TARN) addresses the few-shot problem by working at video-segment level to calculate the relation scores between a query video and other sample videos– the query video is then assigned with the label of the most related video in the sample set. The relation/similarity is calculated in two stages: the embedding stage and the relation stage and section 3.2, page 5. Relation Module, second-paragraph thru page 6 first-paragraph Bishay disclose Segment-by-segment attention: Several recent works in text sequence matching and textual entailment use an attention mechanism, named word-by-word attention, to align the words of two given sentences [2, 28, 31, 44]. Similarly, as shown in the corresponding block of Fig. 1, we adopt the word-by-word attention in our architecture to align the sample and query segment-embeddings (i.e. segment-by-segment attention). Given a sample video S ∈ R N×d and a query video Q ∈ R M×d, where each row in S and Q represents a segment embedding vector of dimension d, and where N and M denote the number of segments in videos S and Q respectively. The segment-by-segment attention is calculated as shown in equation (1) A=softmax((SSW W+b⊗eN)QT), H =ATS and page 6 Bishay disclose where W ∈Rd×d and b∈Rd are parameters to be learned, and the operator “⊗eN” repeats the bias vector b, N times to form a matrix of dimension N ×d. A ∈ RN×M is the attention weight matrix and H is the aligned version of S. Each row vector in H is a weighted sum of the S segment-embeddings, and represents the parts of S that are most similar to the corresponding row vector (segment-embedding) of Q. The row vectors of Q and H are used as inputs to a comparison layer This obviously corresponds to a matrix module configured to determine similarity matrices for a predetermined number of support videos, respectively, based on comparisons of (a) temporally ordered images of a query video with (b) temporally ordered images of the support videos, respectively),
the predetermined number of support videos being less than 100 support videos and the support video includes performance of action (Bishay ,section 1 Introduction, page 2, third-paragraph Bishay disclose network (TARN) addresses the few-shot problem by working at video-segment level to calculate the relation scores between a query video and other sample videos– the query video is then assigned with the label of the most related video in the sample set. Bishay page 4, section 3. Proposed Architecture disclose in this section we introduce a novel deep architecture, named Temporal Attentive Relation Network(TARN) for the problems of Few-Shot Learning(FSL) and Zero-Shot Learning (ZSL) for video-based tasks. Figure 1 shows an overview of the network. TARN learns to compare a query video against a sample set of videos in FSL, or semantic attributes in ZSL, representing a group of actions. Since the few-shot problem is solved by working at video-segment level to calculate the relation scores between a query video and other sample videos it is obvious that support videos i.e. sample videos and their number is less than 100 and as disclosed by Bishay page 4, section 3 the support video includes performance of action. Therefore it is obvious that the system of Bishay includes the predetermined number of support videos being less than 100 support videos and the support video includes performance of action),
the query video including performance of a new action that is not one of the predetermined actions (Bishay ,section 1 Introduction, page 2, third-paragraph Bishay disclose network (TARN) addresses the few-shot problem by working at video-segment level to calculate the relation scores between a query video and other sample videos– the query video is then assigned with the label of the most related video in the sample set and a page 4, section 3. Proposed Architecture disclose in this section we introduce a novel deep architecture, named Temporal Attentive Relation Network(TARN) for the problems of Few-Shot Learning(FSL)and Zero-Shot Learning (ZSL) for video-based tasks. Figure1 shows an overview of the network. TARN learns to compare a query video against a sample set of videos in FSL, or semantic attributes in ZSL, representing a group of actions. In system of Bishay it be would obvious that the query video performance of a new action which is compare to samples video set or semantic attributes represent group of actions to similarity however it is possible that the query video includes action which is not included in sample videos actions which includes only few shots);
a similarity module including the transformer architecture and configured to determine similarity values for the support videos based on the similarity matrices determined based on the support videos, respectively (Bishay page 4, paragraph Sequence matching: lines 1-9 disclose matching video sequences can benefit from comparing video segments as a first step. Previously, Fernando et al.[6] have proposed a method to match video segments from a pair of videos, that share similar temporal evolutions. We also draw our inspiration from works on text matching, where representations of sequence parts are compared and aggregated to match an entire text sequence[44]. An attention mechanism similar to that of [2,31] is used to align the segments of each video in the support set, with respect to the query video. This is done to semantically align the features of the support set videos to the features of the query video. It also transforms the number of segments for each video of the support set to be equal to that of the query video and note: similarity matrix in section 3.2, page 5. Relation Module, second-paragraph thru page 6 first-paragraph disclose similarity matrix. All this obviously corresponds to a similarity module including the transformer architecture and configured to determine similarity values for the support videos based on the similarity matrices determined based on the support videos, respectively and also note: page 6, section Deep metric learning, first and second paragraphs. Furthermore as stated above page 6 first paragraph disclose similarity matrix and page 6, section Deep metric learning, first-paraph disclose deep learning cosine similarity i.e. a similarity module including the transformer architecture and configured to determine similarity values for the support videos based on the similarity matrices determined based on the support videos),
the action module is configured to (Bishay Abstract disclose Abstract lines 1-5, disclose in this paper we propose a novel Temporal Attentive Relation Network (TARN) for the problems of few-shot and zero-shot action recognition and section 1 Introduction, page 2, third-paragraph Bishay disclose network (TARN) addresses the few-shot problem by working at video-segment level to calculate the relation scores between a query video and other sample videos– the query video is then assigned with the label of the most related video in the sample set.) :
determine which one of the support videos has the highest one of the similarity values (Bishay section 1 Introduction, page 2, third-paragraph Bishay disclose network (TARN) addresses the few-shot problem by working at video-segment level to calculate the relation scores between a query video and other sample videos– the query video is then assigned with the label of the most related video in the sample set. The relation/similarity is calculated in two stages: the embedding stage and the relation stage and page 4, section 3. Proposed Architecture disclose in this section we introduce a novel deep architecture, named Temporal Attentive Relation Network(TARN) for the problems of Few-Shot Learning(FSL)and Zero-Shot Learning (ZSL) for video-based tasks. Figure1 shows an overview of the network. TARN learns to compare a query video against a sample set of videos in FSL, or semantic attributes in ZSL, representing a group of actions This obviously corresponds to determine which one of the support videos has the highest one of the similarity values and also note: page 6, section Deep metric learning, first and second paragraphs ); and
set a first indicator of the action in the query video to the same as a second indicator of the new action performed in the one of the support videos having the highest similarity value (Bishay section 1 Introduction, page 2, third-paragraph Bishay disclose network (TARN) addresses the few-shot problem by working at video-segment level to calculate the relation scores between a query video and other sample videos– the query video is then assigned with the label of the most related video in the sample set. The relation/similarity is calculated in two stages: the embedding stage and the relation stage and page 4, section 3. Proposed Architecture disclose in this section we introduce a novel deep architecture, named Temporal Attentive Relation Network(TARN) for the problems of Few-Shot Learning(FSL)and Zero-Shot Learning (ZSL) for video-based tasks. Figure1 shows an overview of the network. TARN learns to compare a query video against a sample set of videos in FSL, or semantic attributes in ZSL, representing a group of actions. This obviously corresponds to set a first indicator of the action in the query video to the same as a second indicator of the new action performed in the one of the support videos having the highest similarity value i.e. labelling and also note: page 6, section Deep metric learning, first and second paragraphs).
In the same field of endeavor Debnath of video and action based on few shot (Debnath, Abstract, especially last lines 28-34 and section I. INTRODUCTION, page 1273, right-column, last paragraph Debnath disclose few -video shot video classification tasks),
Debnath disclose a similarity module including the transformer architecture and configured to determine similarity values for the support videos based on the similarity matrices determined based on the support videos, respectively (Debnath page 1274, section Few-shot, right-column, lines 1-10 disclose The limitations of existing video classification models in dealing with limited amount of data has fueled the development of models that can effectively recognize novel classes with only a few labeled examples [6], [4], which is known as few-shot learning for video classification. One of the earliest work on few-shot video classification, CMN [6] utilizes a multi-saliency algorithm to encode each frame into matrix representation which is later utilized with a Compound Memory Network (CMN) for matching and ranking videos, Debnath, page 1275, section Vision Transformer, left-column, lines 1-10, disclose Our proposed network is inspired by the Vision Transformer (ViT) [30]. ViT employs a Transformer like architecture over patches of the image. It splits an image into fixed-size patches and adds position embeddings with them, and then feeds the resulting sequence to a Transformer encoder. It adds an extra learnable “classification token” to the sequence in order to perform the classification task. In contrast to ViT, our patches are generated by the ResNet and we employ a cosine-similarity-based classifier instead of a linear classifier and Debnath, Fig. 3, section B. Our proposed method, page 1276, right-column below description of Fig. 3, first paragraph, in this paper, instead of the linear classifier, we utilize a distance-based classifier that explicitly reduces intra class variation among features during the training. In particular, we incorporate a cosine similarity function between feature representations and classification weight vectors to compute raw classification scores (see Figure 3d). Here, for an extracted feature f✓+(xi) and weight matrix W+=[w1,w2,...wc], we obtain the similarity scores [si,1,si,2,...,sic] for all c classes by calculating its cosine similarity to each weight vector [w1,w2,..., wc]. All this obviously corresponds to a similarity module including the transformer architecture and configured to determine similarity values for the support videos based on the similarity matrices determined based on the support videos).
Therefore it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to use similarity module including the transformer architecture and configured to determine similarity values for the support videos based on the similarity matrices determined based on the support videos, as shown by combination of Bishay and Debnath because such a process provide accurate few-shot video classification and accurately discern between video actions as stated by Debnath in the Abstract lines 28-34.
Regarding claims 2 and 16 Bishay disclose the predetermined number of support videos is less than or equal to 5 support videos (Bishay Title and page 1, Abstract lines 1-5, disclose in this paper we propose a novel Temporal Attentive Relation Network (TARN) for the problems of few-shot and zero-shot action recognition. At the heart of our network is a meta-learning approach that learns to compare representations of variable temporal length, that is, either two videos of different length (in the case of few-shot action recognition) or a video and a semantic representation such as word vector. This corresponds to action recognition system/process and Bishay page 1, Abstract lines 1-5, disclose in this paper we propose a novel Temporal Attentive Relation Network (TARN) for the problems of few-shot and zero-shot action recognition. At the heart of our network is a meta-learning approach that learns to compare representations of variable temporal length and section 1 Introduction, page 2, third-paragraph Bishay disclose network (TARN) addresses the few-shot problem by working at video-segment level to calculate the relation scores between a query video and other sample videos– the query video is then assigned with the label of the most related video in the sample set. Since Bishay idisclose Temporal Attentive Relation Network (TARN) for the problems of few-shot and zero-shot action recognition there it would be obvious to the predetermined number of support videos [sample videos] is less than or equal to 5 support videos).
Regarding claims 3 and 17 Bishay first fully connected linear layer configured to generate first vector representations of the support videos and output the first vector representations to the matrix module; and a second fully connected linear layer configured to generate a second vector representation of the query vid and output the second vector representation to the matrix module, the matrix module is configured to generate the similarity matrices based on the second vector representation and the first vector representations (Bishay section 1 Introduction, page 2, third-paragraph Bishay disclose network (TARN) addresses the few-shot problem by working at video-segment level to calculate the relation scores between a query video and other sample videos– the query video is then assigned with the label of the most related video in the sample set. The relation/similarity is calculated in two stages: the embedding stage and the relation stage and section 3.2, page 5. Relation Module, second-paragraph thru page 6 first-paragraph Bishay disclose Segment-by-segment attention: Several recent works in text sequence matching and textual entailment use an attention mechanism, named word-by-word attention, to align the words of two given sentences [2, 28, 31, 44]. Similarly, as shown in the corresponding block of Fig. 1, we adopt the word-by-word attention in our architecture to align the sample and query segment-embeddings (i.e. segment-by-segment attention). Given a sample video S ∈ R N×d and a query video Q ∈ R M×d, where each row in S and Q represents a segment embedding vector of dimension d, and where N and M denote the number of segments in videos S and Q respectively. The segment-by-segment attention is calculated as shown in equation (1) A=softmax((SSW W+b⊗eN)QT), H =ATS and page 6 Bishay disclose where W ∈Rd×d and b∈Rd are parameters to be learned, and the operator “⊗eN” repeats the bias vector b, N times to form a matrix of dimension N ×d. A ∈ RN×M is the attention weight matrix and H is the aligned version of S. Each row vector in H is a weighted sum of the S segment-embeddings, and represents the parts of S that are most similar to the corresponding row vector (segment-embedding) of Q. The row vectors of Q and H are used as inputs to a comparison layer and section 3.2, page 6. Relation Module, third-paragraph disclose fully-connected layer and similarity matrices page 6 second thru third paragraphs).
Regarding claims 4 and 15 Bishay disclose he determining the similarity value includes determining the similarity value by a module including the transformer architecture (Bishay page 4, paragraph Sequence matching: lines 1-9 disclose matching video sequences can benefit from comparing video segments as a first step. Previously, Fernando et al.[6] have proposed a method to match video segments from a pair of videos, that share similar temporal evolutions. We also draw our inspiration from works on text matching, where representations of sequence parts are compared and aggregated to match an entire text sequence[44]. An attention mechanism similar to that of [2,31] is used to align the segments of each video in the support set, with respect to the query video. This is done to semantically align the features of the support set videos to the features of the query video. It also transforms the number of segments for each video of the support set to be equal to that of the query video and note: similarity matrix in section 3.2, page 5. Relation Module, second-paragraph thru page 6 first-paragraph disclose similarity matrix. All this obviously corresponds to a similarity module including the transformer architecture and configured to determine similarity values for the support videos based on the similarity matrices determined based on the support videos, respectively and also note: page 6, section Deep metric learning, first and second paragraphs. Furthermore as stated above page 6 first paragraph disclose similarity matrix and page 6, section Deep metric learning, first-paraph disclose deep learning cosine similarity i.e. a similarity module including the transformer architecture and configured to determine similarity values for the support videos based on the similarity matrices determined based on the support videos ) and
Debnath disclose a similarity module including the transformer architecture and configured to determine similarity values for the support videos based on the similarity matrices determined based on the support videos, respectively (Debnath page 1274, section Few-shot, right-column, lines 1-10 disclose The limitations of existing video classification models in dealing with limited amount of data has fueled the development of models that can effectively recognize novel classes with only a few labeled examples [6], [4], which is known as few-shot learning for video classification. One of the earliest work on few-shot video classification, CMN [6] utilizes a multi-saliency algorithm to encode each frame into matrix representation which is later utilized with a Compound Memory Network (CMN) for matching and ranking videos, Debnath, page 1275, section Vision Transformer, left-column, lines 1-10, disclose Our proposed network is inspired by the Vision Transformer (ViT) [30]. ViT employs a Transformer like architecture over patches of the image. It splits an image into fixed-size patches and adds position embeddings with them, and then feeds the resulting sequence to a Transformer encoder. It adds an extra learnable “classification token” to the sequence in order to perform the classification task. In contrast to ViT, our patches are generated by the ResNet and we employ a cosine-similarity-based classifier instead of a linear classifier and Debnath, Fig. 3, section B. Our proposed method, page 1276, right-column below description of Fig. 3, first paragraph, in this paper, instead of the linear classifier, we utilize a distance-based classifier that explicitly reduces intra class variation among features during the training. In particular, we incorporate a cosine similarity function between feature representations and classification weight vectors to compute raw classification scores (see Figure 3d). Here, for an extracted feature f✓+(xi) and weight matrix W+=[w1,w2,...wc], we obtain the similarity scores [si,1,si,2,...,sic] for all c classes by calculating its cosine similarity to each weight vector [w1,w2,..., wc]. All this obviously corresponds to a similarity module including the transformer architecture and configured to determine similarity values for the support videos based on the similarity matrices determined based on the support videos).
Regarding claims 5 and 18 Bishay disclose similarity module further includes a flattening module configured to convert a received similarity matrix into a vector, transformer module is configured to determine a similarity value based on the vector (section 3.2, page 5. Relation Module, second-paragraph thru page 6 first-paragraph Bishay disclose Segment-by-segment attention: Several recent works in text sequence matching and textual entailment use an attention mechanism, named word-by-word attention, to align the words of two given sentences [2, 28, 31, 44]. Similarly, as shown in the corresponding block of Fig. 1, we adopt the word-by-word attention in our architecture to align the sample and query segment-embeddings (i.e. segment-by-segment attention). Given a sample video S ∈ R N×d and a query video Q ∈ R M×d, where each row in S and Q represents a segment embedding vector of dimension d, and where N and M denote the number of segments in videos S and Q respectively. The segment-by-segment attention is calculated as shown in equation (1) A=softmax((SSW W+b⊗eN)QT), H =ATS and page 6 Bishay disclose where W ∈Rd×d and b∈Rd are parameters to be learned, and the operator “⊗eN” repeats the bias vector b, N times to form a matrix of dimension N ×d. A ∈ RN×M is the attention weight matrix and H is the aligned version of S. Each row vector in H is a weighted sum of the S segment-embeddings, and represents the parts of S that are most similar to the corresponding row vector (segment-embedding) of Q. The row vectors of Q and H are used as inputs to a comparison layer. This obviously corresponds to similarity module further includes a flattening module configured to convert a received similarity matrix into a vector, transformer module is configured to determine a similarity value based on the vector) and
Debnath disclose similarity module further includes a flattening module configured to convert a received similarity matrix into a vector, transformer module is configured to determine a similarity value based on the vector (Debnath, Fig. 3, section B. Our proposed method, page 1276, right-column below description of Fig. 3, first paragraph, in this paper, instead of the linear classifier, we utilize a distance-based classifier that explicitly reduces intra class variation among features during the training. In particular, we incorporate a cosine similarity function between feature representations and classification weight vectors to compute raw classification scores (see Figure 3d). Here, for an extracted feature f✓+(xi) and weight matrix W+=[w1,w2,...wc], we obtain the similarity scores [si,1,si,2,...,sic] for all c classes by calculating its cosine similarity to each weight vector [w1,w2,..., wc] and equation 2 shows similarity matrix conversion into vectors).
Regarding claims 6 and 19 Bishay disclose flattening module is configured to convert the received similarity matrix into a vector by concatenating rows of the received similarity matrix (section 3.2, page 5. Relation Module, second-paragraph thru page 6 first-paragraph Bishay disclose Segment-by-segment attention: Several recent works in text sequence matching and textual entailment use an attention mechanism, named word-by-word attention, to align the words of two given sentences [2, 28, 31, 44]. Similarly, as shown in the corresponding block of Fig. 1, we adopt the word-by-word attention in our architecture to align the sample and query segment-embeddings (i.e. segment-by-segment attention). Given a sample video S ∈ R N×d and a query video Q ∈ R M×d, where each row in S and Q represents a segment embedding vector of dimension d, and where N and M denote the number of segments in videos S and Q respectively. The segment-by-segment attention is calculated as shown in equation (1) A=softmax((SSW W+b⊗eN)QT), H =ATS and page 6 Bishay disclose where W ∈Rd×d and b∈Rd are parameters to be learned, and the operator “⊗eN” repeats the bias vector b, N times to form a matrix of dimension N ×d. A ∈ RN×M is the attention weight matrix and H is the aligned version of S. Each row vector in H is a weighted sum of the S segment-embeddings, and represents the parts of S that are most similar to the corresponding row vector (segment-embedding) of Q. The row vectors of Q and H are used as inputs to a comparison layer. This obviously corresponds to flattening module is configured to convert the received similarity matrix into a vector by concatenating rows of the received similarity matrix).
Debnath disclose flattening module is configured to convert the received similarity matrix into a vector by concatenating rows of the received similarity matrix (Debnath, Fig. 3, section B. Our proposed method, page 1276, right-column below description of Fig. 3, first paragraph, in this paper, instead of the linear classifier, we utilize a distance-based classifier that explicitly reduces intra class variation among features during the training. In particular, we incorporate a cosine similarity function between feature representations and classification weight vectors to compute raw classification scores (see Figure 3d). Here, for an extracted feature f✓+(xi) and weight matrix W+=[w1,w2,...wc], we obtain the similarity scores [si,1,si,2,...,sic] for all c classes by calculating its cosine similarity to each weight vector [w1,w2,..., wc] and equation 2 shows similarity matrix conversion into vectors)
Regarding claims 7 and 20 Bishay disclose similarity module further includes an embedding module configured to embed the vector into an embedding, wherein the transformer module is configured to determine a similarity value based on the embedding. (Bishay page 4, Fig. 1 shows embedding feature vector, page 4, section Sequence Matching transforming the segment and matching and page 5, section 3.1, Embedding Module disclose feature vector embedding and page 6, section Deep Learning cosine similarity (EucCos). This obviously corresponds to similarity module further includes an embedding module configured to embed the vector into an embedding, wherein the transformer module is configured to determine a similarity value based on the embedding)
Regarding claims 8 and 21 Debnath disclose the similarity module further includes a positional encoding module configured to add positional encoding to the embedding, the transformer module is configured to determine a similarity value based on the embedding and the added positional encoding (Debnath, page 1275, section B. Our proposed method, right-column last-two paragraphs “Transformer Encoding” disclose feature vector embedding and position encoding, page 1276, Fig. 3 and description below of Fig. 3 disclose feature vector embedding and Cosine transform for determining similarity. This corresponds to the similarity module further includes a positional encoding module configured to add positional encoding to the embedding, the transformer module is configured to determine a similarity value based on the embedding and the added positional encoding).
Claims 9-10 and 22-23 are rejected under 35 U.S.C. 103 as being unpatentable over Bishay et al. (TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognition, arXiv:1907.09021 v1 cs.CV1, 21 Jul 2019, pages 1-14, USPTO-892) in view of Debnath et al. (Cosine Similarity based Few Shot Video Classifier with Attention-based Aggregation, 2022 IEEE 978-16654-9062-7/22, 2022 26th Internation Conference on Pattern Recognition (ICPR), pages 173-1279, USPTO-892) and further in view of Malla (US 20210129871).
Regarding claim 9, Bishay disclose action recognition in video (Bishay page 1, Abstract lines 1-5, disclose in this paper we propose a novel Temporal Attentive Relation Network (TARN) for the problems of few-shot and zero-shot action recognition. At the heart of our network is a meta-learning approach that learns to compare representations of variable temporal length and section 1 Introduction, page 2, third-paragraph Bishay disclose network (TARN) addresses the few-shot problem by working at video-segment level to calculate the relation scores between a query video and other sample videos– the query video is then assigned with the label of the most related video in the sample set. The relation/similarity is calculated in two stages: the embedding stage and the relation stage and page 4, section 3. Proposed Architecture disclose in this section we introduce a novel deep architecture, named Temporal Attentive Relation Network(TARN) for the problems of Few-Shot Learning(FSL)and Zero-Shot Learning (ZSL) for video-based tasks. Figure1 shows an overview of the network. TARN learns to compare a query video against a sample set of videos.
Bishay however has not explicitly a robot including: an actuator, the action recognition system of configured to recognize in video performance of the predetermined actions and performance of the new action, control module configured to selectively actuate the actuator in response to recognition of an action by the action module in the video.
In same field of action recognition in video Malla disclose a robot (Mall Fig. 1, paragraph 0023 autonomous vehicle i.e. robotic vehicle and paragraph 0036) including:
an actuator (Malla Fig. 1, block 116 vehicle controller, paragraph 0036 disclose The ECU 104 may also include a communication device (not shown) for sending data internally within (e.g., between one or more components) the ego vehicle 102 and communicating with externally hosted computing systems (e.g., external to the ego vehicle 102). Generally, the ECU 104 may communicate with the storage unit 118 of the ego vehicle 102 to execute the one or more applications, operating systems, ego vehicle system and subsystem user interfaces, and the like that are stored within the storage unit 118. As discussed below, the storage unit 118 may be configured to store the neural network 112 and one or more components of the neural network 112 and paragraph 0054 Figs . 1 and 5 , Malla disclose In one embodiment, upon the neural network 112 predicting the future trajectories 206 of the agents 204 and the future ego motion 208 of the ego vehicle 102, the future forecasting application 106 may be configured to update the ego vehicle operation policy 126 with vehicle dynamic parameters that may be based on the future ego motion 208 of the ego vehicle 102, data associated with annotated actions associated with each of the agents 204, and the predicted future trajectories 206 of each of the agents 204. The ego vehicle operation policy 126 may also be updated with data associated with the surrounding environment 200 of the ego vehicle 102 including, but not limited to, locations of static objects 202 and additional environmental attributes (e.g., curvature of roadway, configuration of intersections, and the like). As discussed below, the ego vehicle operation policy 126 may be accessed at one or more future points in time to retrieve dynamic parameters that may correspond to the predicted future ego motion of the ego vehicle 102 to autonomously control the operation of the ego vehicle 102 to smoothly navigate a current environment of the ego vehicle 102 based on action priors. This obviously corresponds to actuator i.e. controlling the vehicle based on the surround environment)
the action recognition system of configured to recognize in video performance of the predetermined actions and performance of the new action (Malla Figs.1, 5-6 paragraph 0085, disclose In one embodiment, upon extracting the vehicle dynamic parameters, the vehicle control module 132 may be configured to update the ego vehicle operation policy 126 with the vehicle dynamic parameters that may be based on the future ego motion 208 of the ego vehicle 102, data associated with annotated actions associated with each of the agents 204, and the predicted future trajectories 206 of each of the agents 204. The ego vehicle operation policy 126 may also be updated with data associated with the surrounding environment 200 of the ego vehicle 102 including, but not limited to, locations of static objects 202 and additional environmental attributes (e.g., curvature of roadway, configuration of intersections, and the like and paragraphs 0089-0090, Fig. 6 disclose The method 600 may proceed to block 604, wherein the method 600 may include analyzing the image data and detecting actions associated with agents 204 located within the surrounding environment of the ego vehicle 102. The method 600 may proceed to block 606, wherein the method 600 may include analyzing the dynamic data and processing an ego motion history of the ego vehicle 102. The method 600 may proceed to block 608, wherein the method 600 may include predicting future trajectories 206 of the agents 204 located within the surrounding environment 200 of the ego vehicle 102 and a future ego motion 208 of the ego vehicle 102 within the surrounding environment of the ego vehicle 102 and paragraph 0027 disclose The future forecasting application 106 may be configured to utilize machine learning/deep learning techniques to incorporate the prior positions, actions, and contexts of the agents to simultaneously forecast future trajectories of agents located within the surrounding environment of the ego vehicle 102 at one or more future time steps (e.g., t+1, t+2, t+n). As discussed below, the future forecasting application 106 may output a predicted future ego motion of the ego vehicle 102 at one or more future time steps that may be based on past and current ego-motions of the ego vehicle 102 and the forecasted future trajectories of agents. The future ego motion of the ego vehicle 102 may be used to autonomously control the ego vehicle 102 to operate in a manner to smoothly navigate within the surrounding environment of the ego vehicle 102. This obviously corresponds to the action recognition system of configured to recognize in video performance of the predetermined actions and performance of the new action) .
control module configured to selectively actuate the actuator in response to recognition of an action by the action module in the video (Malla Figs 1 and 5-6, paragraph 0054 Malla disclose In one embodiment, upon the neural network 112 predicting the future trajectories 206 of the agents 204 and the future ego motion 208 of the ego vehicle 102, the future forecasting application 106 may be configured to update the ego vehicle operation policy 126 with vehicle dynamic parameters that may be based on the future ego motion 208 of the ego vehicle 102, data associated with annotated actions associated with each of the agents 204, and the predicted future trajectories 206 of each of the agents 204. The ego vehicle operation policy 126 may also be updated with data associated with the surrounding environment 200 of the ego vehicle 102 including, but not limited to, locations of static objects 202 and additional environmental attributes (e.g., curvature of roadway, configuration of intersections, and the like). As discussed below, the ego vehicle operation policy 126 may be accessed at one or more future points in time to retrieve dynamic parameters that may correspond to the predicted future ego motion of the ego vehicle 102 to autonomously control the operation of the ego vehicle 102 to smoothly navigate a current environment of the ego vehicle 102 based on action priors.,
and paragraph 0085 Malla disclose In one embodiment, upon extracting the vehicle dynamic parameters, the vehicle control module 132 may be configured to update the ego vehicle operation policy 126 with the vehicle dynamic parameters that may be based on the future ego motion 208 of the ego vehicle 102, data associated with annotated actions associated with each of the agents 204, and the predicted future trajectories 206 of each of the agents 204. The ego vehicle operation policy 126 may also be updated with data associated with the surrounding environment 200 of the ego vehicle 102 including, but not limited to, locations of static objects 202 and additional environmental attributes e.g., curvature of roadway, configuration of intersections, and the like and paragraph 0090 Fig 6 disclose The method 600 may proceed to block 604, wherein the method 600 may include analyzing the image data and detecting actions associated with agents 204 located within the surrounding environment of the ego vehicle 102. The method 600 may proceed to block 606, wherein the method 600 may include analyzing the dynamic data and processing an ego motion history of the ego vehicle 102. The method 600 may proceed to block 608, wherein the method 600 may include predicting future trajectories 206 of the agents 204 located within the surrounding environment 200 of the ego vehicle 102 and a future ego motion 208 of the ego vehicle 102 within the surrounding environment of the ego vehicle 102 This obviously corresponds to control module configured to selectively actuate the actuator in response to recognition of an action by the action module in the video as shown Figs.5-6 in Malla).
Regarding claim 10 Malla disclose camera is configure output the video recognition system is configured receive video from the camera (Malla Figs 1 block 108 and Figs. 5 block 502 and Fig. 6 block 602 and paragraphs 0089-0090).
Regarding claim 22, Malla disclose selectively actuating an actuator of a robot in response to recognition of an action in the query video (Malla Figs. 1, blocks 108, 116 and 120, Figs 5-6, blocks 500-510, 514-518 and blocks 602-608 paragraph 0089 Malla FIG. 6 is a process flow diagram of a method 600 for future forecasting using action priors according to an exemplary embodiment of the present disclosure. FIG. 6 will be described with reference to the components and examples of FIGS. 1-4 though it is to be appreciated that the method 600 of FIG. 6 may be used with other systems/components and in additional exemplary scenarios. The method 600 may begin at block 602, wherein the method 600 may include receiving image data associated with a surrounding environment 200 of an ego vehicle 102 and dynamic data associated with dynamic operation of the ego vehicle 102 and paragraph 0090 Malla disclose The method 600 may proceed to block 604, wherein the method 600 may include analyzing the image data and detecting actions associated with agents 204 located within the surrounding environment of the ego vehicle 102. The method 600 may proceed to block 606, wherein the method 600 may include analyzing the dynamic data and processing an ego motion history of the ego vehicle 102. The method 600 may proceed to block 608, wherein the method 600 may include predicting future trajectories 206 of the agents 204 located within the surrounding environment 200 of the ego vehicle 102 and a future ego motion 208 of the ego vehicle 102 within the surrounding environment of the ego vehicle 102 and paragraph 00027 The future forecasting application 106 may be configured to utilize machine learning/deep learning techniques to incorporate the prior positions, actions, and contexts of the agents to simultaneously forecast future trajectories of agents located within the surrounding environment of the ego vehicle 102 at one or more future time steps (e.g., t+1, t+2, t+n). As discussed below, the future forecasting application 106 may output a predicted future ego motion of the ego vehicle 102 at one or more future time steps that may be based on past and current ego-motions of the ego vehicle 102 and the forecasted future trajectories of agents. The future ego motion of the ego vehicle 102 may be used to autonomously control the ego vehicle 102 to operate in a manner to smoothly navigate within the surrounding environment of the ego vehicle 102. and also note: Fig. 5 blocks 506-518. All this obviously corresponds to selectively actuating an actuator of a robot in response to recognition of an action in the query video i.e. video received from the camera and past actions).
Regarding claim 23 Mall disclose receiving the query video from a camera of the robot (Malla Figs. 1, blocks 108, 116 and 120, Figs 5-6, blocks 500-510, 514-518 and blocks 602-608 paragraph 0089 Malla FIG. 6 is a process flow diagram of a method 600 for future forecasting using action priors according to an exemplary embodiment of the present disclosure. FIG. 6 will be described with reference to the components and examples of FIGS. 1-4 though it is to be appreciated that the method 600 of FIG. 6 may be used with other systems/components and in additional exemplary scenarios. The method 600 may begin at block 602, wherein the method 600 may include receiving image data associated with a surrounding environment 200 of an ego vehicle 102 and dynamic data associated with dynamic operation of the ego vehicle 102 and paragraph 0090 Malla disclose The method 600 may proceed to block 604, wherein the method 600 may include analyzing the image data and detecting actions associated with agents 204 located within the surrounding environment of the ego vehicle 102. The method 600 may proceed to block 606, wherein the method 600 may include analyzing the dynamic data and processing an ego motion history of the ego vehicle 102. The method 600 may proceed to block 608, wherein the method 600 may include predicting future trajectories 206 of the agents 204 located within the surrounding environment 200 of the ego vehicle 102 and a future ego motion 208 of the ego vehicle 102 within the surrounding environment of the ego vehicle 102 and paragraph 00027 The future forecasting application 106 may be configured to utilize machine learning/deep learning techniques to incorporate the prior positions, actions, and contexts of the agents to simultaneously forecast future trajectories of agents located within the surrounding environment of the ego vehicle 102 at one or more future time steps (e.g., t+1, t+2, t+n). As discussed below, the future forecasting application 106 may output a predicted future ego motion of the ego vehicle 102 at one or more future time steps that may be based on past and current ego-motions of the ego vehicle 102 and the forecasted future trajectories of agents. The future ego motion of the ego vehicle 102 may be used to autonomously control the ego vehicle 102 to operate in a manner to smoothly navigate within the surrounding environment of the ego vehicle 102. and also note: Fig. 5 blocks 506-518. This obviously corresponds to receiving the query video from a camera of the robot).
Therefore it would have been obvious to one having ordinary skill in the art before filing of the claimed invention to recognize in video performance of the predetermined actions and performance of the new action, selectively actuate the actuator in response to recognition of an action by the action module in the video, receive the query video from a camera of the robot as shown by Malla because such a system provide controlling ego or autonomous vehicle as stated by Malla in the ABSTRACT and paragraph 0002.
Communication
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ISHRAT I SHERALI whose telephone number is (571)272-7398. The examiner can normally be reached Monday-Friday 8:00AM -5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached on 571-272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ISHRAT I SHERALI/Primary Examiner, Art Unit 2667
ISHRAT I. SHERALI
Examiner
Art Unit 2667
Prosecution Insights