DETAILED ACTION
1. This communication is in response to the amendments filed on August 4, 2026 for Application No. 17/817,373 in which Claims 1-18 are presented for examination.
Notice of Pre-AIA or AIA Status
2. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
3. A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 08/04/2026 has been entered.
Response to Arguments
4. The amendments filed on August 4, 2026 have been considered. Claims 1-4, 6-7, 9-13, and 15-18 have been amended. Thus, Claims 1-18 are pending and presented for examination.
5. Applicant's arguments filed August 4, 2026 with respect to the 35 U.S.C. 101 rejection have been fully considered but they are not persuasive.
Applicant’s Arguments on Pg. 8 of Arguments/Remarks state:
“Applicant respectfully submits that amended independent claim 1 is not directed to an abstract idea.
The limitations of amended independent claim 1 cannot reasonably be characterized as mental processes. Rather, when read as a whole, claim 1 defines a specific artificial intelligence (AI) low-latency processing system that employs cooperatively trained neural networks (a timing detector neural network and a decoding neural network) to identify an early subsequence of frames for decoding event information, which is a technological process that cannot practically be performed in the human mind.
In view of the foregoing remarks, the Applicant respectfully submits that amended independent claim 1 is not directed to an abstract idea under Prong One of Step 2A.”
Examiner respectfully disagrees. As amended, the Independent claims still recite an abstract idea at Step 2A Prong 1 – the limitations “[…] identify an early subsequence […]”, “[…] decode the event information [...], “[…] iteratively identify the early subsequence of frames […]”, “output a token sequence […]”, and “generate a control action based on the token sequence” may all be practically performed by mental process, as further supported by the rejection in the subsequent 35 U.S.C. 101 section below. A user may practically observe/analyze a set of video frames to identify/iteratively identify an early subsequence of frames (subset comprising continuous subframes from a starting time until a current time having less than a total number of frames). Additionally, a user may practically decode event information from the identified early subsequence of frames by observing/analyzing the frames and accordingly using judgement/evaluation to decode information (determine events occurring within the frames and correspondingly provide captions with the aid of pen and paper). Further, a user may observe/analyze the decoded event information to correspondingly output a token sequence (sequence of words/caption) which corresponds to said event information. Finally, a user may also generate a control action (with the aid of pen and paper) based on the observed/analyzed token sequence. Further examples are provided in the subsequent 35 U.S.C. 101 section below.
Furthermore, at Step 2A Prong 2 and Step 2B, the various neural networks, including the timing detector neural network and decoding neural network, are recited at a high-level of generality and amount to merely adding the words “apply it” to the judicial exception without significantly more. As admitted by Applicant on Pg. 8 of Arguments/Remarks, these neural networks are already trained/configured and are merely employed to perform the specific operations of the claim language without significantly more – this cannot provide an inventive concept.
Applicant’s Arguments on Pgs. 8-10 of Arguments/Remarks state:
“Step 2A: Prong Two
Notwithstanding the arguments presented above, assuming arguendo that independent claim 1 is directed to an abstract idea or judicial exception as the Office Action contends, Applicant respectfully submits that amended independent claim 1 nevertheless satisfies Prong Two of the Step 2A analysis under the 2019 PEG.
Even if, arguendo, amended independent claim 1 is viewed as an abstract idea (as the Examiner contends), the claimed features integrate that idea into a practical application, which is sufficient to meet Prong Two of the Step 2A analysis under the 2019 PEG. A judicial exception is considered practically applied if the claim as a whole imposes meaningful limits on the abstract idea, rather than merely attempting to monopolize the idea itself.
Conventional video and multimodal processing systems are inherently limited as they (i) process entire sequences of frames before producing output, resulting in latency, and (ii) lack a mechanism to determine when partial input contains sufficient information for accurate decoding of event information. As a result, such systems are unable to trigger actions on time in latency-sensitive environments such as surveillance systems.
The Specification at paragraphs [0006] and [0042] describes "such prior methods are not practical in real-time monitoring, surveillance systems, ... in which it is essential not only to describe events accurately but also to produce captions as soon as possible to find and report the events quickly to take next actions. Thus, low latency captioning for online systems is required to realize such functionality, where the system needs to decide an appropriate timing to generate a correct caption using only the limited number of frames
the system has received so far. Additionally, such a low-latency captioning function allow robots to decide next actions as soon as possible to interact with humans and other robots."
Thus, there exists a technological problem of determining the earliest time at which sufficient information is available to reliably decode event information and trigger downstream actions in low-latency signal processing systems such as surveillance systems.
Independent claim 1 addresses the above-mentioned technological problem by reciting a low-latency AI processing system including a timing detector neural network jointly trained with a decoding neural network to identify an early subsequence of continuous frames extending from a starting time to a dynamically determined current time based on an output of the timing detector neural network indicating sufficiency of event information. As described in the Specification, this enables the system to determine "appropriate timing" for output generation based on partial input data to identify "the smallest number of subframes ... sufficient to decode the training information." See paragraphs [0042] and [0115] of the Specification.
Further, paragraph [0044] of the Specification describes "embodiments of the present invention train the timing detector 103 and the low-latency caption generator 105 using only datasets for offline video captioning, where the detector and the generator are optimized not only to describe events accurately but also to produce captions as soon as possible. Thus, low-latency video captioning is realized by detecting an appropriate timing to generate a correct caption using only the partial video frames 104 until the current time
t. The system may generate a caption 106, which is equal to the ground truth 102 "One of workers is hit by a bulldozer" even though it uses only partial video frames 104."
Further, paragraph [0124] of the Specification describes "[a]ccording to some embodiments described above, a low-latency video QA method can answer a user's questions accurately and quickly without waiting for the end of video clips. The proposed method optimizes each answer's output timing based on a trade-off between latency and answer quality. The system described above can generate answers in early stages of video clips using the MSRVTT-QA and AVSD datasets, achieving between 97% and 99% of the answer quality of the upper bound given by a pre-trained Transformer using the entire video clips, using less than 40% of frames from the beginning.
This constitutes a technological improvement in low-latency AI processing, as the claimed system enables early and reliable decoding of event information without requiring full sequence processing, thereby reducing latency and improving responsiveness in real- time systems such as surveillance systems. See paragraphs [0121]-[0124] of the Specification.
Therefore, amended independent claim 1 integrates any alleged abstract idea into a practical application and is not directed to a judicial exception under Step 2A, Prong Two.”
Examiner respectfully disagrees for substantially the same reasons as stated above. Although Applicant cites to various technological improvements (low-latency processing, triggering downstream actions, etc.), it must be noted that these improvements are not reflected by the currently drafted claim language. As stated above, the “timing detector neural network” and “decoding neural network” are recited at a high-level of generality and are already trained/configured to perform the operations of the claim language without significantly more. More specifically, the claim states “execute a timing detector neural network trained to identify an early subsequence of frames […]”, “execute a decoding neural network trained to decode the event information in the early subsequence of frames […]”, and “the timing detector neural network is jointly trained with the decoding neural network to iteratively identify the early subsequence of frames […]” – as such, all of these limitations amount to mere conclusory statements of what the neural networks are meant to accomplish through their execution, without explaining how the neural networks are trained to complete such objectives. Therefore, there is no technological improvement reflected by the currently drafted claim language, if the claims merely recite an abstract idea and the application of pre-trained neural networks/machine learning models without significantly more.
Applicant’s arguments additionally state that the identification of an early subsequence of continuous frames enables the system to determine “appropriate timing” for output generation based on partial input data to identify sufficient event information for decoding the event information – however, it must be noted that the term “sufficient” is a relative term under 35 U.S.C. 112(b) (See the corresponding rejection in the 35 U.S.C. 112(b) section below). It is not clear what exactly indicates “sufficient” event information for decoding the event information and moreover, the “joint training” of the timing detector neural network and decoding neural network merely states that the networks are “jointly trained” to again “identify the early subsequence of frames” without significantly more. There are no details provided regarding the architecture of both networks, how both networks are jointly configured/arranged, the training completed by both networks, and/or how the joint training improves the iterative identification of the early subsequence of frames, etc. The claims merely state that both neural networks are executed and loosely describes that the networks are “jointly trained” without any additional details. Although Applicant’s specification may list such technological improvements and supposed benefits of the invention, these improvements and benefits are still not reflected by the currently drafted claim language at Step 2A Prong 2, as outlined above.
Applicant’s Arguments on Pgs. 10-11 of Arguments/Remarks state:
“Notwithstanding the remarks under Step 2A, assuming arguendo, that independent claim 1 is directed to an abstract idea or judicial exception as the Office Action contends, Applicant respectfully submits that amended independent claim 1 amounts to "significantly more" than an abstract idea.
The inventive concept of independent claim 1 resides in the non-generic and non- conventional arrangement of its elements, which together enables early and reliable decoding of event information without requiring full sequence processing, thereby reducing latency and improving responsiveness in real-time systems such as surveillance. Furthermore, the particular ordered combination of claimed elements is not well- understood, routine, or conventional in the art.
The claimed joint training of the timing detector neural network and the decoding neural network constitutes a non-conventional arrangement, as it requires interdependent optimization such that the timing detector learns a decision boundary tied to decoding success. This is not a routine or generic use of neural networks, but rather a specific architecture that integrates timing prediction with decoding performance.
This non-conventional arrangement yields concrete technological advantages, including significantly reduced latency, improved responsiveness, and the ability to generate accurate event representations from partial input data in real time. As described in paragraph [0044], the claimed system can correctly generate event descriptions even when the full event has not yet occurred, unlike prior systems.
Therefore, independent claim 1, when taken as a whole, qualifies as significantly more than an abstract idea under Step 2B.
Accordingly, the claimed features, considered individually and as an ordered combination, provide significantly more than any alleged mental process or other abstract idea identified in the Final Office Action. Applicant therefore respectfully submits that amended independent claim 1 satisfies Step 2B and is patent eligible under 35 U.S.C. § 101.
Amended independent claim 10 recites some features that are analogous to the features of amended independent claim 1. Hence, the remarks presented above for amended independent claim 1 apply equally to amended independent claim 10. Claims 2-9 and 11- 18 depend from and incorporate the limitations of amended independent claims 1 and 10, respectively, and therefore, are allowable by virtue of their dependency over amended independent claims 1 and 10 and as well as for the features they recite individually.
Therefore, the Applicant respectfully requests the rejection of claims 1-18 under 35 U.S.C. § 101 be withdrawn.”
Examiner respectfully disagrees for substantially the same reasons as stated above. The newly added limitation “control a surveillance system based on the control action” is still recited at a high-level of generality and amounts to merely indicating a field of use or technological environment in which to apply a judicial exception and does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application - See MPEP 2106.05(h). The controlling of the surveillance system is simply “based on the control action” without significantly more – this cannot provide an inventive concept. Again, although Applicant lists the supposed technological improvements/advantages, this is not reflected by the currently drafted claim language and the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception at Step 2B. This rationale applies to Independent Claims 1, 10, and their respective dependent claims 2-9 and 11-18.
Thus, the 35 U.S.C. 101 rejection is maintained.
6. Applicant’s arguments filed August 4, 2026 with respect to the 35 U.S.C. 103 rejection have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 112
7. The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
8. Claims 1-18 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. The term “sufficient” in Independent Claims 1 and 10 is a relative term which renders the claim indefinite. The term “sufficient” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The claim language states “[…] the current time corresponds to a time associated with an output of the timing detector neural network indicating availability of sufficient event information for decoding the event information […]” – however, the claim language does not provide any threshold/requisite/degree as to what indicates the availability of “sufficient” event information for decoding the event information, as opposed to “insufficient” event information. Moreover, Applicant’s specification Par. [0015-0016] merely recites the term “sufficient” with respect to decoding the event information without providing any such threshold/requisite/degree to clarify the interpretation of this term. This applies to Independent Claims 1, 10, and their respective dependent claims 2-9 and 11-18.
Claim Rejections - 35 USC § 101
9. 35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
10. Claims 1-18 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1:
Step 1: Claim 1 is a system type claim. Therefore, Claims 1-9 are directed to either a process, machine, manufacture, or composition of matter.
2A Prong 1: If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation by mathematical calculation but for the recitation of generic computer components, then it falls within the “Mathematical Concepts” grouping of abstract ideas.
[…] identify an early subsequence of frames from the sequence of frames including at least a portion of the event information (mental process – other than reciting “execute a timing detector neural network”, identifying an early subsequence of frames may be performed manually by a user observing/analyzing the sequence of frames indicative of information and accordingly using judgement/evaluation to identify an “early” subsequence of frames including at least a portion of the analyzed information. For example, a user may observe/analyze a sequence of video frames and accordingly identify an early subsequence (earlier frames in the sequence/time-series) of frames which represent the video and/or information about the video)
[…] decode the event information in the early subsequence of frames […] (mental process – other than reciting “execute a decoding neural network”, decoding event information in the subsequence of frames may be performed manually by a user observing/analyzing the subsequence of frames and accordingly using judgement/evaluation to decode the information from the portion of the information in the subsequence of frames. For example, a user may observe/analyze video frames and accordingly use judgement/evaluation to determine/generate captions for the video based on said analysis)
[…] iteratively identify the early subsequence of frames, the early subsequence of frames comprises continuous subframes from a starting time of a training sequence of frames up to a current time […] the early subsequence of frames is less than a total number of frames in the training sequence of frames (mental process – other than reciting “wherein the timing detector neural network is jointly trained with the decoding neural network”, iteratively identifying the early subsequence of frames may be performed manually by a user iteratively observing/analyzing the sequence of frames and using judgement/evaluation to identify the early subsequence of frames comprising continuous subframes from a starting time up to a current time and having less than a total number of frames in the entire sequence. For example, the user may observe/analyze a set of video frames and use judgement/evaluation to determine the smallest number of frames representative of the video/information in the video and correspondingly determine/generate captions for these video frames)
output a token sequence corresponding to the event information decoded from the early subsequence of frames (mental process – outputting a token sequence corresponding to the event information decoded from the early subsequence of frames may be performed manually by a user observing/analyzing the decoded event information and accordingly using judgement/evaluation to output/generate (with the aid of pen and paper) a token sequence of words corresponding to the decoded event information, based on the preceding analysis. Applicant’s specification Par. [0048] describes how each token in the token sequence may be a single word, a single letter, a single character, or a word piece in a test form – hence, a user is capable of outputting/generating such a token sequence based on the received decoded event information)
generate a control action based on the token sequence (mental process – generating a control action based on the token sequence may be performed manually by a user observing/analyzing the outputted token sequence and accordingly using judgement/evaluation to generate a control action based on the analyzed token sequence. For example, a user may observe/analyze a token sequence which indicates an unauthorized person is entering a restricted area, as a result, the user may generate a control action of contacting authorities, tracking the unauthorized person’s movement, and/or alerting personnel)
2A Prong 2: This judicial exception is not integrated into a practical application.
Additional elements:
an artificial intelligence (Al) low-latency processing system, the low-latency processing system comprising: a processor; and a memory having instructions stored thereon that, when executed by the processor, cause the low-latency processing system to: […] (recited at a high-level of generality (i.e., as a generic system comprising generic computer components) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
collect a sequence of frames jointly including event information dispersed among at least one or more frames in the sequence of frames (Adding insignificant extra-solution activity to the judicial exception – see MPEP 2106.05(g))
[…] wherein the event information is associated with multimodal sensor signals comprising at least: video frames, sound data, or a combination thereof (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying the information is associated with multimodal sensor signals comprising at least video frames, sound data, or a combination thereof without significantly more does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
execute a timing detector neural network trained to […] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of applying a generically trained machine learning model with previously determined data (information/sequence of frames) without significantly more)
execute a decoding neural network trained to […] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of applying a generically trained machine learning model with previously determined data (information/subsequence of frames) without significantly more)
wherein the timing detector neural network is jointly trained with the decoding neural network to […] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of training a machine learning model with previously determined data (training sequence of frames/training information) without significantly more)
the current time corresponds to a time associated with an output of the timing detector neural network indicating availability of sufficient event information for decoding the event information (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the current time corresponds to a time associated with an output of the timing detector neural network indicating availability of generically sufficient event information for decoding the event information without significantly more does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
control a surveillance system based on the control action (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that a surveillance system is controlled generically based on the control action without significantly more does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
an artificial intelligence (Al) low-latency processing system, the low-latency processing system comprising: a processor; and a memory having instructions stored thereon that, when executed by the processor, cause the low-latency processing system to: […] (mere instructions to apply the exception using generic computer components cannot provide an inventive concept)
collect a sequence of frames jointly including event information dispersed among at least one or more frames in the sequence of frames (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
[…] wherein the event information is associated with multimodal sensor signals comprising at least: video frames, sound data, or a combination thereof (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying the information is associated with multimodal sensor signals comprising at least video frames, sound data, or a combination thereof without significantly more does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
execute a timing detector neural network trained to […] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of applying a generically trained machine learning model with previously determined data (information/sequence of frames) without significantly more. This cannot provide an inventive concept)
execute a decoding neural network trained to […] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of applying a generically trained machine learning model with previously determined data (information/subsequence of frames) without significantly more. This cannot provide an inventive concept)
wherein the timing detector neural network is jointly trained with the decoding neural network to […] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of training a machine learning model with previously determined data (training sequence of frames/training information) without significantly more. This cannot provide an inventive concept)
the current time corresponds to a time associated with an output of the timing detector neural network indicating availability of sufficient event information for decoding the event information (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the current time corresponds to a time associated with an output of the timing detector neural network indicating availability of generically sufficient event information for decoding the event information without significantly more does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
control a surveillance system based on the control action (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that a surveillance system is controlled generically based on the control action without significantly more does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
For the reasons above, Claim 1 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 2-9. The additional limitations of the dependent claims are addressed below.
Regarding Claim 2:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 2 depends on.
minimize a multi-task loss function including a time detection loss and an information generation loss (mathematical process – other than reciting “timing detector neural network is jointly trained with the decoder neural network”, minimizing a multi-task loss function may be performed by mathematical process utilizing a multi-task loss function including a time detection loss and an information generation loss)
Step 2A Prong 2 & Step 2B:
wherein the timing detector neural network is jointly trained with the decoding neural network on features of different subsequences of the sequence of frames to […] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner' s note: high level recitation of training a machine learning model with previously determined data without significantly more)
Accordingly, under Step 2A Prong 2 and Step 2B, these additional elements do not integrate the abstract idea into practical application because they do not impose any meaningful limits on practicing the abstract idea, as discussed above in the rejection of claim 1.
Regarding Claim 3:
Step 2A Prong 1:
See the rejection of Claim 2 above, which Claim 3 depends on.
Step 2A Prong 2 & Step 2B:
wherein the multi-task loss function includes three losses defining an accuracy of decoded event information; a difference between the event information decoded from the early subsequence of frames and the event information decoded from a full sequence of frames; and an accuracy of prediction of the timing detector neural network (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying the loss functions without significantly more does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, these additional elements do not integrate the abstract idea into practical application because they do not impose any meaningful limits on practicing the abstract idea, as discussed above in the rejection of claim 1.
Regarding Claim 4:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 4 depends on.
[…] extract features from each frame in the sequence of frames (mental process – other than reciting “feature extractor”, extracting features from each frame may be performed manually by a user observing/analyzing the sequence of frames and accordingly using judgement/evaluation to identify and extract features from each frame)
[…] encode the extracted features of each frame to produce a sequence of encoded features (mental process – other than reciting “feature encoder”, encoding extracted features may be performed manually by a user observing/analyzing each frame and correspondingly using judgement/evaluation to encode the extracted features of each frame to produce a sequence of encoded features)
[…] identify a subsequence of encoded features representing the early subsequence of frames (mental process – other than reciting “timing detector neural network”, identifying a subsequence of encoded features representing the subsequence of frames may be performed manually by a user observing/analyzing the subsequence of frames and correspondingly using judgement/evaluation to identify a subsequence of encoded features based on said frames)
[…] decode the event information (mental process – other than reciting “decoder neural network”, decoding the information may be performed manually by a user observing/analyzing the information and correspondingly using judgement/evaluation to decode said information)
Step 2A Prong 2 & Step 2B:
wherein the processor is configured to execute a feature extractor […] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner's note: high level recitation of applying an already trained feature extractor without significantly more. This cannot provide an inventive concept)
execute a feature encoder […] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner' s note: high level recitation of applying an already trained feature encoder without significantly more. This cannot provide an inventive concept)
submit the sequence of encoded features to the timing detector neural network […] (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim. Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
submit the subsequence of encoded features to the decoding neural network […] (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim. Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
Accordingly, under Step 2A Prong 2 and Step 2B, these additional elements do not integrate the abstract idea into practical application because they do not impose any meaningful limits on practicing the abstract idea, as discussed above in the rejection of claim 1.
Regarding Claim 5:
Step 2A Prong 1:
See the rejection of Claim 1 above, which Claim 5 depends on.
Step 2A Prong 2 & Step 2B:
wherein the processor triggers the execution of modules of the Al low-latency processing system upon receiving a new input frame appended to the sequence of frames (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the processor triggers execution of modules upon receiving a new input frame does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, these additional elements do not integrate the abstract idea into practical application because they do not impose any meaningful limits on practicing the abstract idea, as discussed above in the rejection of claim 1.
Regarding Claim 6:
Step 2A Prong 1:
See the rejection of Claim 1 above, which Claim 6 depends on.
Step 2A Prong 2 & Step 2B:
wherein the event information is a caption for an audio scene, a video scene, or an audio-video scene (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the information is a caption for an audio/video/audio-video scene does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, these additional elements do not integrate the abstract idea into practical application because they do not impose any meaningful limits on practicing the abstract idea, as discussed above in the rejection of claim 1.
Regarding Claim 7:
Step 2A Prong 1:
See the rejection of Claim 1 above, which Claim 7 depends on.
Step 2A Prong 2 & Step 2B:
wherein the event information is an answer to a question about the sequence of frames. (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the information is an answer to a question about the sequence of frames does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, these additional elements do not integrate the abstract idea into practical application because they do not impose any meaningful limits on practicing the abstract idea, as discussed above in the rejection of claim 1.
Regarding Claim 8:
Step 2A Prong 1: See the rejection of Claim 7 above, which Claim 8 depends on
[…] encode the question (mental process – other than reciting “execute a text encoder neural network”, encoding the question may be performed manually by a user observing/analyzing the question and accordingly using judgement/evaluation to encode said question)
Step 2A Prong 2 & Step 2B:
execute a text encoder neural network (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner' s note: high level recitation of applying an already trained machine learning model without significantly more. This cannot provide an inventive concept)
submit the encoded question to the timing detector neural network (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim. Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
submit the question or the encoded question to the decoding neural network (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim. Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
Accordingly, under Step 2A Prong 2 and Step 2B, these additional elements do not integrate the abstract idea into practical application because they do not impose any meaningful limits on practicing the abstract idea, as discussed above in the rejection of claim 1.
Regarding Claim 9:
Step 2A Prong 1:
See the rejection of Claim 1 above, which Claim 9 depends on.
Step 2A Prong 2 & Step 2B:
wherein the sequence of frames includes multi-modal information coming from different sensors of different modalities (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the sequence of frames include multi-modal information coming from different sensors of different modalities does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, these additional elements do not integrate the abstract idea into practical application because they do not impose any meaningful limits on practicing the abstract idea, as discussed above in the rejection of claim 1.
Independent Claim 10 recites substantially the same limitations as Claim 1, in the form of a method. The claim is also directed to performing mental processes/mathematical calculations without significantly more, therefore it is rejected under the same rationale.
For the reasons above, Claim 10 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 11-18. The additional limitations of the dependent claims are addressed below.
Claim 11 recites substantially the same limitations as Claim 2, in the form of a method. The claim is also directed to performing mental processes/mathematical calculations without significantly more, therefore it is rejected under the same rationale.
Claim 12 recites substantially the same limitations as Claim 3, in the form of a method. The claim is also directed to performing mental processes/mathematical calculations without significantly more, therefore it is rejected under the same rationale.
Claim 13 recites substantially the same limitations as Claim 4, in the form of a method. The claim is also directed to performing mental processes/mathematical calculations without significantly more, therefore it is rejected under the same rationale.
Claim 14 recites substantially the same limitations as Claim 5, in the form of a method. The claim is also directed to performing mental processes/mathematical calculations without significantly more, therefore it is rejected under the same rationale.
Claim 15 recites substantially the same limitations as Claim 6, in the form of a method. The claim is also directed to performing mental processes/mathematical calculations without significantly more, therefore it is rejected under the same rationale.
Claim 16 recites substantially the same limitations as Claim 7, in the form of a method. The claim is also directed to performing mental processes/mathematical calculations without significantly more, therefore it is rejected under the same rationale.
Claim 17 recites substantially the same limitations as Claim 9, in the form of a method. The claim is also directed to performing mental processes/mathematical calculations without significantly more, therefore it is rejected under the same rationale.
Claim 18 recites substantially the same limitations as Claim 8, in the form of a method. The claim is also directed to performing mental processes/mathematical calculations without significantly more, therefore it is rejected under the same rationale.
Claim Rejections - 35 USC § 103
11. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
12. Claims 1-6, 9-15, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Mahmud et al. (hereinafter Mahmud) (“Prediction and Description of Near-Future Activities in Video”), in view of Sainath et al. (hereinafter Sainath) (US PG-PUB 20220238101).
Regarding Claim 1, Mahmud teaches an artificial intelligence (AI) low-latency processing system (Mahmud, Pg. 1, Abstract, “In this work, we propose a system that can infer the labels and the captions of a sequence of future activities. Our proposed network for label prediction of a future activity sequence has three branches where the first branch takes visual features from the objects present in the scene, the second branch takes observed sequential activity features, and the third branch captures the last observed activity features. The predicted labels and the observed scene context are then mapped to meaningful captions using a sequence-to-sequence learning-based method.”, therefore, an artificial intelligence (deep network with three branches) low-latency processing system for captioning is disclosed – this is better depicted by Figure 2 on Pg. 3), the low-latency processing system comprising: a processor; and a memory having instructions stored thereon that (See introduction of Sainath reference below), when executed by the processor, cause the low-latency processing system to:
collect a sequence of frames jointly including event information dispersed among at least one or more frames in the sequence of frames, wherein the event information is associated with multimodal sensor signals comprising at least: video frames, sound data, or a combination thereof (Mahmud, Pg. 2, Figure 1, “Fig. 1. There are k activities in the observed portion of a video with starting times (t1s, t2s, ..., tks) and ending times (t1e, t2e, ..., tke). We want to predict the labels and the captions of (k + 1)th, (k + 2)th, ... activities.”, therefore, a sequence of frames jointly including event information dispersed among at least one or more frames in the sequence of frames, wherein the event information is associated with multimodal sensor signals comprising at least a combination of video frames and sound data is disclosed, as better depicted by Figure 1 on Pg. 2);
execute a timing detector neural network trained to identify an early subsequence of frames from the sequence of frames including at least a portion of the event information (Mahmud, Pgs. 4-5, Section 3.1.2 Network Architecture, “Our proposed architecture for jointly predicting the labels and the starting time of a future activity sequence is shown in Fig. 3. The LSTM is used to solve a sequential input, sequential output problem. We use the activity features extracted from three (chosen empirically) previously observed activities as the LSTM input. Increasing the sequence length does not improve the prediction accuracy significantly (see Section 4.3.3 for de tails). We use a two-layer (chosen empirically) LSTM with 256 memory units in each layer. The input of the two (chosen empirically) fully connected layers in the first branch are the visual features extracted from the objects present in the last observed portion of the scene and there are 256 nodes in each layer. The input of the two (chosen empirically) fully connected layers in the third branch are the activity features extracted from the last observed activity and have 256 nodes in each layer too. Parameter sensitivity analysis for the number of nodes used are provided in the supplementary material.”, thus, a timing detector neural network (LSTM) is executed, which is trained to identify an early subsequence of frames from the sequence of frames including at least a portion of event information – this is better illustrated by Figure 3 which shows that three activity features from a subsequence of frames are identified from the previously observed activity features of the entire sequence of frames);
execute a decoding neural network trained to decode the event information from the early subsequence of frames (Mahmud, Pg. 5, Figure 3, “In the bottom figure, the predicted label (‘cut off ends’ in this example) and the scene context (‘cucumber’ in this example) are then used as input to the encoder LSTM layers and finally the decoder LSTM layers generate the captions. Here, EOS denotes End of Sentence.”, thus, a decoding neural network (decoder) trained to decode the event information from the early subsequence of frames (to create captions) is executed), wherein
the timing detector neural network is jointly trained with the decoding neural network to iteratively identify the early subsequence of frames (Mahmud, Pg. 5, Section 3.1.3 Model Training Approach, “This training method differs from our previous approach [12] in terms of the training procedure. We use the popular open-source deep learning package Keras [86] with TensorFlow [87] in the backend which has ready-to-use implementations of LSTM and fully connected layers. The input sequences for the LSTM are chosen in a sliding window manner with a stride of one for data augmentation. […] In the output layer, we use softmax activation function in the logistic regression nodes for predicting the label of each activity in the sequence and ReLU activation function in the regression node for predicting the starting time of the sequence. The parameters of the entire network are jointly optimized”, therefore, the timing detector neural network is jointly trained with the decoding neural network to iteratively identify the early subsequence of frames, as the entire network is jointly trained/optimized, depicted by Figure 3 on Pg. 5),
the early subsequence of frames comprises continuous subframes from a starting time of a training sequence of frames up to a current time (Mahmud, Pg. 3, Section 1.3 Main Contributions, “Our work focuses on providing a description of what activities may happen in the near-future from a sequence of current observations. We first predict the labels of the future activities, which is then followed by captioning. To the best of our knowledge, almost all of the existing works on video captioning focus on the observed portion of the video, and ours is one of the earliest works for captioning a sequence of near-future unobserved events in videos. It provides a richer description than just predicting the label of the next activity and predicts over a longer duration than works on frame prediction.”, thus, as also depicted by Figure 1 on Pg. 2 and Figure 2 on Pg. 3, the early subsequence of frames comprises continuous subframes from a starting time of a training sequence of frames up to a current time (observed frames until current time where the remaining frames are unobserved)),
the current time corresponds to a time associated with an output of the timing detector neural network indicating availability of sufficient event information for decoding the event information (Mahmud, Pg. 4, Section 3.1.1 Role of Different Context Attributes, “Activities are also characterized by the objects present in the last observed portion of the scene during the time of their occurrence which can be referred to as scene context. For many activities, predicting the future has multiple plausible options. To reduce this specific ambiguity, we take scene context into account along with the sequential information. Thus, combining the information obtained from these two different context attributes (temporal sequence and spatial objects), we infer the sequence of future unobserved activities. […] The inter-activity time between different activities depends on their labels. For example, we know from experience that ‘peel’ or ‘cut slices’ takes more time than ‘wash objects’. Thus, by observing the previous activity features we can infer the inter-activity time (difference between the starting time of the observed activity and the future activity) which can be referred to as inter-activity time context”, therefore, the current time corresponds to a time associated with an output of the timing detector neural network indicating availability of sufficient event information for decoding the event information (inter-activity time context and scene context based on observed frames – see also Pg. 6 Section 3.2.3 Model Training Approach which mentions that the decoder considers scene context for decoding event information)), and
the early subsequence of frames is less than a total number of frames in the training sequence of frames (Mahmud, Pgs. 5-6, Section 3.1.3 Model Training Approach, “Here, X = {x(1),...,x(n)} is the set of input feature vectors (activity features of the last three observed activities and features of the objects present in the last observed portion of the scene) in the training dataset, Y = {y(1),...,y(n)} is the corresponding set of labels for those input features, and j = {1,...,c} is the set of class labels. 1(.) is an indicator function. x(i) is the sequential activity features extracted from the previous three activities.”, thus, the early subsequence of frames (three observed activities and features of the video frames) is less than a total number of frames in the training sequence of frames (entire set of activities and features from the whole video)),
output a token sequence corresponding to the event information decoded from the early subsequence of frames (Mahmud, Pg. 5, Figure 3, “In the bottom figure, the predicted label (‘cut off ends’ in this example) and the scene context (‘cucumber’ in this example) are then used as input to the encoder LSTM layers and finally the decoder LSTM layers generate the captions. Here, EOS denotes End of Sentence.”, thus, as shown in the bottom half of Figure 3, a token sequence is outputted (“the person cut the ends off the cucumber EOS”) corresponding to the event information decoded from the early subsequence of frames (captions));
generate a control action based on the token sequence (See introduction of Sainath reference below); and
control a surveillance system based on the control action (See introduction of Sainath reference below).
Although Mahmud teaches an AI low-latency processing system as described above, Mahmud does not explicitly disclose the low-latency processing system comprising: a processor; and a memory having instructions stored thereon […]
However, Sainath teaches an AI low-latency processing system as described above, Mahmud does not explicitly disclose the low-latency processing system (Sainath, Figure 10, label 1010 which depicts an example computing device) comprising: a processor (Sainath, Figure 10, label 1014 which depicts a processor); and a memory having instructions stored (Sainath, Figure 10, label 1025 which depicts a memory subsystem storing instructions to be executed by the processor) thereon […]
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the artificial intelligence (AI) low-latency processing system, as disclosed by Mahmud to include the low-latency processing system comprising: a processor; and a memory having instructions stored thereon […], as disclosed by Sainath. One of ordinary skill in the art would have been motivated to make this modification to enable the use of a processing system comprising a processor and memory which may improve processing/computing efficiency and correspondingly allow for effective communication and storage of information between multiple devices within the system (Sainath, Par. [0103], “Computing device 1010 typically includes at least one processor 1014 which communicates with a number of peripheral devices via bus subsystem 1012. These peripheral devices may include a storage subsystem 1024, including, for example, a memory subsystem 1025 and a file storage subsystem 1026, user interface output devices 1020, user interface input devices 1022, and a network interface subsystem 1016. The input and output devices allow user interaction with computing device 1010. Network interface subsystem 1016 provides an interface to outside networks and is coupled to corresponding interface devices in other computing devices.”)
Mahmud does not explicitly disclose generate a control action based on the token sequence; and control a surveillance system based on the control action.
However, Sainath teaches generate a control action based on the token sequence (Sainath, Par. [0089], “At block 860, the system performs one or more actions based on the text representation of the utterance. For example, the system can render output based on the text representation on a screen of the client device for the speaker. In some implementations, the system can render output based on one or more of the candidate text representations, such as rendering output based on the candidate text representation with the highest probability, while the human is speaking the utterance.”, thus, as also shown by Figure 8, a control action is generated based on the token sequence (text representation of the utterance)); and control a surveillance system based on the control action (Sainath, Par. [0089], “Additionally or alternatively, the system can control one or more client devices based on the text representation of the utterance. For example, the system can turn on four lights in the living room in response to the utterance of “turn on the living room lights”. In a variety of implementations, the system can perform additional and/or alternative actions based on the final text representation of the utterance.”, therefore, a surveillance system (home surveillance) may be controlled based on the control action).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the AI low-latency processing system, as disclosed by Mahmud in view of Sainath to include generate a control action based on the token sequence; and control a surveillance system based on the control action, as disclosed by Sainath. One of ordinary skill in the art would have been motivated to make this modification to enable accurate and efficient system control based on a text representation/token sequence, hence abstracting complexity whilst improving system function (Sainath, Par. [0089], “At block 860, the system performs one or more actions based on the text representation of the utterance. For example, the system can render output based on the text representation on a screen of the client device for the speaker. In some implementations, the system can render output based on one or more of the candidate text representations, such as rendering output based on the candidate text representation with the highest probability, while the human is speaking the utterance. In some such implementations, the system can render output based on the text representation by revising the output rendered while the human was speaking the utterance. Additionally or alternatively, the system can render one or more responses to the text representation of the utterance. For example, the system can render the response of “it is going to rain later today” in response to the human speaking the utterance of “what is the weather today”. Additionally or alternatively, the system can control one or more client devices based on the text representation of the utterance. For example, the system can turn on four lights in the living room in response to the utterance of “turn on the living room lights”. In a variety of implementations, the system can perform additional and/or alternative actions based on the final text representation of the utterance.”).
Regarding Claim 2, Mahmud in view of Sainath teaches the Al low-latency processing system of claim 1, wherein the timing detector neural network is jointly trained with the decoding neural network on features of different subsequences of the sequence of frames to minimize a multi-task loss function including a time detection loss and an information generation loss (Mahmud, Pgs. 5-6, Section 3.1.3 Model Training Approach, “We take the summation of the following two losses to compute the final loss. One is the cross-entropy loss function which is defined as follows: […] Here, X = {x(1),...,x(n)} is the set of input feature vectors (activity features of the last three observed activities and features of the objects present in the last observed portion of the scene) in the training dataset, Y = {y(1),...,y(n)} is the corresponding set of labels for those input features, and j = {1,...,c} is the set of class labels. 1(.) is an indicator function. x(i) is the sequential activity features extracted from the previous three activities. Another is the mean squared loss between the ground truth inter-activity time and the predicted inter-activity time which is defined as follows: […] The parameters of the network are jointly optimized by minimizing both of these losses.”, thus, the timing detector neural network is jointly trained with the decoding neural network on features of different subsequences of the sequence of frames to minimize a multi-task loss function including a time detection loss (mean squared loss between inter-activity time and the predicted inter-activity time) and information generation loss (cross-entropy loss)).
Regarding Claim 3, Mahmud in view of Sainath teaches the Al low-latency processing system of claim 2, wherein the multi-task loss function includes three losses defining:
an accuracy of decoded event information (Mahmud, Pg. 5, Section 3.1.3 Model Training Approach, “We take the summation of the following two losses to compute the final loss. One is the cross-entropy loss function which is defined as follows: […] Here, X = {x(1),...,x(n)} is the set of input feature vectors (activity features of the last three observed activities and features of the objects present in the last observed portion of the scene) in the training dataset, Y = {y(1),...,y(n)} is the corresponding set of labels for those input features, and j = {1,...,c} is the set of class labels. 1(.) is an indicator function. x(i) is the sequential activity features extracted from the previous three activities”, therefore, loss defining an accuracy of decoded event information is disclosed);
a difference between the event information decoded from the early subsequence of frames and the event information decoded from a full sequence of frames (Mahmud, Pgs. 12-13, Section 4.4.3 Analysis of Observation Horizon, “The corresponding qualitative analysis is given in Table 14. Although the number of wrong words in each sentence is similar, there is reasonable difference in the values of the evaluation metrics in Table 13. This is be cause the label prediction performance changes as we change the observed sequence length and this in turn changes the number of such erroneously generated captions.”, thus, a loss defining a difference between the event information decoded from the early subsequence and the event information decoded from a full sequence of frames is disclosed – this is also shown by Table 14 on Pg. 13); and
an accuracy of prediction of the timing detector neural network (Mahmud, Pg. 6, Section 3.1.3 Model Training Approach, “Another is the mean squared loss between the ground truth inter-activity time and the predicted inter-activity time which is defined as follows: […] Here, P = {p(1),...,p(n)} is the set of input feature vectors (activity feature of the last observed activity) in the training dataset, and Q = {q(1),...,q(n)} is the corresponding set of inter-activity times for those input features. ˆq(i) represents the predicted inter-activity time given input p(i) where the ground truth inter activity time is q(i). ˆq(i) is a function of the input features P. The outputs of the training are the labels of the three future activities and the starting time of that activity sequence.”, thus, a loss defining an accuracy of prediction of the timing detector neural network is disclosed).
Regarding Claim 4, Mahmud in view of Sainath teaches the Al low-latency processing system of claim 1, wherein the processor is configured to
execute a feature extractor to extract features from each frame in the sequence of frames (Mahmud, Pg. 5, Section 3.1.2 Network Architecture, “The input of the two (chosen empirically) fully connected layers in the first branch are the visual features extracted from the objects present in the last observed portion of the scene and there are 256 nodes in each layer. The input of the two (chosen empirically) fully connected layers in the third branch are the activity features extracted from the last observed activity and have 256 nodes in each layer too. Parameter sensitivity analysis for the number of nodes used are provided in the supplementary material.”, therefore, a feature extractor (pictured by Figure 3 on Pg. 5) to extract features from each frame in the sequence of frame is executed);
execute a feature encoder to encode the extracted features of each frame to produce a sequence of encoded features (Mahmud, Pg. 5, Figure 3, ““In the top figure, the first two fully connected layers (yellow) incorporate the scene context (extracted from objects in the last frame) which use object features as input. The two LSTM layers (green) are used to incorporate the sequential activity context (‘take out’, ‘wash’, ‘put on’ in this example) which use motion-based features as inputs. The last two fully connected layers (peach) are used to incorporate inter-activity time context which use the last observed activity features (motion-based) (‘put on’ in this example) as input. There is a fully connected layer (blue) where all these layers are merged together.”, thus, the encoder may be executed to encode the extracted features of each frame to produce a sequence of encoded features – also shown by Figure 3);
submit the sequence of encoded features to the timing detector neural network to identify a subsequence of encoded features representing the early subsequence of frames (Mahmud, Pg. 5, Figure 3, “The output layer (gray) performs the final prediction, where for each element of the future activity sequence, the first few nodes (green) are used as the logistic regression nodes for label prediction. The last node (blue) of the output layer is used as the regression node for starting time prediction. All of the layers have 256 nodes.”, thus, the sequence of encoded features is submitted to the timing detector neural network to identify a subsequence of encoded features representing the early subsequence of frames – also shown by Figure 3); and
submit the subsequence of encoded features to the decoding neural network to decode the event information (Mahmud, Pg. 5, Figure 3, “In the bottom figure, the predicted label (‘cut off ends’ in this example) and the scene context (‘cucumber’ in this example) are then used as input to the encoder LSTM layers and finally the decoder LSTM layers generate the captions. Here, EOS denotes End of Sentence.”, thus, a subsequence of encoded features are submitted to the decoding neural network to decode the event information).
Regarding Claim 5, Mahmud in view of Sainath teaches the Al low-latency processing system of claim 1, wherein the processor triggers the execution of modules of the Al low-latency processing system upon receiving a new input frame appended to the sequence of frames (Mahmud, Pg. 4, Section 3.1.2. Network Architecture, “Our proposed architecture for jointly predicting the labels and the starting time of a future activity sequence is shown in Fig. 3. The LSTM is used to solve a sequential input, sequential output problem. We use the activity features extracted from three (chosen empirically) previously observed activities as the LSTM input.”, thus, the execution of modules of the system may be triggered upon receiving a new input frame appended to the sequence – this is shown by Figure 3 on Pg. 5).
Regarding Claim 6, Mahmud in view of Sainath teaches the Al low-latency processing system of claim 1, wherein the event information is a caption for an audio scene, a video scene, or an audio-video scene (Mahmud, Pg. 2, Section 1. Introduction, “Ours is one of the earliest works where we look into the problem of providing captions for a sequence of near-future unobserved events in videos.”, therefore, the event information may comprise a caption for a video scene/audio-video scene).
Regarding Claim 9, Mahmud in view of Sainath teaches the Al low-latency processing system of claim 1, wherein the sequence of frames includes multi-modal information coming from different sensors of different modalities (Mahmud, Pg. 1, Section 1. Introduction, “Generating description of visual content is an active research area in both computer vision and natural language processing community. Since vision and language are two of the richest interaction modalities available to humans, it is crucial to understand the relationship between them. Language is the most natural way to make information from any semantic representation meaningful.”, thus, the sequence of frames includes multi-modal information coming from different sensors of different modalities – this may include video data, sound data, or a combination of both)
Regarding Claim 10, Mahmud in view of Sainath teaches a computer-implemented method for an artificial intelligence (Al) low-latency processing system (Mahmud, Pg. 1, Abstract, “In this work, we propose a system that can infer the labels and the captions of a sequence of future activities. Our proposed network for label prediction of a future activity sequence has three branches where the first branch takes visual features from the objects present in the scene, the second branch takes observed sequential activity features, and the third branch captures the last observed activity features. The predicted labels and the observed scene context are then mapped to meaningful captions using a sequence-to-sequence learning-based method.”, therefore, a computer-implemented method for an artificial intelligence (deep network with three branches) low-latency processing system for captioning is disclosed – this is better depicted by Figure 2 on Pg. 3), including a processor (Sainath, Figure 10, label 1014 which depicts a processor) and a memory storing instructions of the computer-implemented method performing steps using the processor (Sainath, Figure 10, label 1025 which depicts a memory subsystem storing instructions to be executed by the processor), the steps comprising: […]
The rest of the claim language in Claim 10 recites substantially the same limitations as Claim 1, in the form of a method, therefore it is rejected under the same rationale.
The reasons of obviousness have been noted in the rejection of Claim 1 above and applicable herein.
Claim 11 recites substantially the same limitations as Claim 2 in the form of a method, therefore it is rejected under the same rationale.
Claim 12 recites substantially the same limitations as Claim 3 in the form of a method, therefore it is rejected under the same rationale.
Claim 13 recites substantially the same limitations as Claim 4 in the form of a method, therefore it is rejected under the same rationale.
Claim 14 recites substantially the same limitations as Claim 5 in the form of a method, therefore it is rejected under the same rationale.
Claim 15 recites substantially the same limitations as Claim 6 in the form of a method, therefore it is rejected under the same rationale.
Claim 17 recites substantially the same limitations as Claim 9 in the form of a method, therefore it is rejected under the same rationale.
13. Claims 7-8, 16, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Mahmud et al. (hereinafter Mahmud) (“Prediction and Description of Near-Future Activities in Video”), in view of Sainath et al. (hereinafter Sainath) (US PG-PUB 20220238101), further in view of Kim et al. (hereinafter Kim) (US PG-PUB 20210012222).
Regarding Claim 7, Mahmud in view of Sainath teaches the Al low-latency processing system of claim 1.
Although Mahmud Pg. 4 discloses that their integrated approach answers important questions regarding the unobserved portion of a video observed up to a particular time, Mahmud in view of Sainath does not explicitly disclose wherein the information is an answer to a question about the sequence of frames.
However, Kim teaches wherein the information is an answer to a question about the sequence of frames (Kim, Abstract, “In implementations of answering questions during video playback, a video system can receive a question related to a video at a timepoint of the video during playback of the video, and determine audio sentences of the video that occur within a segment of the video that includes the timepoint. The video system can generate a classification vector from words of the question and the audio sentences, and determine an answer to the question utilizing the classification vector. The video system can obtain answer candidates, and the answer to the question can be selected as one of the answer candidates based on matching the classification vector to one of the answer vectors.”, thus, the information may comprise an answer to a question about the sequence of frames in the video).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the AI low-latency processing system of claim 1, as disclosed by Mahmud in view of Sainath to include wherein the information is an answer to a question about the sequence of frames, as disclosed by Kim. One of ordinary skill in the art would have been motivated to make this modification to enable accurate and reliable question answering during video playback and processing for a plurality of different domains (Kim, Par. [0006], “Accordingly, the video system can generate an answer to a question based on context of the video when the question is asked, and the answer is accurate and reliable since it is grounded to the domain knowledge base. Hence, the video system is not limited to answering questions having an answer that may be determined from the question itself, but can also answer questions with an answer based on the context of the video or domain knowledge about the video for any video pertaining to a domain.”).
Regarding Claim 8, Mahmud in view of Sainath in view of Kim teaches the Al low-latency processing system of claim 7, wherein the processor is configured to:
execute a text encoder neural network to encode the question (Kim, Par. [0023], “For instance, the video system may include a question encoder that receives the question (e.g., text of the question in any suitable format, such as concatenated word vectors), and generates a question representation, such as a question feature vector based on the words of the question”, therefore, a text encoder neural network (question encoder) may be executed to encode the question);
submit the encoded question to the timing detector neural network (Mahmud, Pg. 2, Section 1.2 Overview of the Approach, “In this paper, we present an integrated approach to answer two important questions regarding the unobserved portion of a video observed up to a particular time: what activities will hap pen next, and what captions describe them best. We predict the labels of a sequence of future unobserved activities in both coarse (VIRAT Ground Dataset [55]) and fine grained activity datasets (MPII-Cooking Dataset [56], MPII-Cooking 2 Dataset [57] and EPIC-KITCHENS Dataset [58]).” & Pg. 6, Section 3.2.2 Network Architecture, “The input to the encoder LSTM is text e.g., cut apart cucumber, take out egg fridge, cut off ends carrot, etc. obtained by concatenating the corresponding predicted labels and scene context.”, thus, the encoded question may be submitted to the timing detector neural network, as also seen by Figure 3 on Pg. 5); and
submit the question or the encoded question to the decoding neural network (Mahmud, Pg. 6, Section 3.3 Test Case Scenario, “Based on the learned sequence-to-sequence relationship in the training phase, the network predicts the labels of the next three unobserved activities. Using this predicted sequence of la bels and observed scene context, the most likely captions for the future activity sequence are generated by the encoder-decoder LSTM pair.”, therefore, the question/encoded question may be submitted to the decoding neural network);
The reasons of obviousness have been noted in the rejection of Claim 7 above and applicable herein.
Claim 16 recites substantially the same limitations as Claim 7 in the form of a method, therefore it is rejected under the same rationale.
Claim 18 recites substantially the same limitations as Claim 8 in the form of a method, therefore it is rejected under the same rationale.
Conclusion
14. Any inquiry concerning this communication or earlier communications from the examiner should be directed to Devika S Maharaj whose telephone number is (571)272-0829. The examiner can normally be reached Monday - Thursday 8:30am - 5:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached at (571)270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DEVIKA S MAHARAJ/Examiner, Art Unit 2123
Prosecution Insights