Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
The United States Patent & Trademark Office appreciates the application that is submitted by the inventor/assignee. The United States Patent & Trademark Office reviewed the following application and has made the following comments below.
Priority
This application claims benefit of foreign priority under 35 U.S.C. 119(a)-(d) of GB2401572.9, filed in the United Kingdom on 02/06/2024.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 02/06/2025 and 01/23/2026 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Specification
The disclosure is objected to because of the following informalities:
In paragraph [0074], on lines 7-8, “predicate1 and predicate2 may correspond to the same prediction” is suggested to read “predicate1 and predicate2 may correspond to the same predicate”.
In paragraph [0100], “or create new ones as new be” is suggested to read “or create new ones as need be”.
In addition, the disclosure is objected to because it contains an embedded hyperlink and/or other form of browser-executable code. Applicant is required to delete the embedded hyperlink and/or other form of browser-executable code; references to websites should be limited to the top-level domain name without any prefix such as http:// or other browser-executable code. See MPEP § 608.01.
The following embedded hyperlinks should be deleted:
In paragraph [0010] on line 25, (https://arxiv.org/abs/2308.15366)
In paragraph [0070], on line 4, (https://www.w3.org)
In paragraph [0078], on line 3, (https://arxiv.org/abs/2301.12597)
In paragraph [0079], on line 3, (https://arxiv.org/abs/2307.09288)
In paragraph [0100], on line 2, (https://arxiv.org/abs/2303.05499)
Appropriate correction is required.
Claim Objections
Claims 4, 16, and 18 are objected to because of the following informalities:
In Claim 4, on line 2, “recognise” is suggested to read “recognize”.
In Claim 16, on line 3, “uses at least one machine learning model, MLM, …” is suggested to read “uses at least one machine learning model (MLM), …”
In Claim 18, on line 2, “at least one machine learning model, MLM, or a rules engine, …” is suggested to read “at least one machine learning model (MLM), or a rules engine, …”. The Examiner suggests putting “MLM” in parenthesis after “machine learning model” to clarify that MLM is the acronym for “machine learning model” and not a different type of model.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(B) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention. The Examiner strongly suggested that appropriate corrections be made to clarify the claim scope.
With respect to Claim 1, the claim recites the following, each of which renders the claim indefinite:
“a unique identifier” on line 9 (unclear antecedent basis); it is unclear as to whether “a unique identifier” recited in line 9 of claim 1 is the same or different from “unique identifiers” recited in line 3 of claim 1.
“user-relevant content” on line 7 (relative terminology) is a relative term which renders the claim indefinite. The term “user-relevant content” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
Claims 2-18 are also rejected by virtue of their dependence on Claim 1.
With respect to Claim 7, the claim recites the following, each of which renders the claim indefinite:
“the video frames” on line 4 (unclear antecedent basis); there is insufficient antecedent basis for “video frames” as it has not been previously mentioned previously.
With respect to Claim 10, the claim recites the following, each of which renders the claim indefinite:
“user-relevant content” on line 2 (relative terminology) is a relative term which renders the claim indefinite. The term “user-relevant content” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
With respect to Claim 19, the claim recites the following, each of which renders the claim indefinite:
“a unique identifier” on line 10 (unclear antecedent basis); it is unclear as to whether “a unique identifier” recited in line 10 of claim 19 is the same or different from “unique identifiers” recited in line 4 of claim 19.
“user-relevant content” on line 8 (relative terminology) is a relative term which renders the claim indefinite. The term “user-relevant content” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
With respect to Claim 20, the claim recites the following, each of which renders the claim indefinite:
“a unique identifier” on line 10 (unclear antecedent basis); it is unclear as to whether “a unique identifier” recited in line 10 of claim 20 is the same or different from “unique identifiers” recited in line 4 of claim 20.
“said user-relevant content” on line 8 (unclear antecedent basis); there is no previous mention of “user-relevant content” in Claim 20.
“user-relevant content” on line 8 (relative terminology) is a relative term which renders the claim indefinite. The term “user-relevant content” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103(a) are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 14-15, 19, and 20 are rejected under 35 U.S.C. 103(a) as being unpatentable over Nguyen (U.S. Patent Pub. No. 20200302177 A1, hereafter referred to as Nguyen) in view of Jain et al. (U.S. Patent Pub No. 20250077576 A1, hereafter referred to as Jain).
Regarding Claim 1, Nguyen teaches the following:
A method of video surveillance (Paragraph [0049], Fig. 8, Nguyen teaches a method of intelligent video analysis.)
PNG
media_image1.png
721
472
media_image1.png
Greyscale
comprising:
- capturing video data from a plurality of video surveillance cameras (Paragraph [0042], Fig. 2, Nguyen teaches receiving video feeds from multiple sources. Some of the sources can be stationary, e.g., the source (201) can be a stationary camera. Some of the sources can be moving, e.g., the source (202) can be camera or imaging sensor positioned on a manned or unmanned aerial vehicle.)
PNG
media_image2.png
556
917
media_image2.png
Greyscale
- acquiring metadata (Paragraph [0038], Nguyen teaches storing the video feed, classified objects, and classified activities, natural language text, and semantic network in the knowledge base.) generated by performing captioning of the video data (Paragraphs [0036], [0047], Fig. 6, Nguyen teaches the natural language generator (1022) is configured to generate natural language text that describes the video feed from the classified object and activities. For example, the natural language generator can generate natural language text such as “man riding bicycle” to describe the video feed. In another example, the natural language text can be “Car, appeared in Laker Village Dr” as shown Fig. 6., see reference character (602).),
PNG
media_image3.png
496
894
media_image3.png
Greyscale
wherein the metadata comprises semantic data (Paragraphs [0037-38], [0046], Nguyen teaches semantic network is stored in the knowledge base (103).) to represent content in the video data (Paragraphs [0037], [0046], Fig. 5, Nguyen teaches in order to display semantic relationships between the classified objects to a user, the semantic network generator (1023) is configured to produce a semantic network including a graph. Graph (500) may have vertices that represent the classified objects, and edges that connect the vertices and represent semantic relationships between the classified objects. For example, the vertex (501) can represent a person and a vertex (503) can represent a building. The edge (502) that connects the vertices (501) and (503) can represent a semantic relationship that the person enters the building.)
PNG
media_image4.png
398
360
media_image4.png
Greyscale
in combination with the unique identifiers (Paragraphs [0012], [0043], [0046], Fig. 5, Nguyen teaches assigning respective unique identifiers to the classified objects. The semantic network may include a graph (500) with vertices that represent the classified objects, and edges that connect the vertices and represent semantic relationships between the classified objects. For example, the vertex (501) can represent a person and vertex (503) can represent a building. The edge (502) that connects vertices (501) and (503) can represent a semantic relationship that the person enters the building. The Examiner interprets the semantic data is “in combination with” the unique identifiers (unique identifiers assigned to classified objects) since the semantic network represents the semantic relationship(s) between the classified objects. For example, under BRI, “person” (501) and “building” (503) are being interpreted as unique identifiers. The semantic graph represents the semantic relationship between the unique identifiers (person and building).); and
[AltContent: arrow]- identifying user-relevant content (Paragraph [0042], Fig. 2, Nguyen teaches the GUI can enable queries of the knowledge dases y users, such as user (203). The GUI can also enable presentation of selections of the video feed, classified objects/activities, natural language text, and semantic network.) using either one of a user-defined semantic query (Paragraph [0048], Fig. 7, Nguyen teaches the GUI (1041) can enable queries of the knowledge base by the user. The queries can be based on a keyword entered by the user. The queries can be based on objects or events in a given time window and/or in a geographical area that is specified by the user. For example, as indicated by arrow (701), the user can specify a car and choose “Find Similar” to search visually similar cars as the specified car in the video feed. Under Broadest Reasonable Interpretation (BRI), the Examiner interprets “either one of a” and “and” means that only one of either “a user-defined semantic query” OR “a user-selected Video Anomaly Detection (VAD) model” is required to meet the claim limitation.)
PNG
media_image5.png
314
792
media_image5.png
Greyscale
and a (Paragraphs [0035], [0013], Nguyen teaches the detection and recognition module is configured to detect and classify the moving objects using motion compensation, background subtraction, and convolutional neural networks. The Examiner interprets the VAD model to be a machine learning model, in light of paragraph [0030] of applicant’s specification, which states, “the said user-selected VAD model comprises at least one machine learning model, MLM, or a rules engine, configured to perform VAD, on the acquired video data.”), in combination with either one of a unique identifier (Paragraphs [0012], [0041], [0043], Fig. 3, Nguyen teaches assigning respective unique identifiers to the classified objects, and the presentation of selections of the video feed in the GUI includes identifying the classified objects on the video feed and including respective unique identifiers. For example, as shown in an image (300) (e.g., a frame of the video feed), the detection and recognition module (1021) can detect and classify objects as “car” or “person” or “handbag” in the video feed. Unique identifiers are assigned to the classified objects. Under Broadest Reasonable Interpretation (BRI), the Examiner interprets “either one of a” and “and” means that only one of either “a unique identifier” OR “a subset of the unique identifiers” is required to meet the claim limitation.)
PNG
media_image6.png
553
685
media_image6.png
Greyscale
Nguyen does not explicitly disclose the following:
- identified by unique identifiers;
-
- and
Jain is in the same field of art of using natural language processing to manage video data, such as surveillance/security video data. Further, Jain teaches:
[AltContent: arrow]- capturing video data from a plurality of video surveillance cameras (Paragraphs [0028-29], Fig. 2, Jain teaches video surveillance data is captured at a customer site (e.g., surveillance location). As shown in Fig. 2, a customer site (e.g., surveillance location) can include one or more surveillance cameras. The cameras (200) may be deployed to various locations around a customer site. Each camera may stream video data to the natural language video monitoring system (100).)
PNG
media_image7.png
286
758
media_image7.png
Greyscale
identified by unique identifiers (Paragraphs [0055], [0025], Jain teaches video capture device identifiers or “camera ID”.);
- acquiring metadata (Paragraph [0025], Jain teaches storing associated metadata.) in combination with the unique identifiers (Paragraphs [0025-26], Jain teaches the vector database may store descriptor embeddings produced by a video indexing network and associated metadata, such as, video identifier, camera identifier, time of day, date, etc. The video identifier metadata associated with each similar embedding may be used to look up the corresponding video in video data.); and
- identifying user-relevant content (Paragraph [0026], Jain teaches the user interface displays relevant video results to the user. For example, matching videos can be ranked and presented to the user in the order of ranking. Additional information and/or a summary of the search results may be presented, such as a number of clips, or specifics relevant to the query.) using either one of a user-defined semantic query (Paragraph [0022], Jain teaches the query may be received locally (e.g., via a user interface). The query may be “find a person wearing a yellow hat, visible by camera 1 or camera 2, during the day on April 3rd.” In some embodiments, a user interface is provided allowing a user to input text input using a web dashboard, etc. Alternatively, the user can be presented with a predefined set of text queries.) and a (Paragraph [0029], Jain teaches the natural language video monitoring system (100) may include a neural network manager (106) that provides an execution environment for one or more machine learning models. The Examiner interprets the VAD model to be a machine learning model, in light of paragraph [0030] of applicant’s specification, which states, “the said user-selected VAD model comprises at least one machine learning model, MLM, or a rules engine, configured to perform VAD, on the acquired video data.” In addition, under Broadest Reasonable Interpretation (BRI), the Examiner interprets “either one of a” and “and” to mean only one of either “a user-defined semantic query” OR “a user-selected Video Anomaly Detection (VAD) model” is required to meet the claim limitation.), in combination with either one of a unique identifier (Paragraph [0022], Jain teaches the input query (102) may define an object, person, etc. being searched for, a camera location, camera identifier, date, time, or other search parameters.) and a subset of the unique identifiers (Paragraph [0022], Jain teaches for example, the query may be “find a person wearing a yellow hat, visible by camera 1 or camera 2, during the day on April 3rd.” The Examiner interprets camera 1 or camera 2 to be a “subset” of unique identifiers. In addition, under Broadest Reasonable Interpretation (BRI), the Examiner interprets “either one of a” and “and” to mean only one of either “a unique identifier” OR “a subset of unique identifiers” is required to meet the claim limitation.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Nguyen by identifying/assigning the surveillance cameras unique identifiers or “camera ID’s” that is taught by Jain, to make the invention that distinguishes surveillance cameras from other cameras in a network or system and allows the user to define a specific camera identifier within their input query such as “find a person visible by camera 1 or camera 2”, for example (Jain, Paragraph [0022]); thus, one of ordinary skilled in the art would be motivated to combine the references since camera identifiers or camera ID’s assigned to the surveillance cameras so the cameras can be easily recognized, managed, and secured within a network or system so that the user is able to ask a specific question regarding a particular camera or area captured by a specific camera (Jain, Paragraph [0022]). This may mitigate the need for manually reviewing hours of surveillance recordings across tens or hundreds of devices, which may save time, money, and resources (Jain, Paragraph [0016]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 2, Nguyen in view of Jain discloses the method according to claim 1, wherein the semantic data uses subject- predicate-object triples to represent content in the video data (Paragraph [0046], Fig. 5, Nguyen teaches a semantic network may include a graph (500) with vertices that represent the classified objects, and edges that connect the vertices and represent semantic relationships between the classified objects. For example, the vertex (501) can represent a person and vertex (503) can represent a building. The edge (502) that connects vertices (501) and (503) can represent a semantic relationship that the person enters the building. The Examiner interprets “the person enters the building” is a subject-predicate-object triple. “The person” is the subject. “Enters the” is the predicate, and “the building” is the object.).
PNG
media_image8.png
407
354
media_image8.png
Greyscale
In regards to Claim 14, Nguyen in view of Jain discloses the method according to claim 2, wherein capturing the video data (Paragraph [0009], Nguyen teaches processing the video feed in real-time as the video is received.) and acquiring the metadata is performed in a live fashion (Paragraph [0009], Nguyen teaches processing the video feed in real-time as the video feed is received, including: performing object detection and recognition on the video feed to detect and classify objects therein, performing activity recognition to detect and classify activities of at least some of the objects, and outputting classified objects and classified activities in the video feed; generating natural language text that describes the video feed from the classified objects and activities; producing a semantic network including a graph with vertices that represent the classified objects, and edges that connect the vertices and represent semantic relationships between the classified objects, at least some of the semantic relationships corresponding to respective ones of the classified activities; and storing the video feed, classified objects and classified activities, natural language text, and semantic network in a knowledge base.), and identifying the said user- relevant content (Paragraph [0009], Nguyen teaches generating a graphical user interface (GUI) configured to enable queries of the knowledge base, and presentation of selections of the video feed, classified objects and classified activities, natural language text, and semantic network.) is performed in a delayed fashion (Paragraphs [0006-7], Nguyen teaches monitoring and support offline search and retrieval. Highly semantic content information from video feeds can be stored and retrieved efficiently offline.) upon receipt of the said user- defined semantic query (Paragraphs [0007], [0039], Nguyen teaches quick search and retrieval of past events using various types of semantic-based video queries (e.g., by keywords, by geo-spatial areas, or by visual similarity). The GUI is configured to enable queries of the knowledge base (103).) in combination with either one of the said unique identifier and the said subset of the unique identifiers (Paragraph [0039], Nguyen teaches the GUI configured to enable queries of the knowledge base based on similarity between a user-specified object and one or more classified objects in the video feed. For example, the user-specified object can be a white car, and the GUI can enable queries of the knowledge base based on similarity between the white car and other cars in the video feed. The Examiner interprets the user-specified object, such as “white car” to be a unique identifier since it identifies a specific object in the video feed. Under BRI, the Examiner interprets “either one of” and “and” to mean only one of either the unique identifier or the subset of unique identifiers is required to meet the claim limitation.).
In regards to Claim 15, Nguyen in view of Jain discloses the method according to claim 2, further comprising generating a graph, named first graph (Paragraphs [0005], [0009], [0037], Fig. 5, Nguyen teaches graph-based content representation. Specifically, producing a semantic network including a graph with vertices that represent the classified objects, and edges that connect the vertices and represent semantic relationships between the classified objects.), to represent the subject-predicate-object triples (Paragraph [0046], Fig. 5, Nguyen teaches a graph (500) with vertices that represent the classified objects, and edges that connect the vertices and represent semantic relationships between the classified objects. Some of the semantic relationships correspond to some classified activities of the classified objects. For example, in Fig. 5, the vertex (501) can represent a person and vertex (503) can represent a building. The edge (502) that connects the vertices (501) and (503) can represent a semantic relationship that the person enters the building. The Examiner interprets “the person enters the building” is a subject-predicate-object triple. “The person” is the subject. “Enters the” is the predicate, and “the building” is the object.) generated by performing captioning of the video data (Paragraph [0050], Fig. 8, Nguyen teaches at sub-block (8022) generating natural language text that describes the video feed from the classified objects and activities. At sub-block (8023), the method includes producing a semantic network including a graph with vertices that represent the classified objects and edges that connect the vertices and represent semantic relationships between the classified objects.),
PNG
media_image9.png
652
451
media_image9.png
Greyscale
wherein subjects and objects are represented as nodes and predicates as edges (Paragraph [0037], Fig. 5, Nguyen teaches the graph includes vertices that represent the classified objects, and edges that connect the vertices and represent semantic relationships between the classified objects.).
PNG
media_image10.png
497
466
media_image10.png
Greyscale
In regards to Claim 19, Nguyen discloses:
A non-transitory computer readable storage medium storing a program for causing a computer to execute a method (Paragraph [0017], Nguyen teaches the computer-readable storage medium is non-transitory and has computer-readable program code stored therein that in response to execution by a processor, causes an apparatus to perform the method.) of video surveillance (Paragraph [0049], Nguyen teaches a method of intelligent video analysis.) comprising:
- capturing video data from a plurality of video surveillance cameras (Paragraph [0042], Fig. 2, Nguyen teaches receiving video feeds from multiple sources. Some of the sources can be stationary, e.g., the source (201) can be a stationary camera. Some of the sources can be moving, e.g., the source (202) can be camera or imaging sensor positioned on a manned or unmanned aerial vehicle.)
- acquiring metadata (Paragraph [0038], Nguyen teaches storing the video feed, classified objects, and classified activities, natural language text, and semantic network in the knowledge base.) generated by performing captioning of the video data (Paragraph [0036], Nguyen teaches the natural language generator (1022) is configured to generate natural language text that describes the video feed from the classified object and activities. For example, the natural language generator can generate natural language text such as “man riding bicycle” to describe the video feed.), wherein the metadata comprises semantic data (Paragraphs [0037-38], [0046], Nguyen teaches semantic network is stored in the knowledge base (103).) to represent content in the video data (Paragraphs [0037], [0046], Fig. 5, Nguyen teaches in order to display semantic relationships between the classified objects to a user, the semantic network generator (1023) is configured to produce a semantic network including a graph. Graph (500) may have vertices that represent the classified objects, and edges that connect the vertices and represent semantic relationships between the classified objects. For example, the vertex (501) can represent a person and a vertex (503) can represent a building. The edge (502) that connects the vertices (501) and (503) can represent a semantic relationship that the person enters the building.) in combination with the unique identifiers (Paragraphs [0012], [0043], [0046], Fig. 5, Nguyen teaches assigning respective unique identifiers to the classified objects. The semantic network may include a graph (500) with vertices that represent the classified objects, and edges that connect the vertices and represent semantic relationships between the classified objects. For example, the vertex (501) can represent a person and vertex (503) can represent a building. The edge (502) that connects vertices (501) and (503) can represent a semantic relationship that the person enters the building. The Examiner interprets the semantic data is “in combination with” the unique identifiers (unique identifiers assigned to classified objects) since the semantic network represents the semantic relationship(s) between the classified objects. For example, under BRI, “person” (501) and “building” (503) are being interpreted as unique identifiers. The semantic graph represents the semantic relationship between the unique identifiers (person and building).); and
- identifying user-relevant content (Paragraph [0042], Nguyen teaches the GUI can also enable presentation of selections of the video feed, classified objects/activities, natural language text, and semantic network.) using either one of a user-defined semantic query (Paragraph [0048], Fig. 7, Nguyen teaches the GUI (1041) can enable queries of the knowledge base by the user. The queries can be based on a keyword entered by the user. The queries can be based on objects or events in a given time window and/or in a geographical area that is specified by the user. For example, as indicated by arrow (701), the user can specify a car and choose “Find Similar” to search visually similar cars as the specified car in the video feed. Under Broadest Reasonable Interpretation (BRI), the Examiner interprets “either one of a” and “and” means that only one of either “a user-defined semantic query” OR “a user-selected Video Anomaly Detection (VAD) model” is required to meet the claim limitation.) and a (Paragraphs [0035], [0013], Nguyen teaches the detection and recognition module is configured to detect and classify the moving objects using motion compensation, background subtraction, and convolutional neural networks. The Examiner interprets the VAD model to be a machine learning model, in light of paragraph [0030] of applicant’s specification, which states, “the said user-selected VAD model comprises at least one machine learning model, MLM, or a rules engine, configured to perform VAD, on the acquired video data.”), in combination with either one of a unique identifier (Paragraphs [0012], [0041], [0043], Fig. 3, Nguyen teaches assigning respective unique identifiers to the classified objects, and the presentation of selections of the video feed in the GUI includes identifying the classified objects on the video feed and including respective unique identifiers. For example, as shown in an image (300) (e.g., a frame of the video feed), the detection and recognition module (1021) can detect and classify objects as “car” or “person” or “handbag” in the video feed. Unique identifiers are assigned to the classified objects. Under Broadest Reasonable Interpretation (BRI), the Examiner interprets “either one of a” and “and” means that only one of either “a unique identifier” OR “a subset of the unique identifiers” is required to meet the claim limitation.)
Nguyen does not explicitly disclose the following:
- identified by unique identifiers;
-
- user-selected a subset of the unique identifiers.
Jain is in the same field of art of using natural language processing to manage video data, such as surveillance/security video data. Further, Jain teaches:
- capturing video data from a plurality of video surveillance cameras (Paragraphs [0028-29], Fig. 2, Jain teaches video surveillance data is captured at a customer site (e.g., surveillance location). As shown in Fig. 2, a customer site (e.g., surveillance location) can include one or more surveillance cameras. The cameras (200) may be deployed to various locations around a customer site. Each camera may stream video data to the natural language video monitoring system (100).) identified by unique identifiers (Paragraphs [0055], [0025], Jain teaches video capture device identifiers or “camera ID”.);
- acquiring metadata (Paragraph [0025], Jain teaches storing associated metadata.) in combination with the unique identifiers (Paragraphs [0025-26], Jain teaches the vector database may store descriptor embeddings produced by a video indexing network and associated metadata, such as, video identifier, camera identifier, time of day, date, etc. The video identifier metadata associated with each similar embedding may be used to look up the corresponding video in video data.); and
- identifying user-relevant content (Paragraph [0026], Jain teaches the user interface displays relevant video results to the user. For example, matching videos can be ranked and presented to the user in the order of ranking. Additional information and/or a summary of the search results may be presented, such as a number of clips, or specifics relevant to the query.) using either one of a user-defined semantic query (Paragraph [0022], Jain teaches the query may be received locally (e.g., via a user interface). The query may be “find a person wearing a yellow hat, visible by camera 1 or camera 2, during the day on April 3rd.” In some embodiments, a user interface is provided allowing a user to input text input using a web dashboard, etc. Alternatively, the user can be presented with a predefined set of text queries.) and a (Paragraph [0029], Jain teaches the natural language video monitoring system (100) may include a neural network manager (106) that provides an execution environment for one or more machine learning models. The Examiner interprets the VAD model to be a machine learning model, in light of paragraph [0030] of applicant’s specification, which states, “the said user-selected VAD model comprises at least one machine learning model, MLM, or a rules engine, configured to perform VAD, on the acquired video data.” In addition, under Broadest Reasonable Interpretation (BRI), the Examiner interprets “either one of a” and “and” to mean only one of either “a user-defined semantic query” OR “a user-selected Video Anomaly Detection (VAD) model” is required to meet the claim limitation.), in combination with either one of a unique identifier (Paragraph [0022], Jain teaches the input query (102) may define an object, person, etc. being searched for, a camera location, camera identifier, date, time, or other search parameters.) and a subset of the unique identifiers (Paragraph [0022], Jain teaches for example, the query may be “find a person wearing a yellow hat, visible by camera 1 or camera 2, during the day on April 3rd.” The Examiner interprets camera 1 or camera 2 to be a “subset” of unique identifiers. In addition, under Broadest Reasonable Interpretation (BRI), the Examiner interprets “either one of a” and “and” to mean only one of either “a unique identifier” OR “a subset of unique identifiers” is required to meet the claim limitation.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Nguyen by identifying/assigning the surveillance cameras unique identifiers or “camera ID’s” that is taught by Jain, to make the invention that distinguishes surveillance cameras from other cameras in a network or system and allows the user to define a specific camera identifier within their input query such as “find a person visible by camera 1 or camera 2”, for example (Jain, Paragraph [0022]); thus, one of ordinary skilled in the art would be motivated to combine the references since camera identifiers or camera ID’s assigned to the surveillance cameras so the cameras can be easily recognized, managed, and secured within a network or system so that the user is able to ask a specific question regarding a particular camera or area captured by a specific camera (Jain, Paragraph [0022]). This may mitigate the need for manually reviewing hours of surveillance recordings across tens or hundreds of devices, which may save time, money, and resources (Jain, Paragraph [0016]).
In regards to Claim 20, Nguyen discloses:
A video surveillance system (Paragraphs [0032], [0053], Fig. 9, Nguyen teaches a system (100) for intelligent video analysis. The system may be implemented by an apparatus for intelligent video analysis.) comprising one or more processors (Paragraphs [0053-55], Fig. 9, Nguyen teaches the apparatus may include one or more of each of a number of components such as, for example, processor (901) (e.g., processing circuitry).)
PNG
media_image11.png
507
652
media_image11.png
Greyscale
configured to:
- cause capture of video data by a plurality of video surveillance cameras (Paragraph [0042], Fig. 2, Nguyen teaches receiving video feeds from multiple sources. Some of the sources can be stationary, e.g., the source (201) can be a stationary camera. Some of the sources can be moving, e.g., the source (202) can be camera or imaging sensor positioned on a manned or unmanned aerial vehicle.)
- acquire metadata (Paragraph [0038], Nguyen teaches storing the video feed, classified objects, and classified activities, natural language text, and semantic network in the knowledge base.) generated by performing captioning of the video data (Paragraph [0036], Nguyen teaches the natural language generator (1022) is configured to generate natural language text that describes the video feed from the classified object and activities. For example, the natural language generator can generate natural language text such as “man riding bicycle” to describe the video feed.), wherein the metadata comprises semantic data (Paragraphs [0037-38], [0046], Nguyen teaches semantic network is stored in the knowledge base (103).) to represent content in the video data (Paragraphs [0037], [0046], Fig. 5, Nguyen teaches in order to display semantic relationships between the classified objects to a user, the semantic network generator (1023) is configured to produce a semantic network including a graph. Graph (500) may have vertices that represent the classified objects, and edges that connect the vertices and represent semantic relationships between the classified objects. For example, the vertex (501) can represent a person and a vertex (503) can represent a building. The edge (502) that connects the vertices (501) and (503) can represent a semantic relationship that the person enters the building.) in combination with the unique identifiers (Paragraphs [0012], [0043], [0046], Fig. 5, Nguyen teaches assigning respective unique identifiers to the classified objects. The semantic network may include a graph (500) with vertices that represent the classified objects, and edges that connect the vertices and represent semantic relationships between the classified objects. For example, the vertex (501) can represent a person and vertex (503) can represent a building. The edge (502) that connects vertices (501) and (503) can represent a semantic relationship that the person enters the building. The Examiner interprets the semantic data is “in combination with” the unique identifiers (unique identifiers assigned to classified objects) since the semantic network represents the semantic relationship(s) between the classified objects. For example, under BRI, “person” (501) and “building” (503) are being interpreted as unique identifiers. The semantic graph represents the semantic relationship between the unique identifiers (person and building).); and
- identify the said user-relevant content (Paragraph [0042], Nguyen teaches the GUI can also enable presentation of selections of the video feed, classified objects/activities, natural language text, and semantic network.) using either one of a user-defined semantic query (Paragraph [0048], Fig. 7, Nguyen teaches the GUI (1041) can enable queries of the knowledge base by the user. The queries can be based on a keyword entered by the user. The queries can be based on objects or events in a given time window and/or in a geographical area that is specified by the user. For example, as indicated by arrow (701), the user can specify a car and choose “Find Similar” to search visually similar cars as the specified car in the video feed. Under Broadest Reasonable Interpretation (BRI), the Examiner interprets “either one of a” and “and” means that only one of either “a user-defined semantic query” OR “a user-selected Video Anomaly Detection (VAD) model” is required to meet the claim limitation.) and a (Paragraphs [0035], [0013], Nguyen teaches the detection and recognition module is configured to detect and classify the moving objects using motion compensation, background subtraction, and convolutional neural networks. The Examiner interprets the VAD model to be a machine learning model, in light of paragraph [0030] of applicant’s specification, which states, “the said user-selected VAD model comprises at least one machine learning model, MLM, or a rules engine, configured to perform VAD, on the acquired video data.”), in combination with either one of a unique identifier (Paragraphs [0012], [0041], [0043], Fig. 3, Nguyen teaches assigning respective unique identifiers to the classified objects, and the presentation of selections of the video feed in the GUI includes identifying the classified objects on the video feed and including respective unique identifiers. For example, as shown in an image (300) (e.g., a frame of the video feed), the detection and recognition module (1021) can detect and classify objects as “car” or “person” or “handbag” in the video feed. Unique identifiers are assigned to the classified objects. Under Broadest Reasonable Interpretation (BRI), the Examiner interprets “either one of a” and “and” means that only one of either “a unique identifier” OR “a subset of the unique identifiers” is required to meet the claim limitation.)
Nguyen does not explicitly disclose:
- identified by unique identifiers;
-
- user-selecteda subset of the unique identifiers.
Jain is in the same field of art of using natural language processing to manage video data, such as surveillance/security video data. Further, Jain teaches:
- cause capture of video data by a plurality of video surveillance cameras (Paragraphs [0028-29], Fig. 2, Jain teaches video surveillance data is captured at a customer site (e.g., surveillance location). As shown in Fig. 2, a customer site (e.g., surveillance location) can include one or more surveillance cameras. The cameras (200) may be deployed to various locations around a customer site. Each camera may stream video data to the natural language video monitoring system (100).) identified by unique identifiers (Paragraphs [0055], [0025], Jain teaches video capture device identifiers or “camera ID”.);
- acquire metadata (Paragraph [0025], Jain teaches storing associated metadata.) in combination with the unique identifiers (Paragraphs [0025-26], Jain teaches the vector database may store descriptor embeddings produced by a video indexing network and associated metadata, such as, video identifier, camera identifier, time of day, date, etc. The video identifier metadata associated with each similar embedding may be used to look up the corresponding video in video data.); and
- identify the said user-relevant content (Paragraph [0026], Jain teaches the user interface displays relevant video results to the user. For example, matching videos can be ranked and presented to the user in the order of ranking. Additional information and/or a summary of the search results may be presented, such as a number of clips, or specifics relevant to the query.) using either one of a user-defined semantic query (Paragraph [0022], Jain teaches the query may be received locally (e.g., via a user interface). The query may be “find a person wearing a yellow hat, visible by camera 1 or camera 2, during the day on April 3rd.” In some embodiments, a user interface is provided allowing a user to input text input using a web dashboard, etc. Alternatively, the user can be presented with a predefined set of text queries.) and a (Paragraph [0029], Jain teaches the natural language video monitoring system (100) may include a neural network manager (106) that provides an execution environment for one or more machine learning models. The Examiner interprets the VAD model to be a machine learning model, in light of paragraph [0030] of applicant’s specification, which states, “the said user-selected VAD model comprises at least one machine learning model, MLM, or a rules engine, configured to perform VAD, on the acquired video data.” In addition, under Broadest Reasonable Interpretation (BRI), the Examiner interprets “either one of a” and “and” to mean only one of either “a user-defined semantic query” OR “a user-selected Video Anomaly Detection (VAD) model” is required to meet the claim limitation.), in combination with either one of a unique identifier (Paragraph [0022], Jain teaches the input query (102) may define an object, person, etc. being searched for, a camera location, camera identifier, date, time, or other search parameters.) and a subset of the unique identifiers (Paragraph [0022], Jain teaches for example, the query may be “find a person wearing a yellow hat, visible by camera 1 or camera 2, during the day on April 3rd.” The Examiner interprets camera 1 or camera 2 to be a “subset” of unique identifiers. In addition, under Broadest Reasonable Interpretation (BRI), the Examiner interprets “either one of a” and “and” to mean only one of either “a unique identifier” OR “a subset of unique identifiers” is required to meet the claim limitation.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Nguyen by identifying/assigning the surveillance cameras unique identifiers or “camera ID’s” that is taught by Jain, to make the invention that distinguishes surveillance cameras from other cameras in a network or system and allows the user to define a specific camera identifier within their input query such as “find a person visible by camera 1 or camera 2”, for example (Jain, Paragraph [0022]); thus, one of ordinary skilled in the art would be motivated to combine the references since camera identifiers or camera ID’s assigned to the surveillance cameras so the cameras can be easily recognized, managed, and secured within a network or system so that the user is able to ask a specific question regarding a particular camera or area captured by a specific camera (Jain, Paragraph [0022]). This may mitigate the need for manually reviewing hours of surveillance recordings across tens or hundreds of devices, which may save time, money, and resources (Jain, Paragraph [0016]).
Claims 3-11, 13, and 16-17 are rejected under 35 U.S.C. 103(a) as being unpatentable over Nguyen (U.S. Patent Pub. No. 20200302177 A1, hereafter referred to as Nguyen) in view of Jain et al. (U.S. Patent Pub No. 20250077576 A1, hereafter referred to as Jain) in further view of Glesinger et al. (U.S. Patent Pub. No. 20240104305 A1, hereafter referred to as Glesinger).
Regarding Claim 3, Nguyen in view of Jain discloses the method according to claim 2.
Nguyen in view of Jain does not explicitly disclose wherein performing captioning of the video data uses at least one machine learning model (MLM).
Glesinger is in the same field of art of acquiring semantic understanding of video content. Further, Glesinger teaches wherein performing captioning of the video data uses at least one machine learning model (MLM) (Paragraphs [0260-261], Glesinger teaches the audio stream is interpreted by a deep learning model (306). The interpreted audio stream is converted to natural language in a textual format and is mapped to the corresponding time within the video that the item of language occurs.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Nguyen in view of Jain by performing the captioning of the audio stream of the video using a machine learning model that is taught by Glesinger, to make the invention that automatically generates natural language descriptions of visual content (video); thus, one of ordinary skilled in the art would be motivated to combine the references since video analysis often involves human operators manually watching video streams and producing verbal narratives of video contents, however, manually reviewing/narrating the large amounts of streaming and archived video data can be impractical for a human to analyze, understand, and respond (Nguyen, Paragraph [0002]). Therefore, automatic interpretation and captioning of video data by a machine learning model may be faster and more efficient than manual interpretation/narration.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 4, Nguyen in view of Jain in further view of Glesinger teaches the method according to claim 3, wherein the said MLM performs Open- Set Recognition (OSR), to recognise classes of subjects, predicates and/or objects which have not been predefined (Paragraphs [0191-193], Glesinger teaches the candidate semantic chains are matched or compared against the existing corpus of semantic chains (810). If there is not a match, the new semantic chain may be a candidate for inclusion in the corpus of content, particularly if the associated W3 weight is sufficiently high. The system then finalized the new semantic chains. The corpus of semantic chains is updated to include the new semantic chains, and the enhanced corpus of semantic chains is then ready to be applied to interpret additional content. The system can automatically learn categorizations and causal relationships. Under BRI, the Examiner interprets “and/or” to mean only one of either “subjects”, “predicates”, or “objects” have not been predefined to meet the claim limitation.).
In regards to Claim 5, Nguyen in view of Jain in further view of Glesinger discloses the method according to claim 3, wherein the said MLM comprises a Large Language Model (LLM), as a first LLM, the first LLM being configured to perform captioning of the video data (Paragraph [0261], Glesinger teaches the audio stream is interpreted by a deep learning model (306). The interpreted audio stream is converted to natural language in a textual format and is mapped to the corresponding time within the video that the item of language occurs. The audio-derived natural language may be converted to a mathematical description such as by a vector-based embedding process applying a trained LLM.).
In regards to Claim 6, Nguyen in view of Jain in further view of Glesinger discloses the method according to claim 5, wherein the first LLM (Paragraph [0339], Glesinger teaches semantic chains may be embedded by application of a trained LLM.) is configured to generate the subject-predicate-object triples (Paragraph [0157], Glesinger teaches system-based relationships that are represented in the form of subject-predicate-object chain (a syntactic “triple”), wherein the predicate is typically a descriptive phrase, but can be a verb phrase, that semantically connects the subject with the object of the predicate. Since the subject and the object of the predicate can be represented as computer-implemented objects (212).) from the said captioning (Paragraph [0261], Glesinger teaches the audio stream is converted to natural language in a textual format. The audio-derived natural language may be converted to a mathematical description such as by a vector-based embedding process applying a trained LLM.).
In regards to Claim 7, Nguyen in view of Jain in further view of Glesinger discloses the method according to claim 5, wherein the said MLM uses a transformer encoder architecture, a transformer decoder architecture, or a transformer encoder- decoder architecture (Paragraph [0261], Glesinger teaches the audio stream is interpreted by a deep learning model (306), typically a variation or extension of an LSTM-type neural network such as an attention-based model such as a unidirectional or bidirectional encoder-decoder transformer. Under BRI, the Examiner interprets “or” to mean only one of the listed types of transformer architectures is required to meet the claim limitation.) to convert captions generated by the first LLM into the subject- predicate-object triples (Paragraph [0260-261], Glesinger teaches the natural language may be converted to a mathematical description such as by a vector-based embedding process such as applying transformer-based large language models (LLMs), which apply neural networks trained on a corpus of content to generate the vectorized embeddings.) which represent content in the video frames (Paragraph [0260-261], Glesinger teaches the video stream is interpreted by the deep learning-based system. As an example, the abstract concept of sentiment either expressed by the people through audio or visual cues within the video or inferred from the overall milieu within some or all of the video by the system, may be encoded as a natural language-based element or vector-based representation thereof.).
In regards to Claim 8, Nguyen in view of Jain in further view of Glesinger discloses the method according to claim 5, wherein the said MLM comprises another LLM (Paragraphs [0260-261], [0194], Glesinger teaches the video stream is interpreted by a deep learning-based system (304). In some embodiments, the natural language may be converted to a mathematical description such as by applying transformer-based LLM. The audio stream is interpreted by a deep learning model (306). The deep learning-based model may be an LLM.),
PNG
media_image12.png
619
520
media_image12.png
Greyscale
as a second LLM configured to convert captions generated by the first LLM (Paragraph [0261], Glesinger teaches the interpreted audio stream is converted to natural language in a textual format. The audio-derived natural language may be converted to a mathematical description such as by a vector based embedding process applying a trained LLM.) into the subject-predicate-object triples (Paragraph [0157], Glesinger teaches subjects, predicates, and objects of the predicates of semantic chains can be represented in a computer-based system (925) as computer-implemented objects (212). These objects (212) may include representations of semantic chains in vector embedding forms that are generated by trained neural networks.) which represent content in the video frames (Paragraphs [0259-261], Fig. 15, Glesinger teaches the video may be first segmented into a video stream (302) and an audio stream (303). The audio stream is converted to natural language in a textual format and mapped to the corresponding time within the video that the item of language occurs.).
In regards to Claim 9, Nguyen in view of Jain in further view of Glesinger teaches the method according to claim 5, wherein the said MLM comprises another LLM, as a second LLM configured to perform a fine-tuning or embedding training process of the first LLM (Paragraph [0260], [0194], Glesinger teaches the natural language may be converted to a mathematical description such as by a vector-based embedding process such as applying transformer-based large language models (LLMs), which apply neural networks trained on a corpus of content to generate the vectorized embeddings. The deep learning-based model may be an LLM. The Examiner interprets the word “or” to mean only one of performing either a “fine-tuning” or “embedding training process” is required to meet the claim limitation.) to make the captions generated by the first LLM conform to a subject-predicate-object format (Paragraph [0157], Glesinger teaches subjects, predicates, and objects of the predicates of semantic chains can be represented in a computer-based system (925) as computer-implemented objects (212). These objects (212) may include representations of semantic chains in vector embedding forms that are generated by trained neural networks.).
In regards to Claim 10, Nguyen in view of Jain discloses the method according to claim 2, wherein capturing the video data, acquiring the metadata (Paragraph [0009], Nguyen teaches processing the video feed in real-time as the video feed is received, including performing object detection and recognition on the video feed and classify objects therein, performing activity recognition to detect and classify activities of at least some of the objects, and outputting classified objects and classified activities in the video feed.), and identifying the said user-relevant content (Paragraph [0042], Nguyen teaches the GUI can also enable presentation of selections of the video feed, classified objects/activities, natural language text, and semantic network.) are performed in a live fashion (Paragraph [0073], Jain teaches the query system (1010) enables users to search live video.), the method further comprising generating a message, instruction, event and/or additional metadata in a Video Management System (VMS) (Paragraph [0043], Fig. 3, Nguyen teaches the detection and recognition module (1021) can detect and classify objects as “car” or “person” or “handbag” in the video feed. The detection and recognition module can assign respective unique identifiers to the classified objects. For example, the unique identifier “person 50” is assigned to the person in the video feed. The presentation of the video feed in the GUI includes identifying the classified objects on the video feed and including the respective unique identifiers. For example, “person 50” is presented in the video feed to identify the person in the video feed. See Fig. 3. Under BRI, the Examiner interprets “and/or” to mean only one of the listed items is required to meet the claim limitation (message OR instruction OR event OR additional metadata.), when the said user-defined semantic query (Paragraph [0048], Nguyen teaches queries can be based on a keyword entered by the user.) (Under BRI, the Examiner interprets “or” to mean only one of the limitations (either user-defined semantic query or user-selected VAD model) is required to meet the claim limitation.) in combination with either one of the said unique identifier (Paragraph [0048], Nguyen teaches the queries of the knowledge base can be based on a similarity between a user-specified object and one or more classified objects in the video feed. For example, as indicated by arrow (701), the user can specify a car and choose “Find Similar” to search visually similar cars as the specified car in the video feed. Under BRI, the Examiner interprets “either one of the” and “and” to mean only one of either the unique identifier or the subset of unique identifiers is required to meet the claim limitation.)
PNG
media_image13.png
355
906
media_image13.png
Greyscale
and the said subset of the unique identifiers (Paragraph [0048], Fig. 7, Nguyen teaches the queries of the knowledge base can be based on similarity between a user-specified object and one or more of the classified objects in the video feed. For example, as indicated by arrow 701, the user can specify a car and choose “Find Similar” to search visually similar cars as the specified car in the video feed. The Examiner interprets the specified car (701) is a “subset” of the unique identifiers, (i.e., the other tracks in the list).)
Nguyen in view of Jain does not explicitly disclose match at least one corresponding subject-predicate-object and at least one corresponding unique identifier in the metadata.
Glesinger is in the same field of art of acquiring semantic understanding of video content. Further, Glesinger teaches match at least one corresponding subject-predicate-object and at least one corresponding unique identifier in the metadata (Paragraphs [0256], [0253], Glesinger teaches for example, in response to an introspective self-directed interrogative, “Have I been to a baseball park?” the computer-based system (925) could search for stored chains that indicate the computer-based system (925) or elements thereof were at a baseball park. Additionally, the term “baseball park” could be evaluated for matches with the results of the neural network-based or Bayesian program learning-based processing of stored historical images or video of physical locations and associated syntactical elements whereby the computer-based system were proximal to a particular baseball park. The Examiner interprets the question “Have I been to a baseball park?” is a subject-predicate-object wherein the subject is “I”, the predicate is “been to a”, and the object is “baseball park”. Further, the subject-object-predicate (question) is matched to a corresponding “unique identifier” in the metadata, such as syntactical elements in images/videos.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Nguyen in view of Jain by matching the question (subject-predicate-object) to associated metadata in a video/image that is taught by Gelsinger, to make the invention that matches the subject/object with images or videos and associated syntactical elements; thus, one of ordinary skilled in the art would be motivated to combine the references to enable adaptive recommendations in response to direct user requests or queries, including search requests or requests for matching content to other content (Glesinger, Paragraph [0041]). In addition, there is a need for a computer-implemented process that facilitates automatically generating new content that is based upon a semantic-level understanding of what will be particularly engaging to the user (Glesinger, Paragraph [0003]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 11, Nguyen in view of Jain in further view of Glesinger teaches the method according to claim 10, wherein generating the message, instruction, event or additional metadata (Paragraph [0043], Fig. 3, Nguyen teaches the detection and recognition module (1021) can detect and classify objects as “car” or “person” or “handbag” in the video feed. The detection and recognition module can assign respective unique identifiers to the classified objects. For example, the unique identifier “person 50” is assigned to the person in the video feed. The presentation of the video feed in the GUI includes identifying the classified objects on the video feed and including the respective unique identifiers. For example, “person 50” is presented in the video feed to identify the person in the video feed. See Fig. 3. The Examiner interprets assigning “unique identifiers” to the classified objects in the video feed to be an “event” (identified bounding box) or “additional metadata”. Further, under BRI the Examiner interprets “or” to mean only one of either a message, instruction, event, or additional metadata is required to meet the claim limitation.)
PNG
media_image14.png
540
644
media_image14.png
Greyscale
is triggered by a rules engine, as a first rules engine (Paragraph [0182], Glesinger teaches the system might find that by processing some videos of baseball games from the 1970s that the at least some baseballs are, or at least have been, orange, but that far most of the instances that the system has processed are baseballs that are white, and so the system might answer the question, “What color are baseballs?” by indicating that, “Baseballs are usually white in my experience, but can sometimes be orange.” The Examiner interprets the system to be a “rules engine” since the system processes videos and generates the answer to the question based on rules or facts gathered from processing the videos.), based on the said user-defined semantic query (Paragraph [0042], Nguyen teaches the GUI can enable queries of the knowledge base by users such as the user (203). The GUI can enable presentation of selections of the video feed, classified objects and classified activities, natural language text, and semantic network.).
In regards to Claim 13, Nguyen in view of Jain in further view of Glesinger discloses the method according to claim 10, wherein the message comprises an alert to be displayed in the VMS to a user (Paragraph [0043], Fig. 3, Nguyen teaches the presentation of the video feed in the GUI includes identifying classified objects on the video feed and including the respective identifiers. For example, “person 50” is presented in the video feed to identify the person in the video feed. Under BRI, the Examiner interprets the unique identifier/label to be an “alert” since the unique identifier is a signal indicating the presence of an object.).
In regards to Claim 16, Nguyen in view of Jain discloses the method according to claim 15, wherein the semantic data uses subject- predicate-object triples to represent content (Paragraph [0046], Fig. 5, Nguyen teaches a graph (500) with vertices that represent the classified objects, and edges that connect the vertices and represent semantic relationships between the classified objects. Some of the semantic relationships correspond to some classified activities of the classified objects. For example, in Fig. 5, the vertex (501) can represent a person and vertex (503) can represent a building. The edge (502) that connects the vertices (501) and (503) can represent a semantic relationship that the person enters the building. The Examiner interprets “the person enters the building” is a subject-predicate-object triple. “The person” is the subject. “Enters the” is the predicate, and “the building” is the object.) in the video data (Paragraph [0050], Fig. 8, Nguyen teaches receiving a video feed (801). At sub-block (8023) of the method, the method includes producing a semantic network including a graph with vertices that represent the classified objects, and edges that connects the vertices and represent semantic relationships between the classified objects, at least some of the semantic relationships corresponding to respective ones of the classified activities.), wherein performing captioning of the video data (Paragraph [0036], Nguyen teaches the natural language generator (1022) is configured to generate natural language text that describes the video feed from the classified objects and activities. For example, the natural language generator can generate natural language text such as “man riding a bicycle” to describe the video feed.)
Nguyen in view of Jain does not explicitly disclose wherein performing captioning of the video data uses at least one machine learning model, MLM, the method further comprising fact-checking the subject-predicate-object triples of the first graph against at least one ontology-based knowledge graph representing possible subject-predicate-object triples.
Glesinger is in the same field of art of acquiring semantic understanding of video content. Further, Glesinger teaches wherein the semantic data uses subject- predicate-object triples to represent content (Paragraph [0157], Glesinger teaches the term “semantic” may specifically refer to system-based relationships that are represented in the form of a subject-predicate-object chain (a syntactic “triple”).) in the video data (Paragraph [0033], Glesinger teaches the content aspect comprises the information contained in, or references by the objects that are part of the structural aspect. The content aspect of the objects may include video.), wherein performing captioning of the video data uses at least one machine learning model, MLM (Paragraph [0261], Glesinger teaches the audio stream is interpreted by a deep learning model (306). The interpreted audio stream is converted to natural language in textual format and is mapped to the corresponding time within the video that the item of language occurs.), the method further comprising fact-checking the subject-predicate-object triples of the first graph (Paragraph [0189], Glesinger teaches confirm or disconfirming a categorization represented by a semantic chain that comprises a “is a type of” predicate. This may be because the associated W1 or W4 weight is relatively low, which may be indicative that a system has relatively few attribute examples on which to base the associated weight.) against at least one ontology-based knowledge graph (Paragraph [0170], Glesinger teaches the correspondence may be made by alphanumeric-based correspondences, potentially with the aid of synonym tables and/or knowledge networks comprising ontologies.) representing possible subject-predicate-object triples (Paragraph [0263], Glesinger teaches the knowledge network and generalized semantic chains function (312), which may comprise a stored corpus of semantic chains and/or a network of parent/child relationships among natural language-based elements, may perform semantic chaining operations to aid in its interpretations and inferences.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Nguyen in view of Jain by using a machine learning model to perform interpretation of the video audio stream as well as confirming/disconfirming a categorization of the subject-predicate-object triples based on ontological relationships that is taught by Glesinger, to make the invention that enables interpretation of, and facilitating user navigation and interaction with, temporally sequenced media or content objects such as video; thus, one of ordinary skilled in the art would be motivated to combine the references since although semantic graphs (such as fuzzy networks) represent content/objects as well as the relationships between them, they may lack formal consistency with accurate, real-world knowledge whereas ontological graphs ensure that entities and the relationships between them conform to a known consistent schema. In addition, original such as content-specific tags and semantic summaries functions may supplement or substitute for the graph-based database to optimize performance for various functions (Glesinger, Paragraphs [0310-311]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 17, Nguyen in view of Jain in further view of Glesinger discloses the method according to claim 16, wherein the ontology-based knowledge graph is created or selected amongst multiple graphs (Paragraph [0170], Glesinger teaches the correspondence may be made by alphanumeric-based correspondences, potentially with the aid of synonym tables and/or knowledge networks comprising ontologies.) depending on contextual information representing attributes (Paragraph [0260], Glesinger teaches ontological relationships among objects and events.) of the said video surveillance cameras (Paragraph [0181], Glesinger teaches the system (925) is provided a second set of videos (which could be sourced from cameras that stream real-world information to the system) from which the system learns, including learning to generalize, based upon its prior learning to identify specific physical objects within a video. The system infers physical objects that are represented in the second set of videos (which translate to subjects and objects in semantic chains) and infers attributes, relationships, and interactions among the physical objects (which may map to predicates in semantic chains).) and/or an environment in which the video data is captured (Paragraph [0201], Glesinger teaches continuously building and updating its ontological models of the world, whether symbolically or in vector representational forms. Under BRI, the Examiner interprets “the world” to be “an environment”.).
Claim 12 is rejected under 35 U.S.C. 103(a) as being unpatentable over Nguyen (U.S. Patent Pub. No. 20200302177 A1, hereafter referred to as Nguyen) in view of Jain et al. (U.S. Patent Pub No. 20250077576 A1, hereafter referred to as Jain) in further view of Glesinger et al. (U.S. Patent Pub. No. 20240104305 A1, hereafter referred to as Glesinger) in further view of Xiong et al. (U.S. Patent No. 12,494,063 B2, hereafter referred to as Xiong).
Regarding Claim 12, Nguyen in view of Jain in further view of Glesinger discloses the method according to claim 10.
Nguyen in view of Jain in further view of Glesinger does not explicitly disclose wherein the instruction comprises an instruction to modify at least a part of the video data or a display thereof.
Xiong is in the same field of art of performing video processing on video surveillance data. Further, Xiong teaches wherein the instruction (Col. 15, lines 42-48, Xiong teaches alert manager (182) may be configured to overlay graphical elements representing detected events or event indicators on video streams displayed through video manager (180).) comprises an instruction to modify at least a part of the video data or a display thereof (Col. 15, lines 52-63, Xiong teaches alert manager (182) may include an interface to allow users to define one or more post-processing rules for use by the video cameras associated with the system. For example, users may be provided with rule templates and/or system generated rules (e.g., rules generated by calibration system (164)) and allowed to accept, reject, or modify post-processing rules for filtering potential false object detections that should not trigger event notifications.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Nguyen in view of Jain in further view of Glesinger by providing users with rules and allowing them to accept, reject of modify the post-processing rules that is taught by Xiong, to make the invention that reduces false alarms/object detections detected by the camera; thus, one of ordinary skilled in the art would be motivated to combine the references since allowing users to modify post-processing rules may help improve the reliability and accuracy of object detectors in surveillance cameras (Xiong, Col. 1, lines 45-51) .
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 18 is rejected under 35 U.S.C. 103(a) as being unpatentable over Nguyen (U.S. Patent Pub. No. 20200302177 A1, hereafter referred to as Nguyen) in view of Jain et al. (U.S. Patent Pub No. 20250077576 A1, hereafter referred to as Jain) in further view of Balasubramanian et al. (U.S. Patent Pub. No. 20220171995 A1, hereafter referred to as Balasubramanian).
Regarding Claim 18, Nguyen in view of Jain discloses the method according to claim 1, wherein the said MLM, or a rules engine (Paragraph [0054], Fig. 2, Jain teaches when the video data is received, it is processed by one or more video segmentation models (202) and a video indexing model (204). The natural language video monitoring system (100) may include a neural network manager (106) that provides an execution environment for one or more machine learning models. Under BRI, the Examiner interprets “at least one” and “or” to mean only one of a machine learning model, MLM, or a rules engine is required to meet the claim limitation.),
PNG
media_image15.png
354
905
media_image15.png
Greyscale
configured to perform VAD (Paragraph [0030], Jain teaches the video segmentation models (202) may be used to identify a region of interest (if any) in each frame.), on the acquired video data (Paragraph [0029], Jain teaches the video data is processed by one or more video segmentation models (202).) corresponding to the said unique identifier or the said subset of the unique identifiers (Paragraph [0030], Jain teaches each frame may be associated with a video identifier (206). The video identifier may be a time stamp or other arbitrary identifier value that uniquely identifies the corresponding video frame. These video identifiers may be synchronized with the vector database, such that the video index embeddings and the video data share the same video identifiers.).
Nguyen in view of Jain does not explicitly disclose said user-selected VAD model comprises at least one machine learning model, MLM, or a rules engine.
Balasubramanian is in the same field of art of anomaly detection in videos. Further, Balasubramanian teaches said user-selected VAD model comprises at least one machine learning model, MLM, or a rules engine (Paragraph [0064], Balasubramanian teaches a user can dictate what models are to be used and in what order. A selection is made for the user based on input from the user (such as answers about questions about the objective(s)).).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Nguyen in view of Jain by allowing the user to select the model, such as an anomaly detection model to be used to perform anomaly detection on images/video frames that is taught by Balasubramanian, to make the invention that uses a manually selected model to perform video anomaly detection; thus, one of ordinary skilled in the art would be motivated to combine the references since different models may be used for different video/image analysis tasks, such as object analysis, texture analysis, etc. (Balasubramanian, Paragraph [0073]) and manual model selection by the user may enable the user to select the appropriate model for their specific query to identify information relevant to the user.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Pertinent Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Coulter et al. (U.S. Patent Pub. No. 20250053587 A1) teaches methods and systems for identifying anomalous computer events to detect security incidents.
Choe et al. (U.S. Patent Pub. No. 20130129307 A1) teaches a system and method for video surveillance and searching. Video is analyzed and events are automatically detected. Based on the detected events, textual descriptions are generated. The textual descriptions may be used to search for events.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SYDNEY L BLACKSTEN whose telephone number is (571)272-7651. The examiner can normally be reached 8:30am-4:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Oneal Mistry can be reached at 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SYDNEY L BLACKSTEN/Examiner, Art Unit 2674
/ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674
