Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Detailed Action
This action is in response to the claims filed 4/2/2024:
Claims 1 – 20 are pending.
Claims 1, 6, and 12 are independent.
Claim Objections
Claim 5 objected to because of the following informalities: "the first ANN" should read "the first ANN surveillance model" for consistency. Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
“a plurality of network video recorders (NVRs), each configured to” in claim 12.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof. The instant specification makes the structure of a network video recorder ambiguous ([¶0012] “The local devices can be cameras such as network video recorders (NVRs).” or “NVRs can act as standalone smart surveillance systems powered by artificial intelligence (AI) in high-risk, security-vulnerable areas such as factory environments.” And later says “The different NVR 203-N can be a single camera for a particular facility. As another example, the different NVR 203-N can represent more than one camera for the particular facility.”. For these reasons the NVR is not seen as having clear structure in view of the instant specification.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 8-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claims 8 and 10, "the ANN surveillance model" lacks antecedent basis. Claim 6 recites "a first ANN surveillance model," "a federated ANN surveillance model," and "a second ANN surveillance model" such that it's unclear which model is "the ANN surveillance model".
Regarding claims 9 and 11, "the different ANN surveillance model" lacks antecedent basis. Claim 6 recites "a first ANN surveillance model," "a federated ANN surveillance model," and "a second ANN surveillance model" such that it's unclear which model is "the different ANN surveillance model".
Claim limitation “a plurality of network video recorders (NVRs), each configured to” in claim 12 invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function. The instant specification makes the structure of a network video recorder ambiguous ([¶0012] “The local devices can be cameras such as network video recorders (NVRs).” or “NVRs can act as standalone smart surveillance systems powered by artificial intelligence (AI) in high-risk, security-vulnerable areas such as factory environments.” And later says “The different NVR 203-N can be a single camera for a particular facility. As another example, the different NVR 203-N can represent more than one camera for the particular facility.”. For these reasons the NVR is not seen as having clear structure in view of the instant specification. Therefore, the claim is indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
Applicant may:
(a) Amend the claim so that the claim limitation will no longer be interpreted as a limitation under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph;
(b) Amend the written description of the specification such that it expressly recites what structure, material, or acts perform the entire claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(c) Amend the written description of the specification such that it clearly links the structure, material, or acts disclosed therein to the function recited in the claim, without introducing any new matter (35 U.S.C. 132(a)).
If applicant is of the opinion that the written description of the specification already implicitly or inherently discloses the corresponding structure, material, or acts and clearly links them to the function so that one of ordinary skill in the art would recognize what structure, material, or acts perform the claimed function, applicant should clarify the record by either:
(a) Amending the written description of the specification such that it expressly recites the corresponding structure, material, or acts for performing the claimed function and clearly links or associates the structure, material, or acts to the claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(b) Stating on the record what the corresponding structure, material, or acts, which are implicitly or inherently set forth in the written description of the specification, perform the claimed function. For more information, see 37 CFR 1.75(d) and MPEP §§ 608.01(o) and 2181.
Claims 13-20 are rejected with respect to their dependence on rejected claim 12.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-20 are rejected under U.S.C. §103 as being unpatentable over the combination of Chen (US20240135688A1) and Stormacq (“Computer Vision at the Edge with AWS Panorama”, 2021).
PNG
media_image1.png
578
700
media_image1.png
Greyscale
FIG. 4 of Chen
Regarding claim 1, Chen teaches A method, comprising: receiving respective updates to a first artificial neural network (ANN) surveillance model ([¶0071] "the edge device 600 transmits information regarding the local version of the global model to the server system 650 (step 605). […] the server system 650 may use this information to improve other edge devices. For example, this information could be incorporated into training of the global model")
from each of a plurality of network [video] recorders (NVRs) that have trained the first ANN surveillance model;([¶0006] "FIG. 1 includes a high-level illustration of a centralized surveillance system that includes various edge devices that are deployed throughout an environment to be surveilled." [¶0037] "the edge devices 102 a-n in FIG. 1 are cameras" [¶0024] "the collaborative approach mentioned above ensures that each edge device can develop a customized model that is tailored specifically for the data that is being generated, acquired, or otherwise obtained" edge device camera interpreted as NVR in view of the instant specification ([¶0029] "NVR 203-N can be a single camera for a particular facility"))
wherein the first ANN surveillance model is configured to cause the plurality of NVRs to perform a first surveillance function;([¶0020] "the edge devices in a surveillance system may execute models for inference or predictive purposes" [¶0043] 'The main goal of the model will depend on the type of edge device. For example, the main goal for a model employed by a camera may be object detection." [¶0045] "The first (and primary) goal is to have customized models that are independently adapted for the edge devices of a surveillance system" first edge model interpreted as first ANN surveillance model)
aggregating the respective updates into a federated ANN surveillance model;([¶0060] "the pretext tasks may be selected for each edge device using metrics such as gradient diversity or Shapely value to determine the quality of the update from each pretext task. Each pretext task will correspond to a loss function, and using the collective loss of the pretext tasks, the global model can be updated to adapt its representation to account for the data generated by the edge device [...] The function ψi is the aggregator function for m different pretext tasks'" [¶0071] "improvements may be federated across the surveillance system of which the edge device 600 is a part" [¶0082] "information gleaned through adapting local models could be applied in a federated manner as mentioned above" global model interpreted as federated ANN surveillance model)
training, via transfer learning based on the federated ANN surveillance model, a second ANN surveillance model for a different NVR;([¶0020] "Since the cloud model has a richer feature representation than the global and edge models, it can be used in knowledge distillation to distill knowledge to the edge models." [¶0043] "self-supervised learning can be combined with knowledge distillation methods that attempt to learn the representation of data on each edge device, while matching performance against a more powerful model" Chen explicitly performs federated learning and knowledge distillation (transfer learning). The federated learning updates the global model and edge models while the knowledge distillation (transfer learning) uses a cloud model to update the global model and edge models. The second model could be the updated global model or a second edge model. See also FIG. 1)
wherein the second ANN surveillance model is configured to cause the different NVR to perform a second surveillance function that is different than the first surveillance function; and([¶0053] " no matter how diverse the training data is, it cannot capture the entire distribution of images that may be generated by cameras that will employ that model. Therefore, performance of the model will degrade on cameras that generate images which are not comparable to the training data in terms of content." [¶0060] "The aggregator function could have different forms in different edge devices. For example, the aggregator function may have a weighted average over different tasks" See FIG. 4)
deploying the second ANN surveillance model to the different NVR.([¶0061] "The server system may be responsible for computing updates for the local models deployed on edge devices" See FIG. 4).
However, Chen does not explicitly teach from each of a plurality of network video recorders (NVRs) that have trained the first ANN surveillance model;.
PNG
media_image2.png
334
762
media_image2.png
Greyscale
FIG. 2 of Stormacq
Stormacq, in the same field of endeavor, teaches from each of a plurality of network video recorders (NVRs) that have trained the first ANN surveillance model;([p. 1] "Today, the AWS Panorama Appliance is generally available to all of you. The AWS Panorama Appliance is a computervision (CV) appliance designed to be deployed on your network to analyze images provided by your on-premises cameras. [...] when it’s time to analyze images from one or multiple video feeds" [p. 11] "There is a usage charge of $8.33 / month / camera feed. AWS Panorama stores versioned copies of all assets deployed to the AWS Panorama Appliance (including ML models and business logic) in the cloud. You are charged $0.10 per-GB, per-month for this storage.").
Chen as well as Stormacq are directed towards performing machine learning on camera data on edge devices. Therefore, Chen as well as Stormacq are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Chen with the teachings of Stormacq by applying monetization and using video data in the federated learning system of Chen. Stormacq provides as additional motivation for combination ([p. 1] “Every week, I read about new and innovative use cases for computer vision. Some customers are using CV to verify pallet trucks are parked in designated areas to ensure worker safety in warehouses, some are analyzing customer walking flows in retail stores to optimize space and product placement, and some are using it to recognize cats and mice, just to name a few. AWS customers agree the cloud is the most convenient place to train computer vision models thanks to its virtually infinite access to storage and compute resources. In the cloud, data scientists have access to powerful tools such as Amazon SageMaker and a wide variety of compute resources and frameworks.”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 2, the combination of Chen, and Stormacq teaches The method of claim 1, further comprising deploying the federated ANN surveillance model to the plurality of NVRs.(Chen [¶0082] "information gleaned through adapting local models could be applied in a federated manner as mentioned above" [¶0054] "a global model has been trained by a server system for object detection. This global model can then be provided to cameras deployed throughout an environment of interest" See also FIG. 4).
Regarding claim 3, the combination of Chen, and Stormacq teaches The method of claim 2, wherein the federated ANN surveillance model is configured to cause the plurality of NVRs to perform the first surveillance function.(Chen [Abstract] "a surveillance system may implement self-supervised learning and knowledge distillation that rely on collaboration between its edge devices and a server system" [¶0020] "edge devices in a surveillance system may execute models for inference or predictive purposes" [¶0078] " This local version of the global model may be referred to as a “local model.” Thereafter, the edge device may apply the local model to data that is generated in order to produce outputs that are representative of inferences or predictions." See also FIG. 4).
Regarding claim 4, the combination of Chen, and Stormacq teaches The method of claim 2, further comprising: charging a first entity controlling the plurality of NVRs a first price for deployment of the federated ANN surveillance model to each of the plurality of NVRs; and(Stormacq [p. 1] "Today, the AWS Panorama Appliance is generally available to all of you. The AWS Panorama Appliance is a computervision (CV) appliance designed to be deployed on your network to analyze images provided by your on-premises cameras. [...] when it’s time to analyze images from one or multiple video feeds" [p. 11] "There is a usage charge of $8.33 / month / camera feed. AWS Panorama stores versioned copies of all assets deployed to the AWS Panorama Appliance (including ML models and business logic) in the cloud. You are charged $0.10 per-GB, per-month for this storage.")
charging a second entity controlling the different NVR a second price for deployment of the second ANN surveillance model to the different NVR.(Stormacq [p. 1] "Today, the AWS Panorama Appliance is generally available to all of you. The AWS Panorama Appliance is a computervision (CV) appliance designed to be deployed on your network to analyze images provided by your on-premises cameras. [...] when it’s time to analyze images from one or multiple video feeds" [p. 11] "There is a usage charge of $8.33 / month / camera feed. AWS Panorama stores versioned copies of all assets deployed to the AWS Panorama Appliance (including ML models and business logic) in the cloud. You are charged $0.10 per-GB, per-month for this storage.").
Regarding claim 5, the combination of Chen, and Stormacq teaches The method of claim 1, wherein receiving the respective updates comprises receiving the respective updates to the first ANN without receiving video data from the plurality of NVRs.(Chen [¶0046] "the goal is to learn a global model using decentralized data on edge devices without directly accessing the data generated by those edge devices. In such settings, the edge devices maintain control of their own data but can update the global model provided by a server system using their own data").
Regarding claim 6, Chen teaches An apparatus comprising: a memory; a processor coupled to the memory and configured to:([¶0087] "the routines executed to implement the embodiments of the disclosure may be implemented as part of an operating system or a specific application, component, program, object, module, or sequence of instructions (collectively referred to as “computer programs”). The computer programs typically comprise one or more instructions (e.g., instructions 904, 908, 928) set at various times in various memory and storage devices in an electronic device. When read and executed by the processors 902, the instruction(s) cause the processing system 900 to perform operations to execute elements involving the various aspects of the present disclosure.")
receive respective updates to a first artificial neural network (ANN) surveillance model ([¶0071] "the edge device 600 transmits information regarding the local version of the global model to the server system 650 (step 605). […] the server system 650 may use this information to improve other edge devices. For example, this information could be incorporated into training of the global model")
from each of a plurality of network [video] recorders (NVRs) that have trained the first ANN surveillance model;([¶0006] "FIG. 1 includes a high-level illustration of a centralized surveillance system that includes various edge devices that are deployed throughout an environment to be surveilled." [¶0037] "the edge devices 102 a-n in FIG. 1 are cameras" [¶0024] "the collaborative approach mentioned above ensures that each edge device can develop a customized model that is tailored specifically for the data that is being generated, acquired, or otherwise obtained" edge device camera interpreted as NVR in view of the instant specification ([¶0029] "NVR 203-N can be a single camera for a particular facility"))
aggregate the respective updates into a federated ANN surveillance model;([¶0060] "the pretext tasks may be selected for each edge device using metrics such as gradient diversity or Shapely value to determine the quality of the update from each pretext task. Each pretext task will correspond to a loss function, and using the collective loss of the pretext tasks, the global model can be updated to adapt its representation to account for the data generated by the edge device [...] The function ψi is the aggregator function for m different pretext tasks'" [¶0071] "improvements may be federated across the surveillance system of which the edge device 600 is a part" [¶0082] "information gleaned through adapting local models could be applied in a federated manner as mentioned above" global model interpreted as federated ANN surveillance model)
deploy the federated ANN surveillance model to the plurality of NVRs;([¶0061] "The server system may be responsible for computing updates for the local models deployed on edge devices" See FIG. 4)
train, via transfer learning based on the federated ANN surveillance model, a second ANN surveillance model for a different NVR; and ([¶0020] "Since the cloud model has a richer feature representation than the global and edge models, it can be used in knowledge distillation to distill knowledge to the edge models." [¶0043] "self-supervised learning can be combined with knowledge distillation methods that attempt to learn the representation of data on each edge device, while matching performance against a more powerful model" Chen explicitly performs federated learning and knowledge distillation (transfer learning). The federated learning updates the global model and edge models while the knowledge distillation (transfer learning) uses a cloud model to update the global model and edge models. The second model could be the updated global model or a second edge model. See also FIG. 1)
deploy the second ANN surveillance model to the different NVR; wherein the different NVR is part of a different surveillance system than the plurality of NVRs. ([¶0061] "The server system may be responsible for computing updates for the local models deployed on edge devices" See FIG. 4. The model/camera system for each edge device is interpreted as a different surveillance system respective to the other surveillance systems).
However, Chen does not explicitly teach from each of a plurality of network video recorders (NVRs) that have trained the first ANN surveillance model;.
Stormacq, in the same field of endeavor, teaches from each of a plurality of network video recorders (NVRs) that have trained the first ANN surveillance model;([p. 1] "Today, the AWS Panorama Appliance is generally available to all of you. The AWS Panorama Appliance is a computervision (CV) appliance designed to be deployed on your network to analyze images provided by your on-premises cameras. [...] when it’s time to analyze images from one or multiple video feeds" [p. 11] "There is a usage charge of $8.33 / month / camera feed. AWS Panorama stores versioned copies of all assets deployed to the AWS Panorama Appliance (including ML models and business logic) in the cloud. You are charged $0.10 per-GB, per-month for this storage.").
Chen as well as Stormacq are directed towards performing machine learning on camera data on edge devices. Therefore, Chen as well as Stormacq are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Chen with the teachings of Stormacq by applying monetization and using video data in the federated learning system of Chen. Stormacq provides as additional motivation for combination ([p. 1] “Every week, I read about new and innovative use cases for computer vision. Some customers are using CV to verify pallet trucks are parked in designated areas to ensure worker safety in warehouses, some are analyzing customer walking flows in retail stores to optimize space and product placement, and some are using it to recognize cats and mice, just to name a few. AWS customers agree the cloud is the most convenient place to train computer vision models thanks to its virtually infinite access to storage and compute resources. In the cloud, data scientists have access to powerful tools such as Amazon SageMaker and a wide variety of compute resources and frameworks.”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 7, the combination of Chen, and Stormacq teaches The apparatus of claim 6, wherein the second ANN surveillance model is trained to perform a different surveillance function than the first ANN surveillance model.(Chen [¶0053] " no matter how diverse the training data is, it cannot capture the entire distribution of images that may be generated by cameras that will employ that model. Therefore, performance of the model will degrade on cameras that generate images which are not comparable to the training data in terms of content." [¶0060] "The aggregator function could have different forms in different edge devices. For example, the aggregator function may have a weighted average over different tasks" See FIG. 4).
Regarding claim 8, the combination of Chen, and Stormacq teaches The apparatus of claim 6, wherein the processor is further configured to deploy the ANN surveillance model to the plurality of NVRs prior to receiving the respective updates.(Chen [¶0040] "In a centralized surveillance system, a global model is created and then trained by the server system 106 in a supervised manner until acceptable performance is attained on a test dataset. The global model is then deployed to the edge devices 102 a-n for inference." [¶0071] "the edge device 600 transmits information regarding the local version of the global model to the server system 650 (step 605). The server system 650 may simply store this information, for example, in a digital profile associated with the surveillance system of which the edge device is a part. Alternatively, the server system 650 may use this information to improve other edge devices" Chen's sequence is explicit: the server first creates/trains and deploys the global model to edge devices; each edge device then creates or tunes a local version; and later the edge device transmits information about that local version to the server. Therefore, the processor deploys the model before receiving local update information).
Regarding claim 9, the combination of Chen, and Stormacq teaches The apparatus of claim 6, wherein the processor is further configured to: charge a first price to a first entity controlling the plurality of NVRs for deployment of the federated ANN surveillance model to each of the plurality of NVRs; and(Stormacq [p. 1] "Today, the AWS Panorama Appliance is generally available to all of you. The AWS Panorama Appliance is a computervision (CV) appliance designed to be deployed on your network to analyze images provided by your on-premises cameras." [p. 11] "There is a usage charge of $8.33 / month / camera feed. AWS Panorama stores versioned copies of all assets deployed to the AWS Panorama Appliance (including ML models and business logic) in the cloud. You are charged $0.10 per-GB, per-month for this storage.")
charge a second price to a second entity controlling the different NVR for deployment of the different ANN surveillance model to the different NVR.(Stormacq [p. 1] "Today, the AWS Panorama Appliance is generally available to all of you. The AWS Panorama Appliance is a computervision (CV) appliance designed to be deployed on your network to analyze images provided by your on-premises cameras." [p. 11] "There is a usage charge of $8.33 / month / camera feed. AWS Panorama stores versioned copies of all assets deployed to the AWS Panorama Appliance (including ML models and business logic) in the cloud. You are charged $0.10 per-GB, per-month for this storage.").
Regarding claim 10, the combination of Chen, and Stormacq teaches The apparatus of claim 6, wherein the plurality of NVRs are components of a first discrete surveillance system;(Chen [Abstract] "a surveillance system may implement self-supervised learning and knowledge distillation that rely on collaboration between its edge devices and a server system" [¶0020] "edge devices in a surveillance system may execute models for inference or predictive purposes" [¶0078] " This local version of the global model may be referred to as a “local model.” Thereafter, the edge device may apply the local model to data that is generated in order to produce outputs that are representative of inferences or predictions." See also FIG. 4)
wherein the processor is further configured to: receive second respective updates to the ANN surveillance model from each of a second plurality of NVRs that are components of a second discrete surveillance system; and(Chen [¶0060] "the pretext tasks may be selected for each edge device using metrics such as gradient diversity or Shapely value to determine the quality of the update from each pretext task. Each pretext task will correspond to a loss function, and using the collective loss of the pretext tasks, the global model can be updated to adapt its representation to account for the data generated by the edge device [...] The function ψi is the aggregator function for m different pretext tasks'" [¶0071] "improvements may be federated across the surveillance system of which the edge device 600 is a part" [¶0082] "information gleaned through adapting local models could be applied in a federated manner as mentioned above" global model interpreted as federated ANN surveillance model. See FIG. 4)
aggregate the second respective updates with the respective updates into the federated ANN surveillance model.(Chen [¶0060] "the pretext tasks may be selected for each edge device using metrics such as gradient diversity or Shapely value to determine the quality of the update from each pretext task. Each pretext task will correspond to a loss function, and using the collective loss of the pretext tasks, the global model can be updated to adapt its representation to account for the data generated by the edge device [...] The function ψi is the aggregator function for m different pretext tasks'" [¶0071] "improvements may be federated across the surveillance system of which the edge device 600 is a part" [¶0082] "information gleaned through adapting local models could be applied in a federated manner as mentioned above" global model interpreted as federated ANN surveillance model. See FIG. 4).
Regarding claim 11, the combination of Chen, and Stormacq teaches The apparatus of claim 10, wherein the processor is further configured to: charge a first price to a first entity controlling the plurality of NVRs for deployment of the federated ANN surveillance model to each of the plurality of NVRs;(Stormacq [p. 1] "Today, the AWS Panorama Appliance is generally available to all of you. The AWS Panorama Appliance is a computervision (CV) appliance designed to be deployed on your network to analyze images provided by your on-premises cameras." [p. 11] "There is a usage charge of $8.33 / month / camera feed. AWS Panorama stores versioned copies of all assets deployed to the AWS Panorama Appliance (including ML models and business logic) in the cloud. You are charged $0.10 per-GB, per-month for this storage.")
charge a second price to a second entity controlling the second plurality of NVRs for deployment of the federated ANN surveillance model to each of the second plurality of NVRs; and(Stormacq [p. 1] "Today, the AWS Panorama Appliance is generally available to all of you. The AWS Panorama Appliance is a computervision (CV) appliance designed to be deployed on your network to analyze images provided by your on-premises cameras." [p. 11] "There is a usage charge of $8.33 / month / camera feed. AWS Panorama stores versioned copies of all assets deployed to the AWS Panorama Appliance (including ML models and business logic) in the cloud. You are charged $0.10 per-GB, per-month for this storage.")
charge a third price to a third entity controlling the different NVR for deployment of the different ANN surveillance model to the different NVR.(Stormacq [p. 1] "Today, the AWS Panorama Appliance is generally available to all of you. The AWS Panorama Appliance is a computervision (CV) appliance designed to be deployed on your network to analyze images provided by your on-premises cameras." [p. 11] "There is a usage charge of $8.33 / month / camera feed. AWS Panorama stores versioned copies of all assets deployed to the AWS Panorama Appliance (including ML models and business logic) in the cloud. You are charged $0.10 per-GB, per-month for this storage.").
Regarding claim 12, Chen teaches A system, comprising: a plurality of network [video] recorders (NVRs), each configured to:([¶0006] "FIG. 1 includes a high-level illustration of a centralized surveillance system that includes various edge devices that are deployed throughout an environment to be surveilled." [¶0037] "the edge devices 102 a-n in FIG. 1 are cameras" [¶0024] "the collaborative approach mentioned above ensures that each edge device can develop a customized model that is tailored specifically for the data that is being generated, acquired, or otherwise obtained" edge device camera interpreted as NVR in view of the instant specification ([¶0029] "NVR 203-N can be a single camera for a particular facility"))
receive respective video data from operation in a respective location;([¶0071] "the edge device 600 transmits information regarding the local version of the global model to the server system 650 (step 605). […] the server system 650 may use this information to improve other edge devices. For example, this information could be incorporated into training of the global model")
provide respective updates to the respective ANN surveillance model to a server; the server, configured to: aggregate the respective updates into a federated ANN surveillance model;([¶0060] "the pretext tasks may be selected for each edge device using metrics such as gradient diversity or Shapely value to determine the quality of the update from each pretext task. Each pretext task will correspond to a loss function, and using the collective loss of the pretext tasks, the global model can be updated to adapt its representation to account for the data generated by the edge device [...] The function ψi is the aggregator function for m different pretext tasks'" [¶0071] "improvements may be federated across the surveillance system of which the edge device 600 is a part" [¶0082] "information gleaned through adapting local models could be applied in a federated manner as mentioned above" global model interpreted as federated ANN surveillance model. See FIG. 4)
train, via transfer learning based on the federated ANN surveillance model, a different ANN surveillance model for a different NVR in a different location; and([¶0040] "the different locations in which the edge devices 102 a-n are deployed" [¶0020] "Since the cloud model has a richer feature representation than the global and edge models, it can be used in knowledge distillation to distill knowledge to the edge models." [¶0043] "self-supervised learning can be combined with knowledge distillation methods that attempt to learn the representation of data on each edge device, while matching performance against a more powerful model" See FIG. 4)
deploy the different ANN surveillance model to the different NVR.([¶0061] "The server system may be responsible for computing updates for the local models deployed on edge devices" See FIG. 4).
However, Chen does not explicitly teach A system, comprising: a plurality of network [video] recorders (NVRs), each configured to:
train a respective artificial neural network (ANN) surveillance model with the respective video data; and.
Stormacq, in the same field of endeavor, teaches A system, comprising: a plurality of network [video] recorders (NVRs), each configured to:([p. 1] "Today, the AWS Panorama Appliance is generally available to all of you. The AWS Panorama Appliance is a computervision (CV) appliance designed to be deployed on your network to analyze images provided by your on-premises cameras. [...] when it’s time to analyze images from one or multiple video feeds" [p. 11] "There is a usage charge of $8.33 / month / camera feed. AWS Panorama stores versioned copies of all assets deployed to the AWS Panorama Appliance (including ML models and business logic) in the cloud. You are charged $0.10 per-GB, per-month for this storage.")
train a respective artificial neural network (ANN) surveillance model with the respective video data; and([p. 1] "AWS customers agree the cloud is the most convenient place to train computer vision models" [p. 2] "The demo application from this blog uses a machine learning model to recognize objects in frames of video from a network camera" [p. 5] "The AWS Panorama Appliance supports multiple ML Model frameworks. Models may be trained on AmazonSageMaker or any other solution of your choice").
Chen as well as Stormacq are directed towards performing machine learning on camera data on edge devices. Therefore, Chen as well as Stormacq are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Chen with the teachings of Stormacq by applying monetization and using video data in the federated learning system of Chen. Stormacq provides as additional motivation for combination ([p. 1] “Every week, I read about new and innovative use cases for computer vision. Some customers are using CV to verify pallet trucks are parked in designated areas to ensure worker safety in warehouses, some are analyzing customer walking flows in retail stores to optimize space and product placement, and some are using it to recognize cats and mice, just to name a few. AWS customers agree the cloud is the most convenient place to train computer vision models thanks to its virtually infinite access to storage and compute resources. In the cloud, data scientists have access to powerful tools such as Amazon SageMaker and a wide variety of compute resources and frameworks.”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 13, the combination of Chen, and Stormacq teaches The system of claim 12, wherein the plurality of NVRs are further configured to provide a first surveillance function according to operation of the respective ANN surveillance model; and(Chen [¶0078] " following deployment within an environment to be surveilled by the user. For example, after being deployed, the edge device may initiate a connection with the server system and then request the most recent version of the global model. This local version of the global model may be referred to as a “local model.” Thereafter, the edge device may apply the local model to data that is generated in order to produce outputs that are representative of inferences or predictions.")
wherein the different NVR is configured to provide a second surveillance function according to operation of the different ANN surveillance model; wherein the first surveillance function is different than the second surveillance function.(Chen [¶0053] " no matter how diverse the training data is, it cannot capture the entire distribution of images that may be generated by cameras that will employ that model. Therefore, performance of the model will degrade on cameras that generate images which are not comparable to the training data in terms of content." [¶0060] "The aggregator function could have different forms in different edge devices. For example, the aggregator function may have a weighted average over different tasks" See FIG. 4).
Regarding claim 14, the combination of Chen, and Stormacq teaches The system of claim 12, wherein the server is further configured to deploy the federated ANN surveillance model to the plurality of NVRs.(Chen [¶0082] "information gleaned through adapting local models could be applied in a federated manner as mentioned above" [¶0054] "a global model has been trained by a server system for object detection. This global model can then be provided to cameras deployed throughout an environment of interest" See also FIG. 4).
Regarding claim 15, the combination of Chen, and Stormacq teaches The system of claim 14, wherein the plurality of NVRs are further configured to operate according to the federated ANN surveillance model.(Chen [¶0082] "information gleaned through adapting local models could be applied in a federated manner as mentioned above" [¶0054] "a global model has been trained by a server system for object detection. This global model can then be provided to cameras deployed throughout an environment of interest" See also FIG. 4).
Regarding claim 16, the combination of Chen, and Stormacq teaches The system of claim 15, wherein the plurality of NVRs are components of a discrete surveillance system; and(Chen [¶0061] "The server system may be responsible for computing updates for the local models deployed on edge devices" See FIG. 4. The model/camera system for each edge device is interpreted as a different surveillance system respective to the other surveillance systems. D1/D2 interpreted as being a plurality of NVRs that are components of a discrete surveillance system)
wherein the different NVR is not part of the discrete surveillance system.(Chen [¶0061] "The server system may be responsible for computing updates for the local models deployed on edge devices" See FIG. 4. The model/camera system for each edge device is interpreted as a different surveillance system respective to the other surveillance systems).
Regarding claim 17, the combination of Chen, and Stormacq teaches The system of claim 15, wherein a first subset of the plurality of NVRs are components of a first discrete surveillance system;(Chen [¶0061] "The server system may be responsible for computing updates for the local models deployed on edge devices" See FIG. 4. The model/camera system for each edge device is interpreted as a different surveillance system respective to the other surveillance systems)
wherein a second subset of the plurality of NVRs are components of a second discrete surveillance system; and wherein the different NVR is a standalone NVR.(Chen [¶0061] "The server system may be responsible for computing updates for the local models deployed on edge devices" See FIG. 4. The model/camera system for each edge device is interpreted as a different surveillance system respective to the other surveillance systems).
Regarding claim 18, the combination of Chen, and Stormacq teaches The system of claim 15, wherein a first subset of the plurality of NVRs are components of a first discrete surveillance system; wherein a second subset of the plurality of NVRs are components of a second discrete surveillance system; and wherein the different NVR is a component of a third discrete surveillance system.(Chen [¶0061] "The server system may be responsible for computing updates for the local models deployed on edge devices" See FIG. 4. The model/camera system for each edge device is interpreted as a different surveillance system respective to the other surveillance systems. FIG. 4 shows three discrete surveillance systems all with respective NVR and model.).
Regarding claim 19, the combination of Chen, and Stormacq teaches The system of claim 18, wherein the server is further configured to: charge a first price to a first entity controlling the first discrete surveillance system for the federated ANN surveillance model; (Stormacq [p. 1] "Today, the AWS Panorama Appliance is generally available to all of you. The AWS Panorama Appliance is a computervision (CV) appliance designed to be deployed on your network to analyze images provided by your on-premises cameras. [...] when it’s time to analyze images from one or multiple video feeds" [p. 11] "There is a usage charge of $8.33 / month / camera feed. AWS Panorama stores versioned copies of all assets deployed to the AWS Panorama Appliance (including ML models and business logic) in the cloud. You are charged $0.10 per-GB, per-month for this storage.")
charge a second price to a second entity controlling the second discrete surveillance system for the federated ANN surveillance model; and (Stormacq [p. 1] "Today, the AWS Panorama Appliance is generally available to all of you. The AWS Panorama Appliance is a computervision (CV) appliance designed to be deployed on your network to analyze images provided by your on-premises cameras. [...] when it’s time to analyze images from one or multiple video feeds" [p. 11] "There is a usage charge of $8.33 / month / camera feed. AWS Panorama stores versioned copies of all assets deployed to the AWS Panorama Appliance (including ML models and business logic) in the cloud. You are charged $0.10 per-GB, per-month for this storage.")
charge a third price to a third entity controlling the third discrete surveillance system. (Stormacq [p. 1] "Today, the AWS Panorama Appliance is generally available to all of you. The AWS Panorama Appliance is a computervision (CV) appliance designed to be deployed on your network to analyze images provided by your on-premises cameras. [...] when it’s time to analyze images from one or multiple video feeds" [p. 11] "There is a usage charge of $8.33 / month / camera feed. AWS Panorama stores versioned copies of all assets deployed to the AWS Panorama Appliance (including ML models and business logic) in the cloud. You are charged $0.10 per-GB, per-month for this storage.").
Regarding claim 20, the combination of Chen, and Stormacq teaches The system of claim 12, wherein the plurality of network video recorders (NVRs) are configured not to send the video data to the server.(Chen [¶0046] "the goal is to learn a global model using decentralized data on edge devices without directly accessing the data generated by those edge devices. In such settings, the edge devices maintain control of their own data but can update the global model provided by a server system using their own data").
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Xian Jiatong University (CN115470703A) is directed towards a federated transfer learning system for a surveillance camera network.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SIDNEY VINCENT BOSTWICK whose telephone number is (571)272-4720. The examiner can normally be reached M-F 7:30am-5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang can be reached on (571)270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SIDNEY VINCENT BOSTWICK/Examiner, Art Unit 2124
Prosecution Insights