Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on February 14, 2023 and July 02, 2024 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Status of Claims
The present application is being examined under the claims filed with Applicant’s Request for Continued Examination (RCE) on May 19, 2026. The status of said claims are as follows:
Claims 1-20 are pending.
Claims 1, 7, and 14 have been amended.
Claims 2-6, 8-13, and 15-20 remain as previously presented.
Response to Arguments
Regarding 35 U.S.C. § 103
Applicant argues that Sulzer fails to disclose the amended limitations of independent claims 1, 7, and 14, including mapping the actual label entered by the end-user to a threat level and at least one corresponding premises security action. Applicant further argues that Sulzer’s labels are used during expert-driven server-side training and that Sulzer’s alert configuration instead relies on algorithmic confidence thresholds. Applicant also argues that there would have been no motivation to combine Sulzer with Sanketi because Sulzer relies on expert annotation while Sanketi seeks to perform training without user intervention.
Applicant’s arguments directed to Sulzer and the combination of Sulzer with Sanketi have been considered but are moot because the present rejection no longer relies on Sulzer. The present rejection applies Dice (US20220335816A1) as the primary reference and relies on Dice for the premises-security/video-analytics context, the user-selected/user-defined analytics-tag feedback, the mapping of the analytics tag to a security-event priority and corresponding premises-security action, and training of the object-detection algorithm using video frames associated with the user-defined analytics tag, as set forth in detail in the rejection below.
Applicant’s remaining arguments have been fully considered but are not persuasive.
Applicant argues that Sanketi does not disclose the claimed specific end-user feedback because Sanketi’s on-device training may occur without the user’s intervention. This argument is not persuasive because the present rejection does not rely on Sanketi for the end-user-entered label or corresponding feedback. Dice teaches the user-selected/user-defined analytics tag and associated video examples, while Sanketi is relied upon for the on-device machine-learning framework, including device-resident example storage, retraining or updating an existing machine-learning model, subsequent use of the retrained model, and model distribution, as applicable to the claims.
Applicant further argues that Zhou is silent with respect to “an alarm configuration comprising a mapping of the at least one label entered by the end-user to a threat level and at least one corresponding premises security action”. This argument is not persuasive because the present rejection does not rely on Zhou for the alarm-configuration limitation. Dice is relied upon for that limitation. Zhou is relied upon for the separate limitation requiring that training the pretraining machine-learning model comprise adding at least one new classifier to the pretrained machine-learning model.
Applicant additionally argues that the actual label entered by the end-user must be mapped to both a threat level and a corresponding security action. The present rejection specifically addresses this amended relationship. As explained below, Dice teaches a user-selected/user-defined analytics tag associated with an alarm signal and further teaches user-selectable priorities associated with analytics tags/object-detection parameters, with different priorities resulting in different alarm actions. Under the broadest reasonable interpretation, Dice’s security-event priority constitutes the recited threat level. Accordingly, Dice teaches or suggests the claimed mapping of the end-user-entered label to a threat level and corresponding premises security action.
Applicant argues that independent claims 7 and 14 are patentable for reasons similar to those asserted with respect to claim 1. Applicant further argues that dependent claims 2-6, 8-13, and 15-20 are allowable by virtue of their dependency from allegedly allowable independent claims and because they recite additional features allegedly not disclosed or suggested by the art. These arguments are not persuasive. Independent claims 1, 7, and 14 remain rejected for the reasons set forth below, and the additional limitations of dependent claims 2-6, 8-13, and 15-20 have been separately considered and are taught or suggested by the applied combination for the reasons specifically set forth in the rejection.
Accordingly, Applicant’s arguments have been considered but do not overcome the rejection of claims 1-20 under 35 U.S.C. § 103 over Dice in view of Sanketi and further in view of Zhou.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Clifford V. Dice et al. (US20220335816A1), henceforth "Dice" in view of Pannag Sanketi et al. (US20220004929A1), henceforth "Sanketi" and further in view of Yingbo Zhou et al. (WO2020069039A1), henceforth "Zhou".
Regarding claim 1, Dice in view of Sanketi and further in view of Zhou, teach a user device for a premises security system that comprises a plurality of premises devices, the user device being associated with an end-user and configured with a pretrained machine learning model for detecting object types, the user device comprising processing circuitry being configured to:
“receive at least one media file from a media file database ” – Dice teaches this limitation in part. Dice teaches receiving and processing digital-video clips and frames stored in a digital-video database:
“If gatekeeper server 30 determines motion positioned within a region of interest, gatekeeper server 30 gathers the sequence of digital video frames associated with the motion.” (Dice, pg. 9, ¶[0076])
“The gathering can be accomplished by the receipt of gatekeeper server 30 of the identification of the sequence of digital video frames associated with each motion from motion detection module 46 and obtaining the identified digital video frames from an associated digital video clip, which may be stored in digital video database 44.” (Dice, pg. 9, ¶[0076])
Thus, Dice teaches receiving a media file from a media-file database. Dice does not expressly teach that the database resides on, or otherwise associated with, the subscriber’s user device.
“identify, using the pretrained machine learning model, at least one dominant object in the at least one media file;” – Dice teaches this limitation. Dice’s AI object-detection module detects one or more objects, their object types or classes, and their positions in video frames. Dice then identifies a tracked object by comparing the detected object type with active analytics tags and determining whether the detected object is positioned in an associated region of interest:
“Object detection module 48 may be configured to detect the presence of one or more objects within the one or more of the sequential video frames indicating motion.” (Dice, pg. 9, ¶[0077])
“Object detection module 48 may be configured to determine one or more object detection parameters, such as, without limitation, the type of object, the class of object, the position of the object within each digital video frame, and the identification of the one or more frames containing the object.” (Dice, pg. 9, ¶[0077])
“gatekeeper server 30 compares the type of object with one or more active analytics tags to determine the presence of a tracked object.” (Dice, pg. 9, ¶[0078])
“If gatekeeper server 30 determines the presence of a tracked object, then gatekeeper server 30 determines if the location of the object detected intersects with a region of interest associated with an active analytics tag.” (Dice, pg. 9, ¶[0078])
Under the broadest reasonable interpretation, the detected tracked object selected for continued analytics and alarm processing constitutes the recited “dominant object”.
“receive an end-user feedback indication associated with the at least one dominant object, the end-user feedback indication comprising at least one label entered by the end-user;” – Dice teaches this limitation. Dice teaches receiving operator-selected and operator-defined analytics tags, including user-defined tags that are associated with video frames and used to train an object-detection algorithm. Dice expressly states that the analytics tag may be user-selected by the client. The analytics tag constitutes the claimed label because it identifies an object type, class, behavior, clothing type, weapon type, animal type, or other object-related classification:
“Alarm monitoring module 36 may be configured to receive one or more operator selected analytics tags.” (Dice, pg. 3, ¶[0042])
“Alarm monitoring module 36 may also be configured to configure one or more operator defined analytics tags.” (Dice, pg. 3, ¶[0042])
“For instance, if camouflage is not a pre-determined analytics tag, then the object detection algorithm can be trained to detect camouflage in response to analyzing several digital video frames including camouflage.” (Dice, pg. 3-4, ¶[0042])
“The analytics tag can be user selected by the client using functionality on digital video analytics server 18.” (Dice, pg. 4, ¶[0045])
“Non-limiting examples of analytics tags include a vehicle or a human being. Other non-limiting examples of analytics tags include vehicles such as passenger car, truck, and heavy machinery; human behavior such as arguing, agitation, and violent motions; clothing such as face masks, scarfs, ski masks, camouflage, combat uniform, and bullet proof vest; firearms such as long guns and short guns; and animals such as horses, goats, dogs and cats.” (Dice, pg. 4, ¶[0043])
Thus, Dice teaches receiving a user-selected or user-defined object label associated with the detected object.
“manage an alarm configuration comprising a mapping of the at least one label entered by the end-user to a threat level and at least one corresponding premises security action:” – Dice teaches this limitation. Dice teaches associating each user-selected tag with a video alarm signal and configuring actions based on a video alarm signal associated with an analysis tag. Dice further teaches priorities associated with object-detection parameters and analytics tags, with the priorities being user-selectable through an interface. Higher and lower priorities cause different alarm actions:
“Each analytics tag may be associated with a video alarm signal.” (Dice, pg. 4, ¶[0045])
“The analytics tag may be part of the alarm event data indicative of the digital video alarm event. This association may be made through alarm monitoring module 36.” (Dice, pg. 4, ¶[0045])
“Alarm monitoring module 36 may be configured to perform one or more actions based on the video alarm signal associated with the receipt of data indicative that an alarm event is associated with an analytics tag.” (Dice, pg. 4, ¶[0046])
“These actions may differ based on the zone relating to the data indicative of the alarm event.” (Dice, pg. 4, ¶[0046])
Dice further teaches assigning different priorities based on the security significance of the detected object and event:
“Alarm monitoring module 36 may be configured to assign a higher and lower priority to these alarm events. For instance, the suspicious bag may be a higher priority alarm event than the animal near the fence.” (Dice, pg. 9-10, ¶[0082])
“Alarm monitoring module 36 may be configured to format and display a GUI to obtain an operator's selection of the priorities associated with digital video alarm signals including one or more regions of interest, one or more object detection parameters, and/or one or more analytics tags.” (Dice, pg. 10, ¶[0082])
Dice teaches corresponding differentiated premises-security actions:
“a first higher priority action may be taken in response to receiving a first digital video alarm signal associated with a first priority and a second lower priority action may be taken in response to receiving a second digital video alarm signal associated with a second priority lower than the first priority.” (Dice, pg. 10, ¶[0082])
“The higher priority action may be transmitting the digital video alarm signal to an authority server.” (Dice, pg. 10, ¶[0082])
“The lower priority action may not include transmitting the digital video alarm signal to any authority server. Instead, the lower priority action may be displaying the digital video clip on a GUI hosted by alarm monitoring module 36.” (Dice, pg. 10, ¶[0082])
Under the broadest reasonable interpretation, Dice’s “priority” constitutes the claimed threat level. The priority represents the relative security risk or seriousness of the detected tagged object or event. For example, a suspicious bag as compared with an animal near a fence, or an event indicating intrusion or potential loss of life as compared with an object in an approved area. Dice therefore teaches an alarm configuration mapping a user-selected analytics tag to a threat level and a corresponding premises-security action.
“” – Dice teaches this limitation in part. Dice teaches training an object-detection algorithm based on the combination of a user-defined analytics tag and digital-video frames associated with that tag:
“digital video frames associated with a user defined analytics tag may be transmitted to an object detection algorithm to train the object detection algorithm.” (Dice, pg. 3, ¶[0042])
“For instance, if camouflage is not a pre-determined analytics tag, then the object detection algorithm can be trained to detect camouflage in response to analyzing several digital video frames including camouflage.” (Dice, pg. 3-4, ¶[0042])
Thus, Dice teaches training the object-detection algorithm based on the end-user-entered label and associated media. Dice does not expressly teach that the training occurs on the user device, that the trained algorithm is generated by retraining a pretrained machine-learned model resident on the user device, or that the training adds a new classifier to the pretrained model.
Dice does not teach these limitations and/or portions of:
“associated with the user device;”
“generate an updated machine learning model by training the pretrained machine learning model … the training of the pretrained machine learning model comprising adding at least one new classifier to the pretrained machine learning model.”
Sanketi, however, teaches these limitations and/or portions of:
“associated with the user device;” – Sanketi teaches a centralized example database located on the user computing device and receiving training examples from applications on that device:
“the computing device 102 can further include a centralized example database 124 that stores training examples received from the applications 120a-c.” (Sanketi, pg. 6, ¶[0066])
“the on-device machine learning platform 122 can receive training examples from the applications 120a-c via an API (which may be referred to as the "collection API") and can manage storage of the examples in the centralized example database 124.” (Sanketi, pg. 6, ¶[0066])
When Dice’s digital-video frames or clips are used as Sanketi’s training examples, the combined system stores the claimed media files in a database associated with the user device.
“generate an updated machine learning model by training the pretrained machine learning model” – Sanketi teaches retraining or updating an existing machine-learned model on the user device based on training examples stored in the devices centralized example database, and thereafter using the retrained model to provide inferences:
“the applications 120a-c can communicate with the on-device machine learning platform 122 via an API (which may be referred to as the "training API") to cause re-training or updating of a machine-learned model 132a-c based on training examples stored in the centralized example database 124.” (Sanketi, pg. 7, ¶[0076])
“the on-device machine learning platform 122 can run training of the model 132a-c … based on previously collected examples.” (Sanketi, pg. 7, ¶[0076])
“After retraining of the model 132a-c, the re-trained model 132a-c can be used to provide inferences” (Sanketi, pg. 7, ¶[0077])
“Typically, these inferences will have higher accuracy since the model 132a-c has been re-trained on data that is specific to the user.” (Sanketi, pg. 7, ¶[0077])
Thus, Sanketi teaches generating an updated on-device model by retraining an existing model using device-stored training examples.
Neither Dice nor Sanketi teach these remaining limitations and/or portions of:
“the training of the pretrained machine learning model comprising adding at least one new classifier to the pretrained machine learning model.”
Zhou, however, teaches these remaining limitations and/or portions of:
“the training of the pretrained machine learning model comprising adding at least one new classifier to the pretrained machine learning model.” – Zhou teaches training an image-classification neural network based on a pretrained model and adding a task-specific classifier for a new task:
“In the Classifier model, the classifier in the last layer of the neural network may be tuned, while the former twenty file layers are transferred from the ImageNet pretrained model and are kept fixed during training.” (Zhou, pg. 15, ¶[0060])
“In this case, each task may add a task specific classifier and batch normalization layers, thus keeping the size of the Classifier model small.” (Zhou, pg. 15, ¶[0060])
Zhou further teaches merging newly created classifiers into the maintained network:
“after parameter optimizer 170 retrains the optimal architecture on a current task Ti, parameter optimizer 170 may merge the newly created and tuned parameters in the layers, task specific adapters and classifier(s) in the task specific neural network with the maintained super network S.” (Zhou, pg. 11, ¶[0041])
Thus, Zhou teaches training a pretrained image-processing neural network by adding a new task-specific classifier to the pretrained model.
It would have been obvious to one of ordinary skill in the art to further modify the Dice-Sanketi system to employ Zhou’s task-specific technique when the end-user defines a new analytics tag representing a new object type or class.
Dice expressly contemplates a new user-defined analytics tag, such as “camouflage”, that was not previously recognized by the object-detection algorithm. Zhou provides a known and compact mechanism for extending a pretrained image-processing model to recognize a new task or class: retraining the pretrained feature-extraction layers and adding a task-specific classifier. Zhou explains that this approach keeps the classifier model small.
A person of ordinary skill would therefore have been motivated to use Zhou’s task-specific classifier addition to implement Dice’s new user-defined analytics tag in Sanketi’s on-device retraining framework, thereby enabling the pretrained object-detection model to recognize an additional user-defined object class while retraining the existing pretrained model structure and limiting additional model size.
Accordingly, it would have been obvious to one of ordinary skill in the art to modify Dice’s digital-video alarm analytics system:
According to Sanketi, to store the model and labeled video training examples on the subscriber’s user device and retrain the model locally based on those examples; and
According to Zhou, to implement the newly defined analytics-tag class by adding a task-specific classifier to the pretrained model.
The resulting system would comprise a user device associated with an end-user that receives stored media, detects a dominant object, receives a user-entered object label, maps the label to a threat level and corresponding security action, and generates an updated model by training a pretrained model based on the label and media while adding a new classifier, as recited in claim 1. Therefore, claim 1 would have been obvious over Dice in view of Sanketi and further in view of Zhou.
Regarding claim 2, Dice in view of Sanketi and further in view of Zhou, teach the user device of claim 1, wherein the processing circuitry is further configured to:
“determine or modify an alarm configuration comprising a mapping of at least one detected object type to at least one corresponding premises security action;” – Dice teaches this limitation. Dice teaches configuring analytics tags corresponding to detected object types and using the association between the detected object type, an active analytics tag, and a region of interest to determine whether an alarm signal is transmitted. Thus, Dice maps a detected object type to a corresponding alarm/security action:
“gatekeeper server 30 compares the type of object with one or more active analytics tags to determine the presence of a tracked object.” (Dice, pg. 9, ¶[0078])
“For instance, if the region of interest is a doorway and a human is an analytics tag associated with the doorway, then an alarm signal is transmitted if the type of object is a human and the location of the human intersects with the doorway.” (Dice, pg. 9, ¶[0078])
“Conversely, an alarm signal is not transmitted when the type of object is not associated with a region of interest. For instance, if the region of interest is a doorway and a human is the only analytics tag associated with the doorway, then an alarm signal is not transmitted if the type of object is an animal or other non-human object type even if the location of the detected object intersects with the doorway.” (Dice, pg. 9, ¶[0078])
Dice further teaches that the alarm-monitoring module determines an alarm event when the detected object corresponds to an active analytics tag:
“Alarm monitoring module 36 may be configured to determine a video alarm event when an object detected is associated with an analytics tag, the analytics tag is associated with a region of interest, and the object detected intersects with a region of interest.” (Dice, pg. 8, ¶[0071])
“For instance, if person is active as an analytics tag for region of interest 224 (i.e. sidewalk region 226), any time digital video analytics server 18 detects a person within region of interest 224, an alarm event is determined.” (Dice, pg. 8, ¶[0071])
Dice additionally teaches differentiated alarm actions based on object-detection parameters. Dice’s claims expressly recite determining different priorities in response to object-detection parameters, including different object types, and transmitting a higher-priority alarm to an authority server.
Thus, Dice teaches determining or modifying an alarm configuration in which a detected object type, such as a human versus an animal, is associated with whether and how a premise-security alarm action is performed.
“and cause transmission of the alarm configuration ” – Dice teaches this limitation in part. Dice teaches transmitting information representing the alarm configuration and detected-object relationship to an alarm-monitoring server/module that perform the corresponding alarm action. Dice expressly teaches transmitting an alarm signal to the alarm-monitoring module based on object-detection parameters:
“gatekeeper server 30 is configured to transmit an alarm signal to alarm monitoring module 36 in response to one or more object detection parameters.” (Dice, pg. 9, ¶[0078])
Dice further teaches that the resulting analytics tag is transmitted to the alarm-monitoring server:
“Rather, the resulting analytics tag is transmitted to alarm monitoring server 22.” (Dice, pg. 4, ¶[0046])
Dice’s alarm-monitoring module then uses the analytics-tag configuration to determine the video alarm event and perform the corresponding action. For example, a detected person associated with an active analytics tag causes an alarm event to be determined and transmitted. Thus, Dice teaches transmitting alarm-configuration information to a control device (alarm monitoring server 22 / alarm monitoring module 36) for performing a premises-security action based on that configuration. Dice does not expressly teaches transmitting the updated machine-learning model from the user device to the control device for use in performing the premises-security action.
Dice does not teach these limitations and/or portions of:
“the updated machine learning model”
Sanketi, however, teaches these remaining limitations and/or portions of:
“the updated machine learning model” – Sanketi teaches that after on-device training, the device sends an update representing the retrained machine-learning model to a server:
“The device 216 can implement the training plan 208 to perform on-device training based on locally stored data. After such on device learning, the device 216 can provide an update to the federated server 212.” (Sanketi, pg. 8, ¶[0090])
“For example, the update can describe one or more parameters of the re-trained model or one or more changes to the parameters of the model that occurred during the re-training of the model.” (Sanketi, pg. 8, ¶[0090])
Sanketi similarly teaches:
“the on-device platform 122 can determine an update that describes the parameters of a re-trained machine-learned model 132a-c or changes to the parameters of the machine learned model 132a-c that occurred during the re-training of model 132a-c (e.g., a ‘gradient’).” (Sanketi, pg. 7, ¶[0078])
“The platform 122 can transmit the update to a central server computing device (e.g., "the cloud") …” (Sanketi, pg. 7, ¶[0078])
Sanketi further teaches a background training process in which:
“The training process can pick up a model state and training plan, repeatedly pick data from the cache to train the model, and then eventually publish statistic(s) and model update message(s) (e.g., to a cloud server).” (Sanketi, pg. 12, ¶[0165])
Thus, Sanketi teaches transmitting information constituting the trained state/parameters of the updated machine-learning model from the user device to another computing device after on-device retraining.
It would have been obvious to one of ordinary skill in the art to further configure the Dice-Sanketi system discussed with respect to claim 1 such that the user device transmits the alarm configuration and the updated machine-learning model/model parameters to Dice’s alarm-monitoring/control device.
Dice already teaches that its alarm-monitoring server receives analytics-tag and object-detection information and uses that information to determine and perform security actions. Sanketi teaches that an on-device retrained model can be represented by its retrained model parameters and transmitted from the user device to a server.
A person of ordinary skill would have been motivated to transmit the updated model together with the alarm configuration to Dice’s control-side processing system so that the control system could apply the same newly personalized object-detection model and user-defined alarm rules when processing subsequent premises video. Such modification would predictably maintain consistency between the user-trained object classification and the corresponding alarm actions performed by the control system.
Regarding claim 3, Dice in view of Sanketi and further in view of Zhou, teach the user device of claim 1, wherein the processing circuitry is further configured to:
“cause transmission of the updated machine learning model to another user device of the premises security system for further training the updated machine learning model to generate a further updated machine learning model;”
“responsive to causing transmission of the updated machine learning model to the other user device, receive the further updated machine learning model, the further updated machine learning model being based at least in part on additional feedback indications received on the other user device;”
“based at least in part on the further updated machine learning model.” – Dice teaches this limitation in part. Dice teaches applying an object-detection module to additional digital-video media to detect objects and determine their object types or classes:
“Object detection module 48 is configured to detect objects within the digital video snapshot.” (Dice, pg. 12, ¶[0097])
“Object detection module 48 is configured to transmit one or more object detected parameters along with an identification of the object detected. The one or more object detection parameters may include an object type, an object class, and/or an object location within the digital video snapshot” (Dice, pg. 12, ¶[0097])
Thus, Dice teaches directing an object type from additional media. Dice does not teach performing that detection specifically using a further updated machine-learning model received after additional training on another user device.
Dice does not teach these limitations and/or portions of:
“cause transmission of the updated machine learning model to another user device of the premises security system for further training the updated machine learning model to generate a further updated machine learning model;”
“responsive to causing transmission of the updated machine learning model to the other user device, receive the further updated machine learning model, the further updated machine learning model being based at least in part on additional feedback indications received on the other user device;”
“and detect at least one object type of at least one additional media file ”
Sanketi, however, teaches these limitations and/or portions of:
“cause transmission of the updated machine learning model to another user device of the premises security system for further training the updated machine learning model to generate a further updated machine learning model;” – In Sanketi’s federated-learning implementation, a model and training plan are distributed to user devices, where the model is further trained to generate a model update:
“As illustrated in FIG. 6B, the training plan and model can be distributed by the federated learning service.” (Sanketi, pg. 12, ¶[0166])
“The training process can perform training in the background to generate a model update. The model update can be uploaded to the federated learning service (e.g., for use in aggregation).” (Sanketi, pg. 12, ¶[0166])
“In addition, in some implementations, a quality assurance ( e.g., a semi-automated quality assurance) can extract learned models and distribute the models back to the devices.” (Sanketi, pg. 12, ¶[0166])
Sanketi further explains that federated learning operates across multiple devices:
“After such on device learning, the user device can provide an update to a central authority.” (Sanketi, pg. 8, ¶[0097])
“The central authority can receive many of such updates from multiple devices and can aggregate the updates to generate an updated global model.” (Sanketi, pg. 8, ¶[0098])
“The updated global model can then be re-sent to the user device. This scheme enables cross-device models to be trained and evaluated without centralized data collection.” (Sanketi, pg. 8, ¶[0098])
Thus, Sanketi teaches distributing a machine-learning model among multiple user devices and further training the model using device-specific training data to produce further model updates.
“responsive to causing transmission of the updated machine learning model to the other user device, receive the further updated machine learning model, the further updated machine learning model being based at least in part on additional feedback indications received on the other user device;” – Sanketi teaches that different user devices locally train models using their respective locally generated training data, provide corresponding model updates, and that a central authority aggregates the updates into a further updated global model which is transmitted back to the user device:
“In a third data flow which can be referred to as federated learning, the training data created on the user device is used to train or re-train the model on the device.” (Sanketi, pg. 8, ¶[0096])
“After such on device learning, the user device can provide an update to a central authority.” (Sanketi, pg. 8, ¶[0097])
“The central authority can receive many of such updates from multiple devices and can aggregate the updates to generate an updated global model The updated global model can then be re-sent to the user device.” (Sanketi, pg. 8, ¶[0098])
Likewise:
“The federated server 212 can receive many of such updates from multiple devices and can aggregate the updates to generate an updated global model. The updated global model can then be re-sent to the device 216.” (Sanketi, pg. 8, ¶[0091])
As applied to the Dice-Sanketi combination of claim 1, the locally generated training data includes Dice’s user-defined analytics-tag feedback associated with detected objects. Dice teaches that digital-video frames associated with a user-defined analytics tag are sued to train the object-detection algorithm. Accordingly, when that functionality is implemented across Sanketi’s multiple user devices, the model update contributed by another user device is based at least in part on additional end-user feedback received at that other device. Thus, the aggregated model returned to the first user device constitutes the claimed further updated machine-learning model based at least in part on additional feedback indications received at another user device.
“based at least in part on the further updated machine learning model.” – Sanketi teaches that after retraining, the retrained machine-learning model is used to provide subsequent inferences:
“After retraining of the model 132a-c, the re-trained model 132a-c can be used to provide inferences” (Sanketi, pg. 7, ¶[0077])
“Typically, these inferences will have higher accuracy since the model 132a-c has been re-trained on data that is specific to the user.” (Sanketi, pg. 7, ¶[0077])
Dice teaches that the relevant inference is detection of an object and its object type or class form additional video media. Dice’s object-detection module detects objects from digital-video snapshots and determines an “object type” and “object class”. (Dice, ¶[0097])
Thus, applying Dice’s object-detection functionality using Sanketi’s further updated federated model teaches detecting an object type in an additional media file based on the further updated machine-learning model.
It would have been obvious to one of ordinary skill in the art to configure the Dice-Sanketi system of claim 1 to use Sanketi’s disclosed federated-learning technique to propagate the user-trained object-detection model among multiple user devices for further training.
Dice already teaches improving its object detector using user-defined analytics tags and corresponding video frames. Sanketi teaches a known technique for improving machine-learning models across multiple user devices without centrally collecting the underlying user training data: distribute a model to devices, locally retrain the model using device-specific training data, aggregate the resulting model updates, and redistribute the further updated model. Sanketi expressly states that this scheme enables “cross-device models to be trained and evaluated without centralized data collection”.
A person of ordinary skill would have been motivated to apply Sanketi’s federated-learning technique to Dice’s user-trained surveillance model to improve the model using additional labeled security examples obtained from other user devices while preserving the privacy of the underlying video and user-feedback data. The modification would predictably permit object-detection knowledge learned form additional user feedback at other premises devices to improve subsequent object detection at the first user’s device.
Regarding claim 4, Dice in view of Sanketi and further in view of Zhou, teach the user device of claim 1, wherein the feedback indication comprises at least one of:
“an object type label;” – Dice teaches this limitation. Dice teaches receiving an analytics-tag input through a GUI, where the analytics tag expressly identifies a type or class of object to be detected. Dice provides examples including “vehicle”, “human being”, passenger car, truck, firearms, dogs, and cats:
“The GUI configured to display by alarm monitoring module 36 may be configured to receive an analytics tag input.” (Dice, pg. 8, ¶[0070])
“In one or more embodiments, an analytics tag is a type or class of object configured to be detected within the region of interest.” (Dice, pg. 8, ¶[0070])
“Non-limiting examples of analytics tags include vehicle or human being.” (Dice, pg. 8, ¶[0070])
Dice further teaches that these labels may be user-defined and used as feedback for training the object-detection algorithm, as relied upon with respect to claim 1. Thus, Dice’s user-entered analytics tag constitutes the recited object type label.
Dice therefore teaches at least one of the alternatives recited by claim 4. Because claim 4 requires the feedback indication to comprise “at least one of” the listed alternatives, Dice’s teaching of an object type label is sufficient to satisfy the additional limitation.
“an accuracy indication;”
“and a threat level indicator.”
Because claim 4 requires the feedback indication to comprise at least one of the recited alternatives, Dice’s teaching of an object type label satisfies the additional limitation of claim 4. Sanketi and Zhou are relied upon as set forth with respect to claim 1. No additional teaching from Sanketi or Zhou is necessary for the additional limitation of claim 4.
Regarding claim 5, Dice in view of Sanketi and further in view of Zhou, teach the user device of claim L, wherein the processing circuitry is further configured to:
“receive at least one additional media file from a premises device in the premises security system;” – Dice teaches this limitation. Dice teaches a monitored client site having a client network that obtains digital video from network cameras at the monitored premises and selectively transmits the video for analytics processing:
“Client site 14 can be a site that is monitored for alarm events. Non-limiting examples of client site 14 include vehicle dealerships, construction sites, department stores, personal residences, office buildings, and manufacturing facilities.” (Dice, pg. 3, ¶[0040])
“client network 12 obtains digital video data depicting different views of client site 14 and selectively transmits the digital video data to digital video analytics server 18, which is configured to analyze the digital video data.” (Dice, pg. 3, ¶[0040])
Dice further teaches that the selected regions of interest are associated with a network camera and corresponding video clips and frames:
“The selected regions of interest may be associated with a network camera and digital video clips and frames as set forth above.” (Dice, pg. 8, ¶[0072])
Thus, Dice teaches receiving additional media in the form of digital-video clips or frames from a premises devices, such as a network camera, in the premises-security system.
“identify an additional dominant object in the at least one additional media file;” – Dice teaches this limitation. Dice teaches processing subsequent digital-video clips using an AI object-detection module to detect one or more objects in the video:
“Object detection module 48 may be configured to detect the presence of one or more objects within the one or more of the sequential video frames indicating motion.” (Dice, pg. 9, ¶[0077])
“Object detection module 48 may be configured to determine one or more object detection parameters, such as, without limitation, the type of object, the class of object, the position of the object within each digital video frame, and the identification of the one or more frames containing the object.” (Dice, pg. 9, ¶[0077])
Dice then compares the detected object with active analytics tags to determine whether the object is a tracked object for continued alarm processing:
“gatekeeper server 30 is configured to transmit an alarm signal to alarm monitoring module 36 in response to one or more object detection parameters.” (Dice, pg. 9, ¶[0078])
Thus, under the broadest reasonable interpretation discussed with respect to claim 1, Dice’s detected and tracked object constitutes the claimed additional dominant object.
“detect at least one classified object type of the additional dominant object ” – Dice teaches this limitation in part. Dice expressly teaches determining the type or class of the detected object using an artificial-intelligence object-detection module:
“Object detection module 48 may be a module configured to implement one or more artificial intelligence (AI) algorithms to detect objects in the sequential video frames of the digital video clip.” (Dice, pg. 9, ¶[0077])
“Object detection module 48 may be configured to determine one or more object detection parameters, such as, without limitation, the type of object, the class of object …” (Dice, pg. 9, ¶[0077])
Dice also illustrates detecting an object and displaying the corresponding analytics tag; for example, the detected object may be identified as a “person”.
Thus, Dice teaches detecting a classified object type of the additional dominant object. Dice does not expressly teach that this subsequent detection is performed using the updated machine-learning model resident on the claimed user device.
“receive at least one additional feedback indication associated with the at least one classified object type;” - Dice teaches this limitation. Dice teaches receiving user/operator-selected or user-defined analytics tags corresponding to object types or classes and using those tags in connection with the object-detection system:
“Alarm monitoring module 36 may be configured to receive one or more operator selected analytics tags.” (Dice, pg. 3, ¶[0042])
“Alarm monitoring module 36 may also be configured to configure one or more operator defined analytics tags.” (Dice, pg. 3, ¶[0042])
“For instance, digital video frames associated with a user defined analytics tag may be transmitted to an object detection algorithm to train the object detection algorithm.” (Dice, pg. 3, ¶[0042])
As discussed with respect to claim 1, Dice’s analytics tag constitutes an object-type/class label supplied by the user. Accordingly, receiving another user-selected or user-defined analytics tag associated with a subsequently detected/classified object teaches the recited additional feedback indication.
“” - Dice teaches this limitation in part. Dice expressly teaches training its object-detection algorithm using the combination of a user-defined analytics tag and video frames associated with that tag:
“digital video frames associated with a user defined analytics tag may be transmitted to an object detection algorithm to train the object detection algorithm.” (Dice, pg. 3, ¶[0042])
“if camouflage is not a pre-determined analytics tag, then the object detection algorithm can be trained to detect camouflage in response to analyzing several digital video frames including camouflage.” (Dice, pg. 3-4, ¶[0042])
Thus, Dice teaches training based on both user feedback (the user-defined analytics tag) and corresponding additional media frames. Dice does not expressly tach that the model being further trained is the updated machine-learning model generated on the user device as recited in claim 1.
Dice does not expressly teach these limitations and/or portions of:
“using the updated machine learning model;”
“and further train the updated machine learning model”
Sanketi, however, teaches these limitations and/or portions of:
“using the updated machine learning model;” – Sanketi teaches that after an existing machine-learned model is retrained or updated, the retrained model is subsequently used to make further inferences:
“After retraining of the model 132a-c, the re-trained model 132a-c can be used to provide inferences as described elsewhere herein.” (Sanketi, pg. 7, ¶[0077])
“Typically, these inferences will have higher accuracy since the model 132a-c has been re-trained on data that is specific to the user.” (Sanketi, pg. 7, ¶[0077])
Accordingly, when Sanketi’s on-device retraining framework is used with Dice’s object-detection application as set forth for claim 1, the updated machine-learning model is used on later media to generate Dice’s inference, namely, detection and classification of the additional object type.
“and further train the updated machine learning model” – Sanketi teaches further retraining or updating an existing machine-learned model using additional training examples collected at the user device:
“the applications 120a-c can communicate with the on-device machine learning platform 122 via an API (which may be referred to as the "training API") to cause re-training or updating of a machine-learned model 132a-c based on training examples stored in the centralized example database 124.” (Sanketi, pg. 7, ¶[0076])
“the on-device machine learning platform 122 can run training of the model 132a-c … based on previously collected examples.” (Sanketi, pg. 7, ¶[0076])
“the training can be performed in the background at scheduled times and/or when the device is idle.” (Sanketi, pg. 7, ¶[0076])
Sanketi also teaches continued collection of training examples at the user device:
“the computing device can further include a centralized example database that stores training examples received from the one or more applications.” (Sanketi, pg. 2, ¶[0032])
“the on-device machine learning platform can receive training examples from the applications via an API (which may be referred to as the ‘collection API’)” (Sanketi, pg. 2, ¶[0032])
Thus, Sanketi teaches an iterative workflow in which an existing/retrained model is used for inference, additional training examples are subsequently collected, and that model is again retrained or updated using the collected examples. When applied to Dice, those additional training examples comprise the subsequently received video frames and corresponding user-defined analytics-tag feedback. Dice expressly teaches that those two items are used together to train its object detector (Dice, pg. 3, ¶[0042]).
It would have been obvious to one of ordinary skill in the art to configure the Dice-Sanketi system discussed with respect to claim 1 to continue collecting newly labeled security-video examples and periodically further retain the already updated on-device model.
Regarding claim 6, Dice in view of Sanketi and further in view of Zhou, teach the user device of claim 1, wherein the processing circuitry is further configured to
“cause transmission of the updated machine learning model to a remote server for providing on-device object detection for at least one additional device associated with a user account of the premises security system.” – Dice does not teach this limitation. Sanketi, however, teaches this limitation. Sanketi teaches transmitting an updated machine-learning model to a remote server for distribution to additional account-associated devices for on-device inference. Sanketi teaches publishing model-update messages to a cloud server and, in a federated-learning implementation, uploading a model update to the federated-learning service and distributed learned models back to devices:
“The training process can pick up a model state and training plan, repeatedly pick data from the cache to train the model, and then eventually publish statistic(s) and model update message(s) (e.g., to a cloud server).” (Sanketi, pg. 12, ¶[0165])
Sanketi further teaches that the uploaded model update is generated by retraining the model and that learned models are subsequently distributed back to devices:
“The training process can perform training in the background to generate a model update. The model update can be uploaded to the federated learning service (e.g., for use in aggregation).” (Sanketi, pg. 12, ¶[0166])
Sanketi further teaches tying training data to a primary account, updating the related model, and downloading the account-related model from the cloud when it is not available on a device:
“The on-device platform can tie the training data to the primary account, and update the related model. If the model for the account is not available on the device, the on-device platform can download the model for the client application from cloud or use a base model ( e.g., average model in Federated Leaming).” (Sanketi, pg. 12, ¶[0160])
Thus, when Sanketi’s model-distribution technique is applied to Dice’s premises-security system, the updated object-detection model is transmitted to a remote server and provided to another account-associated premises device for on-device object detection.
It would have been obvious to one of ordinary skill in the art to configure the Dice-Sanketi system discussed with respect to claim 1 to transmit the updated object-detection model to a remote server for distribution to additional account-associated premises devise, thereby enabling those devices to perform the same personalized object detection locally.
Regarding claim 7, Dice in view of Sanketi and further in view of Zhou, teach a premises security system comprising at least one premises device for monitoring a premises, the premises security system comprising:
“a user device associated with an end-user ” - Dice teaches this limitation in part. Dice teaches a subscriber-associated user computer and user mobile device, including a smartphone, connected to the premises-security system:
“Alarm monitoring module 36 is in communication with user computer 38 (e.g. desktop or notebook computer) and user mobile device 40 (e.g. smart phone) via second external communication network 26.” (Dice, pg. 4, ¶[0047])
“User computer 38 and/or user mobile device 40 may be used by a subscriber to access and to execute functionality stored in computer instructions on alarm monitoring module 36 and/or alarm monitoring database 34.” (Dice, pg. 4, ¶[0047])
Dice also teaches an AI-based object-detection algorithm, but Dice does not expressly teach that the pretrained model is stored and trained on the subscriber’s user device.
“receive at least one media file from a media file database ” – Dice teaches this limitation in part. Dice teaches obtaining digital-video frames from an associated digital-video clip stored in digital video database 44:
“gatekeeper server 30 gathers the sequence of digital video frames associated with the motion.” (Dice, pg. 9, ¶[0076])
“The gathering can be accomplished by the receipt of gatekeeper server 30 of the identification of the sequence of digital video frames associated with each motion from motion detection module 46 and obtaining the identified digital video frames from an associated digital video clip, which may be stored in digital video database 44.” (Dice, pg. 9, ¶[0076])
Thus, Dice teaches receiving media from a media-file database, but does not expressly teach that the media-file database is associated with the user device.
“identify, using the pretrained machine learning model, at least one dominant object in the at least one media file;” – Dice teaches this limitation. Dice teaches applying an AI object-detection algorithm to video frames to detect objects and determine the object type/class:
“Object detection module 48 may be a module configured to implement one or more artificial intelligence (AI) algorithms to detect objects in the sequential video frames of the digital video clip.” (Dice, pg. 9, ¶[0077])
“Object detection module 48 may be configured to detect the presence of one or more objects within the one or more of the sequential video frames indicating motion.” (Dice, pg. 9, ¶[0077])
“Object detection module 48 may be configured to determine one or more object detection parameters, such as, without limitation, the type of object, the class of object, the position of the object within each digital video frame, and the identification of the one or more frames containing the object.” (Dice, pg. 9, ¶[0077])
Dice further compares a detected object with active analytics tags to determine whether the object is a tracked object for continued alarm processing. (Dice, ¶[0078]). Thus, under the broadest reasonable interpretation, Dice’s detected and tracked object constitutes the recited dominant object.
“receive an end-user feedback indication associated with the at least one dominant object, the end-user feedback indication comprising at least one label entered by the end-user;” – Dice teaches this limitation. Dice teaches receiving user-selected and user-defined analytics tags and associating those tags with video frames for training an object-detection algorithm:
“Alarm monitoring module 36 may be configured to receive one or more operator selected analytics tags.” (Dice, pg. 3, ¶[0042])
“Alarm monitoring module 36 may also be configured to configure one or more operator defined analytics tags.” (Dice, pg. 3, ¶[0042])
“digital video frames associated with a user defined analytics tag may be transmitted to an object detection algorithm to train the object detection algorithm.” (Dice, pg. 3, ¶[0042])
Dice expressly states:
“the analytics tag can be user selected by the client using functionality on digital video analytics server 18.” (Dice, pg. 4, ¶[0045])
Dice’s analytics tags identify object types/classes, including persons, vehicles, firearms, clothing, and animals. (Dice, ¶¶ [0043]-[0044]). Thus, Dice’s user-selected/user-defined analytics tag constitutes the recited label entered by the end-user.
“manage an alarm configuration comprising a mapping of the at least one label entered by the end-user to a threat level and at least one corresponding premises security action;” – Dice teaches this limitation. Dice teaches associating the user-selected analytics tag with an alarm signal and performing actions based on that association:
“Each analytics tag may be associated with a video alarm signal.” (Dice, pg. 4, ¶[0045])
“The analytics tag may be part of the alarm event data indicative of the digital video alarm event. This association may be made through alarm monitoring module 36.” (Dice, pg. 4, ¶[0045])
Dice further teaches:
“Alarm monitoring module 36 may be configured to perform one or more actions based on the video alarm signal associated with the receipt of data indicative that an alarm event is associated with an analytics tag.” (Dice, pg. 4, ¶[0046])
Dice additionally teaches determining different priorities from object-detection parameters and processing the corresponding alarm signals according to these priorities. For example, a first priority may be higher than a second priority, and the higher-priority processing may include transmission of the alarm signal to an authority server whereas the lower-priority processing does not. (Dice, pg. 20, claims 11-15)
Under the broadest reasonable interpretation, Dice’s security-event priority constitutes the claimed threat level, because the priority reflects the relative security significance of the detected tagged object/event and determines the corresponding security response. Thus, Dice teaches a configuration mapping the user-selected analytics tag to a threat level and corresponding premises-security action.
“” – Dice teaches this limitation in part. Dice expressly teaches training its object-detection algorithm using the combination of user-defined analytics-tag feedback and video frames associated with that feedback:
“digital video frames associated with a user defined analytics tag may be transmitted to an object detection algorithm to train the object detection algorithm.” (Dice, pg. 3, ¶[0042])
“if camouflage is not a pre-determined analytics tag, then the object detection algorithm can be trained to detect camouflage in response to analyzing several digital video frames including camouflage.” (Dice, pg. 3-4, ¶[0042])
Thus, Dice teaches training based on both the end-user feedback indication and the media file. Dice does not expressly teach generating the claimed updated model by on-device retraining of a pretrained model, or training that comprises adding at least one new classifier to the pretrained model.
“” – Dice does not teach this limitation.
“and the control device comprising processing circuitry, the processing circuitry being configured to:” – Dice teaches this limitation. Dice teaches digital video analytics server 18 comprising a computer that executes stored machine instructions:
“a digital video analytics server including a digital video analytics computer having non-transitory memory configured to store machine instructions that are to be executed by the digital video analytics computer.” (Dice, pg. 1, ¶[0003])
Thus, Dice teaches the recited control device comprising processing circuitry.
“” – Dice does not teach this limitation.
“receive or determine an alarm configuration comprising a mapping of at least one classified object type to at least one corresponding premises security action;” – Dice teaches this limitation. Dice teaches receiving alarm-monitoring parameters from alarm-monitoring server 22 and applying those parameters during video analytics. Dice states:
“gatekeeper server 30 receives a digital video alarm monitoring mode from alarm monitoring module 36.” (Dice, pg. 6, ¶[0060])
“In the case of an active monitoring status, one or more digital video alarm monitoring parameters may also be received by gatekeeper server 30 from alarm monitoring module 36.” (Dice, pg. 6, ¶[0060])
Dice further teaches that gatekeeper server 30 compares the detected object type to active analytics tags and determines whether to transmit an alarm signal:
“gatekeeper server 30 compares the type of object with one or more active analytics tags to determine the presence of a tracked object.” (Dice, pg. 9, ¶[0078])
“if the region of interest is a doorway and a human is an analytics tag associated with the doorway, then an alarm signal is transmitted if the type of object is a human and the location of the human intersects with the doorway.” (Dice, pg. 9, ¶[0078])
“an alarm signal is not transmitted if the type of object is an animal or other non-human object type even if the location of the detected object intersects with the doorway.” (Dice, pg. 9, ¶[0078])
Thus, Dice teaches an alarm configuration mapping a classified object type to a corresponding premises-security action, e.g., transmitting or suppressing an alarm signal.
“receive at least one media file from the at least one premises device;” – Dice teaches this limitation. Dice teaches receiving digital-video clips originating from premises network cameras. Its claimed system expressly recites:
“receiving the digital video clip associated with the region of interest from a network camera device on a client network.” (Dice, pg. 19, claim 3)
Dice’s specification likewise teaches that client network 12 obtains video from the monitored client site and transmits that video to digital-video analytics server 18. (Dice, pg. 3, ¶[0040])
“detect at least one classified object type based at least in part on the media file ” – Dice teaches this limitation in part. Dice teaches detecting a classified object type from received video using an AI object-detection algorithm:
“Object detection module 48 may be configured to determine one or more object detection parameters, such as, without limitation, the type of object, the class of object” (Dice, pg. 9, ¶[0077])
Dice therefore teaches detecting the classified object type based on the media file. Dice does not expressly teach that the object detection is performed using the model previously updated on the user device.
“and perform at least one premises security action based on the alarm configuration and the detected at least one classified object type.” – Dice teaches this limitation. Dice teaches that the detected object type is compared against the configured analytics tag and that the resulting security action depends on that comparison:
“if the region of interest is a doorway and a human is an analytics tag associated with the doorway, then an alarm signal is transmitted if the type of object is a human and the location of the human intersects with the doorway.” (Dice, pg. 9, ¶[0078])
Conversely, the alarm signal is suppressed when the detected object type does not correspond to the configured analytics tag.
Thus, Dice teaches performing a premises-security action based on both the alarm configuration and the detected classified object type.
Dice does not teach these limitations and/or portions of:
“and configured with a pretrained machine learning model for detecting object types, the user device comprising processing circuitry configured to:”
“associated with the user device;”
“generate an updated machine learning model by … pretrained machine learning model … the training of the pretrained machine learning model comprising adding at least one new classifier to the pretrained machine learning model;”
“and cause transmission of the updated machine learning model to a control device;”
“receive the updated machine learning model from the user device;”
“and the updated machine learning model;”
Sanketi, however, teaches these limitations and/or portions of:
“and configured with a pretrained machine learning model for detecting object types, the user device comprising processing circuitry configured to:” – Sanketi teaches a user computing device, including a smartphone or tablet, having processors, memory, an on-device machine-learning platform, and locally stored machine-learned models:
“The computing device 102 can be any type of computing device including, for example, a desktop, a laptop, a tablet computing device, a smartphone … Thus, in some implementations, the computing device 102 can be a mobile computing device and/or a user computing device.” (Dice, pg. 5, ¶[0057])
“The computing device 102 includes one or more processors 112 and a memory 114.” (Dice, pg. 5, ¶[0058])
Sanketi further teaches that computing device 102 stores and implements one or more machine-learned models 132a-c locally. (Sanketi, ¶¶ [0060]-[0063]).
When Sanketi’s on-device architecture is applied to Dice’s object-detection application, the locally stored machine-learned model is Dice’s pretrained object-detection model.
“associated with the user device;” – Sanketi teaches that the user computing device includes a centralized example database that stores training examples received and managed by the on-device machine-learning platform:
“the computing device 102 can further include a centralized example database 124 that stores training examples received from the applications 120a-c. In particular, the on-device machine learning platform 122 can receive training examples from the applications 120a-c via an API (which may be referred to as the "collection API") and can manage storage of the examples in the centralized example database 124.” (Dice, pg. 6, ¶[0066])
Thus, when Dice’s security-video frames constitute Sanketi’s training examples, the combined system teaches storing and receiving those media examples form a database associated with the user device.
“generate an updated machine learning model by training the pretrained machine learning model ” – Sanketi teaches this limitation in part. Sanketi teaches retraining or updating a locally stored machine-learned model using training examples stored on the user device:
“the applications 120a-c can communicate with the on-device machine learning platform 122 via an API (which may be referred to as the "training API") to cause re-training or updating of a machine-learned model 132a-c based on training examples stored in the centralized example database 124.” (Sanketi, pg. 7, ¶[0076])
“the on-device machine learning platform 122 can run training of the model 132a-c (e.g., by interacting with a machine learning engine 128 to cause training of the model 132a-c by the engine 128) based on previously collected examples.” (Sanketi, pg. 7, ¶[0076])
When applied to Dice, the training examples are the security-video frames associated with Dice’s user-defined analytics-tag feedback. Dice ¶[0042] expressly teaches using those two items together for model training.
“and cause transmission of the updated machine learning model to a control device;” and “receive the updated machine learning model from the user device;” – Sanketi teaches these limitations. After on-device retraining, Sanketi determines an update representing the retrained machine-learning model and transmits it form the user device to a central server:
“the on-device platform 122 can determine an update that describes the parameters of a re-trained machine-learned model 132a-c or changes to the parameters of the machine learned model 132a-c that occurred during the re-training of model 132a-c (e.g., a ‘gradient’).” (Sanketi, pg. 7, ¶[0078])
“The platform 122 can transmit the update to a central server computing device (e.g., ‘the cloud’)…” (Sanketi, pg. 7, ¶[0078])
Sanketi additionally teaches that model artifacts/parameters may be downloaded from a server and executed by a computing device. (Sanketi, pg. 5 and 10, ¶¶ [0062]-[0063], [0124]).
Thus, Sanketi teaches transmission of the trained model state/model parameters from the user device to a central computing device. When applied to Dice, that central computing device is Dice’s control-side digital-video analytics server.
“and the updated machine learning model;” – Sanketi teaches using the retrained model for subsequent inference:
“After retraining of the model 132a-c, the re-trained model 132a-c can be used to provide inferences …” (Sanketi, pg. 7, ¶[0077])
Thus, when Sanketi’s updated/retrained model is used in Dice’s object-detection application, the subsequent inference is Dice’s detection of the object type/class from the received security-video media.
Neither Dice nor Sanketi teaches these remaining limitations and/or portions of:
“… the training of the pretrained machine learning model comprising adding at least one new classifier to the pretrained machine learning model;”
Zhou, however, teaches these remaining limitations and/or portions of:
“… the training of the pretrained machine learning model comprising adding at least one new classifier to the pretrained machine learning model;” – Zhou teaches training an image-classification neural network using a pretrained ImageNet model while adding a new task-specific classifier:
“the former twenty file layers are transferred from the ImageNet pretrained model and are kept fixed during training.” (Zhou, pg. 15, ¶[0060])
“In this case, each task may add a task specific classifier and batch normalization layers, thus keeping the size of the Classifier model small.” (Zhou, pg. 15, ¶[0060])
Zhou further teaches merging newly created and turned classifiers into the maintained neural network:
“parameter optimizer 170 may merge the newly created and tuned parameters in the layers, task specific adapters and classifier(s) in the task specific neural network with the maintained super network
S
.” (Zhou, pg. 11, ¶[0041])
Thus, Zhou teaches training a pretrained model by adding a new classifier.
It would have been obvious to one of ordinary skill in the art to implement Dice’s user-defined analytics-tag training and security-video example collection using Sanketi’s on-device machine-learning architecture, such that the user device locally trains Dice’s object-detection model and transmits the resulting updated model to Dice’s control-side video-analytics device for subsequent object detection and alarm processing. Sanketi expressly teaches that on-device retraining personalizes the models to user-specific data and that the retrained model can thereafter be used for inference. (Sanketi, pg. 7, ¶¶ [0076]-[0077])
It would have been obvious to one of ordinary skill in the art to further modify the Dice-Sanketi system to use Zhou’s task-specific classifier addition when the end-user defines a new analytics tag representing an object type or class not previously recognized by the pretrained object-detection model.
Regarding claim 8, Dice in view of Sanketi and further in view of Zhou, teach the premises security system of Claim 7, wherein the processing circuitry of the control device is further configured to:
“” – Dice does not teach this limitation.
“” – Dice does not teach this limitation.
“detect at least one additional classified object type of at least one additional media file ” – Dice teaches this limitation in part. Dice teaches AI-based detection of object types and classes from video frames:
“Object detection module 48 may be configured to determine one or more object detection parameters, such as, without limitation, the type of object, the class of object” (Dice, pg. 9, ¶[0077])
Thus, Dice teaches detecting an additional classified object type from additional media, but does not teach using the claimed further updated machine learning model.
“and perform at least one premises security action based on the at least one additional classified object type and the security configuration.” – Dice teaches this limitation. Dice teaches comparing a detected object type with an active analytics tag and transmitting or suppressing an alarm signal based on the configured association:
“gatekeeper server 30 compares the type of object with one or more active analytics tags to determine the presence of a tracked object.” (Dice, pg. 9, ¶[0078])
and, for example, transmits an alarm signal when a detected human corresponds to the configured human analytics tag.
“if the region of interest is a doorway and a human is an analytics tag associated with the doorway, then an alarm signal is transmitted if the type of object is a human and the location of the human intersects with the doorway.” (Dice, pg. 9, ¶[0078])
Thus, Dice teaches performing a premises security action based on the detected classified object type and the security configuration.
Dice does not teach these limitations and/or portions of:
“cause transmission of the updated machine learning model to another user device of the premises security system for further training the updated machine learning model to generate a further updated machine learning model;”
“responsive to causing transmission of the updated machine learning model to the other user device, receive the further updated machine learning model, the further updated machine learning model being based at least in part on additional feedback indications received on the other user device;”
“based on the further updated machine-learning model;”
Sanketi, however, teaches these remaining limitations and/or portions of:
“cause transmission of the updated machine learning model to another user device of the premises security system for further training the updated machine learning model to generate a further updated machine learning model;” – Sanketi teaches distributing a model to devices in a federated-learning system and training the models at the devices:
“the training plan and model can be distributed by the federated learning service. The training process can perform training in the background to generate a model update.” (Sanketi, pg. 12, ¶[0166])
Thus, when applied to the Dice-Sanketi system of claim 7, the control device distributes the updated model to another user device for further training.
“responsive to causing transmission of the updated machine learning model to the other user device, receive the further updated machine learning model, the further updated machine learning model being based at least in part on additional feedback indications received on the other user device;” – Sanketi teaches locally retraining the model using training data created on a user device and returning the resulting model update to a central authority:
“the training data created on the user device is used to train or re-train the model on the device.” (Sanketi, pg. 8, ¶[0096])
“After such on device learning, the user device can provide an update to a central authority.” (Sanketi, pg. 8, ¶[0097])
Sanketi further teaches that the central authority receives updates from multiple devices and generates an updated model (Sanketi, pg. 8, ¶[0098]).
When applied to Dice, the additional training data includes Dice’s user-defined analytics-tag feedback and associated video frames, which Dice expressly uses together to train its object detector. (Dice, pg. 3, ¶[0042])
Thus, the combination teaches receiving a further updated model based at least in part on additional feedback indications received on the other user device.
“based on the further updated machine-learning model;” – Sanketi teaches use of the further trained model for subsequent inference:
“After retraining of the model 132a-c, the re-trained model 132a-c can be used to provide inferences” (Sanketi, pg. 7, ¶[0077])
Thus, when Sanketi’s further updated model is applied to Dice’s object-detection functionality, the further updated model is used to detect an additional classified object type from additional video media.
It would have been obvious to one of ordinary skill in the art to configure the Dice-Sanketi system discussed with respect to claim 7 to distribute the updated object-detection model to another user device for further training and use the resulting further updated model for subsequent premises-security object detection.
Zhou is relied upon as set forth with respect to claim 7 for the underlying requirement that training the pretrained machine learning model comprises adding at least one new classifier.
Regarding claim 9, Dice in view of Sanketi and further in view of Zhou, teach the premises security system of Claim 7, wherein the processing circuitry of the control device is further configured to:
“cause transmission of ” – Dice teaches this limitation in part. Dice teaches a premises device in the form of a network camera having a processor and memory that stores local video data and performs digital video analysis:
“a network camera includes … a processor, and memory. The memory is configured to store firmware and video data (e.g., video frames and video sequence recordings) … [and] functions may include … digital video processing functions and digital video analysis functions.” (Dice, pg. 4, ¶[0049])
Thus, Dice teaches a premises device capable of analyzing locally stored media, but does not teach transmission of the updated machine learning model to the premises device for performing the object detection.
“responsive to causing transmission ” – Dice teaches this limitation in part. Dice teaches that its object-detection functionality generates and transmits an indication identifying the detected object type or class:
“Object detection module 48 may be configured to determine one or more object detection parameters, such as, without limitation, the type of object, the class of object … [and] transmit one or more of the one or more object detection parameters to gatekeeper server 30.” (Dice, pg. 9, ¶[0077])
Thus, Dice teaches receiving a detection indication corresponding to a classified object type, but does not teach that the indication results from a premises device operating the transmitted updated model on locally stored media.
“and perform at least one premises security action based on the at least one additional classified object type and the security configuration.” – Dice teaches this limitation. Dice teaches transmitting or suppressing an alarm according to the detected object type and the configured analytics tag:
“if the region of interest is a doorway and a human is an analytics tag associated with the doorway, then an alarm signal is transmitted if the type of object is a human …” (Dice, pg. 9, ¶[0078])
Thus, Dice teaches performing a premises security action based on the classified object type and the security configuration.
Dice does not teach these limitations and/or portions of:
“the updated machine learning model”
“of the updated machine learning model to the premises device”
Sanketi, however, teaches these remaining limitations and/or portions of:
“the updated machine learning model” – Sanketi teaches distributing a machine-learning model to a device and performing inference locally on the device:
“The cloud server 210 provides the inference plan 206 and the training plan 208 to a device 214.” (Sanketi, pg. 8, ¶[0087])
“The device 214 can implement the inference plan 206 to generate inferences.” (Sanketi, pg. 8, ¶[0088])
Sanketi further teaches that model parameters may be downloaded to and cached on the device for subsequent predictions:
“In addition, the platform 122 can cache the content ( e.g., within the repository 130) so that it can be used for subsequent prediction requests.” (Sanketi, pg. 5, ¶[0063]).
Thus, when applied to Dice, the updated object-detection model is transmitted to Dice’s processor-equipped premises camera and used to detect object types from the locally stored video identified in Dice, pg. 4, ¶[0049])
“of the updated machine learning model to the premises device” – Sanketi teaches locally executing the downloaded model to obtain an inference:
“given a uniform resource identifier (URI) for a prediction plan ( e.g., instructions for running the model to obtain inferences/predictions) and model parameters, the on-device machine learning platform 122 can download the URI content ( e.g., prediction plan and parameters) and obtain one or more inferences/predictions” (Sanketi, pg. 5, ¶[0063])
When applied to Dice’s object-detection functionality, the resulting inference is the detected object type/class, which Dice teaches transmitting as an object-detection parameter to the control-side processing device. (Dice, pg. 9, ¶[0077]). Thus, the combination teaches receiving a detection indication corresponding to the object type identified by the premises device using the updated model.
It would have been obvious to one of ordinary skill in the art to configure the Dice-Sanketi system discussed with respect to claim 7 to provide the updated object-detection model to Dice’s processor-equipped premises camera so that locally stored video may be analyzed on the premises device and the resulting object detection returned for alarm processing.
Regarding claim 10, Dice in view of Sanketi and further in view of Zhou, teach the premises security system of Claim 7, wherein the feedback indication comprises at least one of:
“an object type label;” – Dice teaches this limitation. Dice teaches receiving an analytics-tag input, wherein the analytics tag identifies the type or class of object to be detected:
“The GUI configured to display by alarm monitoring module 36 may be configured to receive an analytics tag input … In one or more embodiments, an analytics tag is a type or class of object configured to be detected within the region of interest.” (Dice, pg. 8, ¶[0070])
Thus, Dice’s analytics tag constitutes the recited object type label.
“an accuracy indication;” and “and a threat level indicator.” – Because claim 10 requires the feedback indication to comprise at least one of the recited alternatives, Dice’s teaching of an object type label satisfies the additional limitation of claim 10.
Regarding claim 11, Dice in view of Sanketi and further in view of Zhou, teach the premises security system of Claim 10, wherein the processing circuitry of the control device is further configured to
“determine or modify the security configuration based least in part based on the threat level indication associated with at least one classified object type.” – Dice teaches this limitation. Dice teaches determining a priority based on object-detection parameters, including the detected object type, and configuring the corresponding alarm response according to that priority:
“Alarm monitoring module 36 may be configured to determine a priority in response to one or more object detection parameters of a digital video clip.” (Dice, pg. 10, ¶[0082])
“The higher priority action may be transmitting the digital video alarm signal to an authority server. The authority server may be a police server, a fire department server, a government authority server, a private security server, and/or another alarm server. The lower priority action may not include transmitting the digital video alarm signal to any authority server.” (Dice, pg. 10, ¶[0082])
Dice further explains that:
“the presence of a certain object type within a certain region of interest may have a higher priority than finding a different object type within the same region of interest.” (Dice, pg. 10, ¶[0082])
Thus, under the broadest reasonable interpretation of Dice’s priority as the claimed threat level indication, Dice teaches determining or modifying the security configuration based at least in part on a threat level associated with a classified object type.
Regarding claim 12, Dice in view of Sanketi and further in view of Zhou, teach the premises security system of Claim 7, wherein the processing circuitry of the control device is further configured to:
“receive at least one additional media file from a premises device in the premises security system;” – Dice teaches this limitation. Dice teaches receiving digital video from a network camera device:
“receiving the digital video clip associated with the region of interest from a network camera device on a client network.” (Dice, pg. 19, claim 3)
Thus, Dice teaches receiving an additional media file from a premises device.
“identify an additional dominant object in the at least one additional media file;” – Dice teaches this limitation. Dice teaches detecting objects within video frames:
“Object detection module 48 may be configured to detect the presence of one or more objects within the one or more of the sequential video frames indicating motion.” (Dice, pg. 9, ¶[0077])
Thus, under the broadest reasonable interpretation discussed with respect to claim 7, Dice teaches identifying an additional dominant object in the additional media file.
“detect at least one additional classified object type of the additional dominant object ” – Dice teaches this limitation in part. Dice teaches determining the type or class of the detected object:
“Object detection module 48 may be configured to determine one or more object detection parameters, such as, without limitation, the type of object, the class of object” (Dice, pg. 9, ¶[0077])
Thus, Dice teaches detecting an additional classified object type, but does not expressly teach performing the detection using the previously updated machine learning model.
“receive, from at least one user device in the premises security system, at least one additional feedback indication associated with the at least one additional classified object type;” – Dice teaches this limitation. Dice teaches displaying its alarm-monitoring GUI through a user computer or user mobile device:
“Alarm monitoring module 36 may be configured to display GUI 600 through user computer 38 and/or user mobile device 40.” (Dice, pg. 14, ¶[0113])
Dice further teaches receiving an object-type/class analytics-tag input through the GUI:
“The GUI configured to display by alarm monitoring module 36 may be configured to receive an analytics tag input … an analytics tag is a type or class of object configured to be detected within the region of interest.” (Dice, pg. 8, ¶[0070])
Thus, Dice teaches receiving, from a user device, additional feedback associated with a classified object type.
“” – Dice teaches this limitation in part. Dice teaches using user-defined analytics-tag feedback to train its object-detection algorithm:
“digital video frames associated with a user defined analytics tag may be transmitted to an object detection algorithm to train the object detection algorithm.” (Dice, pg. 3, ¶[0042])
Thus, Dice teaches training based on user-defined feedback, but does not expressly teach that the model being further trained is the previously updated machine learning model.
Dice does not teach these limitations and/or portions of:
“using the updated machine learning model;”
“and further train the updated machine learning model”
Sanketi, however, teaches these remaining limitations and/or portions of:
“using the updated machine learning model;” – Sanketi teaches using a retrained model for subsequent inference:
“After retraining of the model 132a-c, the re-trained model 132a-c can be used to provide inferences” (Sanketi, pg. 7, ¶[0077])
Thus, when applied to Dice’s object-detection functionality, the updated model is used to detect the additional classified object type.
“and further train the updated machine learning model” – Sanketi teaches further retraining or updating an existing machine-learning model:
“the applications 120a-c can communicate with the on-device machine learning platform 122 … to cause re-training or updating of a machine-learned model 132a-c based on training examples stored in the centralized example database 124.” (Sanketi, pg. 7, ¶[0076])
When applied to Dice, the additional training examples are associated with Dice’s additional user-defined analytics-tag feedback, which Dice expressly uses to train its object detector. (Dice, pg. 3, ¶[0042])
It would have been obvious to one of ordinary skill in the art to configure the Dice-Sanketi system discussed with respect to claim 7 to continue receiving additional user-labeled security examples and further retrain the already updated object-detection model using the additional feedback.
Regarding claim 13, Dice in view of Sanketi and further in view of Zhou, teach the premises security system of Claim 7, wherein the processing circuitry of the control device is further configured to
“cause transmission of ” – Dice teaches this limitation in part. Dice teaches additional premises devises associated with a subscriber account:
“A zone may also be assigned to a subscriber account associated with one or more client sites.” (Dice, pg. 4, ¶[0048])
and teaches that the premises network cameras include processors and memory capable of performing:
“digital video processing functions and digital video analysis functions” (Dice, pg. 4, ¶[0049])
Thus, Dice teaches additional account-associated premises devices capable of local video analysis, but does not expressly teach transmitting the updated machine learning model to a remote server for providing the model to such devices.
Dice does not teach these limitations and/or portions of:
“the updated machine learning model to a remote server for providing on-device object detection”
Sanketi, however, teaches these remaining limitations and/or portions of:
“the updated machine learning model to a remote server for providing on-device object detection” – Sanketi teaches transmitting a model update to a remote cloud server and distributing learned models back to devices:
“The model update can be uploaded to the federated learning service (e.g., for use in aggregation). In addition, in some implementations, a quality assurance (e.g., a semi-automated quality assurance) can extract learned models and distribute the models back to the devices.” (Sanketi, pg. 12, ¶[0166])
Sanketi further teaches that model artifacts generated in the cloud may be:
“shipped to devices” (Sanketi, pg. 4, ¶[0047])
Thus, when Sanketi’s model-distribution technique is applied to Dice’s object-detection system, the uploaded object-detection model is transmitted to a remote server and provided to an additional account-associated premises device for on-device object detection.
It would have been obvious to one or ordinary skill in the art to configure the Dice-Sanketi system discussed with respect to claim 7 to transmit the updated object-detection model to a remote server for distribution to additional account-associated premises devices, thereby enabling those devices to perform the same personalized object detection locally.
Regarding claims 14-20
Claims 14-20 recite substantially the same limitations already analyzed above with respect to corresponding system claims 7-13, respectively, but recast in method form. Accordingly, the teachings of Dice, Sanketi, and Zhou identified above with respect to claims 7-13 are equally applicable to the corresponding limitations of claims 14-20, as discussed below.
Regarding claim 14, Dice in view of Sanketi and further in view of Zhou teach the method of claim 14 for substantially the same reasons set forth above with respect to claim 7. The user-device and control-device steps correspond to the respective limitations of claim 7, including receiving media, identifying a dominant object, receiving a label, managing an alarm configuration, generating an updated model, receiving the updated model, detecting a classified object type, and performing a security action. Accordingly, the combination teaches or suggests each limitation of claim 14 for the reasons previously set forth with respect to claim 7.
Regarding claim 15, the additional method steps correspond to claim 8 and are taught or suggested by Dice and Sanketi for the reasons set forth above with respect to claim 8. In particular, Sanketi teaches distributing an updated model for further training at another device, receiving a further updated model, and using a retrained model for inference, thereby teaching the recited further-training and subsequent use limitations.
Regarding claim 16, the additional method steps correspond to claim 9 and are taught or suggested by Dice and Sanketi for the reasons set forth above with respect to claim 9. Sanketi teaches providing a model for local inference, and Dice teaches premises devices capable of local video analysis and control-side processing, thereby teaching transmitting the model to a premises device, performing local detection, and returning a detection indication.
Regarding claim 17, the additional limitation corresponds to claim 10 and is taught by Dice for the reasons set forth above with respect to claim 10. Dice’s analytics tag constitutes the recited object type label, satisfying the “at least one of” requirement.
Regarding claim 18, the additional limitation corresponds to claim 11 and is taught by Dice for the reasons set forth above with respect to claim 11. Dice teaches determining security priority based on object type and using that priority to control alarm behavior, which corresponds to the recited threat level indication and its use in modifying the security configuration.
Regarding claim 19, the additional method steps correspond to claim 12 and are taught or suggested by Dice and Sanketi for the reasons set forth above with respect to claim 12. The combination teaches receiving additional media, identifying an additional object and object type, receiving additional feedback, and further training the model based on that feedback.
Regarding claim 20, the additional method step corresponds to claim 13 and is taught or suggested by Dice and Sanketi for the reasons set forth above with respect to claim 13. Dice teaches additional account-associated premises devices capable of local video analysis, and Sanketi teaches transmitting an updated model to a remote server and distributing it for on-device inference, thereby teaching providing the model to additional devices for on-device object detection.
Accordingly, Dice in view of Sanketi and further in view of Zhou teaches or suggests all limitations of claims 14-20.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Paul Coleman whose telephone number is (571)272-4687. The examiner can normally be reached Mon-Fri.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PAUL COLEMAN/ Examiner, Art Unit 2126
/DAVID YI/ Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights