Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-7, 12-13, 15-16 and 18-20 is rejected under 35 U.S.C. 103 as being unpatentable over Carranza et al. (US 2019/0043207 A1) in view of Momeyer et al. (US 2012/0033876 A1).
Regarding claim 1, 18 and 20, Carranza teaches one or more computers configured to perform operations by teaching a surveillance system including a surveillance orchestration device having one or more processors, memory elements, communication interfaces, and data storage, where the data storage contains camera orchestration logic including “logic and/or instructions that can be executed by a processor” (paras 0047-0048). Carranza further teaches smart cameras having processors, memory elements, communication interfaces, and vision sensors (para 0049).
Carranza teaches determining whether sensor data maintained by a battery-powered device for a property is to be processed by the battery-powered device or by another device for the property by teaching a surveillance system used for “security and surveillance,” “smart homes,” “smart buildings,” and IoT applications (para 0035), wherein the system includes smart cameras and other edge resources in a local area network (paras 0037-0043). Carranza teaches that the smart cameras include vision sensors used to generate video streams associated with the environment in which the smart camera is deployed, and that a smart camera may “process video streams using its own processor” and/or “transmit video streams and/or associated metadata” to another component, such as the surveillance orchestration device, “for processing and/or storage purposes” (paras 0047-0049). Carranza further teaches IoT devices including “locks, cameras, alarms, motion sensors, and the like” in home automation systems (para 0088), and teaches that an IoT device may be battery-powered and may include a battery monitor/charger for tracking the state of charge of the battery, where battery parameters may be used to determine actions performed by the IoT device (paras 0122-0123).
Carranza does not expressly teach determining whether the sensor data should be processed by the battery-powered device or another device based on processing/resource constraints. Momeyer teaches determining whether captured image data should be processed locally, shared between local and remote processing, or remotely. Momeyer teaches determining capacity to collaborate with a remote network, testing bandwidth, determining device performance constraints such as CPU speed/availability and DSP hardware/software configuration, accessing user preferences or bandwidth cost, and considering power limitations based on either “the power consumption for locally performing the image processing” or “the power required to transmit varying amounts of image data” (para 0057). Momeyer further teaches that possible modes are determined based on what portions of digital image processing can be performed locally or remotely, and that a selection is made for an optimum solution based on user preferences, traffic optimization, or reducing time to complete digital image processing (para 0059). Momeyer also teaches accessing a lookup table “to decide device versus server distributed image processing,” including a local processing mode, a shared processing mode, and a remote processing mode (paras 0060-0062).
Therefore, Carranza in view of Momeyer teaches using a result of determining whether the sensor data should be processed by the battery-powered device or by the other device for the property because Momeyer’s selected local, shared, or remote processing mode is used to determine whether image/video data is processed locally by the device, partially processed locally and remotely, or sent to another device/server for processing (paras 0059-0062).
Carranza in view of Momeyer teaches sending an instruction configured to cause the battery-powered device to process the sensor data, or send the sensor data to the other device for the property for processing because Carranza teaches that the surveillance system may proactively configure cameras and may use metadata processor insights “to retro-feed certain cameras via the control plane with tailored configuration settings” (para 0055), and teaches configuring cameras to capture, detect, and/or identify objects under optimal conditions (paras 0039, 0055, 0081). Momeyer teaches that, when local processing mode is selected, local image processing is performed, and when remote processing mode is selected, “the captured clip is sent to the server” rather than processing locally (paras 0060-0062).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Carranza’s surveillance/IoT camera system to include Momeyer’s determination of whether captured image/video data should be processed locally, shared, or remotely based on device performance, bandwidth, power, and processing constraints, in order to improve processing efficiency, conserve battery/power resources of battery-powered cameras/IoT devices, reduce bandwidth usage, and allocate image/video processing to an appropriate available processing device.
Regarding claim 2 and 19, Carranza further teaches in response to determining that another device should process the sensor data, selecting, from a plurality of other devices that can process the sensor data and that do not include the battery-powered device, the other device that should process the sensor data by teaching a surveillance/IoT system including multiple edge devices, cameras, edge processing devices, gateways, cloud resources, IoT devices, data aggregators, and sensors (paras 0037-0043, 0099-0105). Carranza teaches that the fog/IoT network includes gateways, data aggregators, and sensors, where “the data aggregators 926 may collect data from any number of the sensors 928, and perform the back-end processing function for the analysis,” and that sensors may collect data and allow “the data aggregators 926 or gateways 904 to process the data” (para 0101). Carranza further teaches that IoT devices forming a fog device may “determine needed resources in response to conditions, queries, and device failures,” and that the fog device may select particular IoT devices, with data then being “aggregated and analyzed by any combination of the sensors 928, data aggregators 926, or gateways 904” (para 0104). Thus, Carranza teaches selecting from a plurality of available processing devices, such as sensors, data aggregators, and gateways, that can process sensor data.
Carranza in view of Momeyer further teaches sending a second instruction to the other device that was selected from the plurality of other devices and to cause the other device to process the sensor data because Carranza teaches that the surveillance system may proactively configure devices and may use metadata processor insights “to retro-feed certain cameras via the control plane with tailored configuration settings” (para 0055), and further teaches configuring cameras/devices to capture, detect, and/or identify objects under optimal conditions (paras 0039, 0055, 0081). Momeyer further teaches selecting local, shared, or remote processing modes, where a local/remote component is determined to be available for shared processing and the result is sent to the server, or, in remote processing mode, “the captured clip is sent to the server” for processing (paras 0060-0062).
Regarding claim 3, Carranza further teaches each device from the plurality of other devices executes a property security application that can analyze sensor data from a plurality of different types of sensors by teaching that the disclosed system may be used for “security and surveillance,” “smart homes,” “smart buildings,” and IoT applications (para 0035). Carranza teaches that cloud resources may include “security services,” “IoT application and management services,” “data storage,” and “computational services,” including “data analytics” (para 0042). Carranza further teaches IoT/home automation devices including “locks, cameras, alarms, motion sensors, and the like” (para 0088), and teaches that IoT devices can include external sensors such as “optical light sensors, camera sensors, temperature sensors,” and other sensors (para 0120). Carranza also teaches that the data from sensors may be “aggregated and analyzed by any combination of the sensors 928, data aggregators 926, or gateways 904” (para 0104). Thus, Carranza teaches other devices executing security/surveillance or IoT application functionality that can analyze sensor data from different types of sensors.
Carranza in view of Momeyer teaches selecting the other device from the plurality of other devices uses data indicating, for at least some devices from the plurality of other devices, one or more attributes of the respective other device because Carranza teaches that IoT devices forming a fog device may “determine needed resources in response to conditions, queries, and device failures” (para 0104). Carranza further teaches an IoT device with a battery monitor/charger that tracks state of charge and communicates battery information to the processor, where battery parameters may be used to determine actions performed by the IoT device, such as “transmission frequency, mesh network operation, sensing frequency, and the like” (para 0123). Momeyer further teaches determining device and network attributes, including testing bandwidth, determining device performance constraints such as CPU speed/availability and DSP hardware/software configuration, accessing bandwidth cost, and considering power limitations based on local image-processing power consumption or power required to transmit image data (para 0057). Momeyer further teaches selecting local, shared, or remote processing modes based on such capacity and performance attributes (paras 0059-0062).
Carranza in view of Momeyer further teaches sending the second instruction causes the respective property security application for the other device to process the sensor data because Carranza teaches that the surveillance system may use metadata processor insights “to retro-feed certain cameras via the control plane with tailored configuration settings” (para 0055), and teaches that the surveillance system configures cameras to capture, detect, and/or identify objects under optimal conditions (paras 0039, 0055, 0081). Momeyer teaches that, when a shared or remote processing mode is selected, image data or processing results are sent to a server for additional processing (paras 0060-0062). Therefore, the combined system teaches sending an instruction to the selected other device to cause the selected device’s security/surveillance application functionality to process the sensor data.
Regarding claim 4, Carranza further teaches maintaining, in memory, data that indicates an accuracy of at least some devices from the plurality of other devices by teaching that the surveillance orchestration device includes memory elements and data storage containing a camera layout map and object behavioral data, wherein the camera layout map identifies the layout of the respective cameras in the surveillance system, including the position and orientation of each camera, and is used to facilitate camera orchestration decisions (para 0048). Carranza further teaches that the system uses metadata to predict a future state of an object and identify the best camera to capture the object under optimal conditions, thereby improving the chances of successfully identifying the object using computer vision algorithms, such as facial recognition (paras 0021, 0024-0026, 0048, 0060-0063, 0079-0082).
Carranza further teaches selecting the other device from the plurality of other devices using at least some of the data that indicates the accuracy of at least some devices from the plurality of other devices by teaching that, based on an object’s predicted future state, the surveillance system identifies a second camera having the best perspective or view for capturing the object under optimal conditions at a predicted future location and time, and configures that camera to capture, detect, and/or identify the object (paras 0021, 0060-0063, 0079-0082). Carranza teaches that this selection improves efficiency and accuracy by selecting cameras that are more likely to successfully capture and identify the object, rather than processing each camera stream independently (paras 0019-0021).
Momeyer further teaches selecting a processing mode based on device/network capability and processing constraints, including determining device performance constraints, bandwidth, cost, power limitations, and selecting an optimum local/shared/remote processing solution based on those constraints (paras 0057-0062). Thus, Carranza in view of Momeyer teaches selecting the other processing device using data indicating expected processing/capture accuracy and device capability.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Carranza’s surveillance/IoT camera system to include Momeyer’s device-capability-based processing selection so that the system selects a processing device using stored data indicating which device is expected to more accurately and efficiently process the sensor data, in order to improve object detection/identification accuracy, reduce unnecessary processing, and allocate image/video processing to a device better suited for the particular processing task.
Regarding claim 5, Carranza in view of Momeyer further teaches determining, for the sensor data maintained by the battery-powered device, whether the battery-powered device or the device should process the sensor data uses one or more of a transmission resource cost, a type of the sensor data, an identifier for the battery-powered device, calibration data for the battery-powered device, a potential event represented by the sensor data, a requested task for processing the sensor data, or a time period during which the sensor data was captured.
Momeyer teaches determining whether captured image data should be processed locally, shared, or remotely using a transmission resource cost by teaching determining capacity to collaborate with a remote network, testing bandwidth, determining device performance constraints, accessing “cost to use bandwidth,” and considering power limitations based on either “the power consumption for locally performing the image processing” or “the power required to transmit varying amounts of image data” (para 0057). Momeyer further teaches selecting an optimum local/remote processing solution based on user preferences, traffic optimization, or reducing the time to complete the digital image processing (para 0059). Momeyer also teaches accessing a lookup table “to decide device versus server distributed image processing,” including local processing, shared processing, and remote processing modes (paras 0060-0062).
Carranza further teaches using a type of sensor data, a potential event represented by the sensor data, and a time period during which the sensor data was captured by teaching that metadata generated from video streams may indicate object type, identity, physical characteristics, behavioral characteristics, orientation, distance from cameras, position, speed, direction of travel, path, and trajectory (paras 0021-0024, 0074-0077). Carranza further teaches that a future time and position of the object can be predicted, and that a camera with the best perspective of the predicted future position can be configured to capture the object at the predicted time using optimal camera settings (paras 0021-0024).
Carranza also teaches a requested task for processing the sensor data by teaching processing video streams using appropriate computer vision algorithms for particular use cases, including object detection, facial recognition, license plate recognition, object tracking, and object identification (paras 0054-0056, 0074-0082). Carranza further teaches that the surveillance system may decide to selectively forego, avoid, and/or disable processing associated with remaining cameras based on a prediction that a selected camera would be successful, thereby conserving computing resources (paras 0063-0067).
Regarding claim 6, Carranza further teaches in response to determining that another device should process the sensor data, determining a time period during which the other device should process the sensor data by teaching that metadata from a first camera is used to predict a future state of an object, including a future time and position of the object, and that the camera with the best perspective of the predicted future position is configured to subsequently capture the object at the predicted time using optimal camera settings (paras 0021, 0024). Carranza further teaches predicting the location/path of the object at “various moments in the future,” identifying a second camera for capturing the object at a “second point in time,” and configuring the second camera to capture, detect, and/or identify the object at the “second point in time” using tailored camera settings (paras 0076-0081).
Carranza further teaches sending, to the other device, a second instruction that indicates the time period and causes the other device to process the sensor data during the time period by teaching that the metadata processor evaluates metadata, generates predictions, and proactively configures surveillance cameras to subsequently capture, detect, and/or identify objects under optimal conditions, and that the predictions may be used to “retro-feed certain cameras via the control plane with tailored configuration settings” (paras 0055-0056). Carranza also teaches configuring the second camera to capture the object at the second point in time and then tracking and/or identifying the object based on the second video stream (paras 0079-0082).
Momeyer further teaches that when remote or shared processing is selected, image data or partial processing results are sent to another device/server for processing, including a shared processing mode in which a result is sent to the server and a remote processing mode in which “the captured clip is sent to the server” for processing (paras 0060-0062).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Carranza’s surveillance/IoT camera system to include Momeyer’s local/shared/remote processing selection so that, when another device is selected to process sensor/video data, the system sends an instruction indicating the predicted time period for processing the data, in order to coordinate camera/device processing with predicted object movement, improve object detection/identification accuracy, and conserve processing resources.
Regarding claim 7, Carranza further teaches in response to determining that another device should process the sensor data, selecting, from a plurality of processing operations, a processing operation the other device should execute to process the sensor data by teaching that video frames may be processed using appropriate computer vision algorithms required for the particular use case, including “facial recognition, license plate recognition, object tracking, and so forth” (para 0054). Carranza further teaches detecting an object in a first video stream using object detection algorithms, attempting to identify the object using an object identification algorithm, such as facial recognition or license plate recognition, and tracking and/or identifying the object based on a second video stream (paras 0074-0082).
Momeyer further teaches selecting from a plurality of processing operations for image processing by teaching local processing, shared processing, and remote processing modes. In local processing mode, Momeyer teaches performing “histogram edge detection & Scale Invariant Feature Transform (SIFT)” and sending a feature vector to the server; in shared processing mode, Momeyer teaches performing “histogram edge detection” and sending the result to the server; and in remote processing mode, Momeyer teaches sending the captured clip to the server for processing (paras 0060-0062).
Carranza further teaches sending, to the other device, a second instruction that indicates the processing operation and causes the other device to perform the processing operation on at least a portion of the sensor data by teaching that the metadata processor may proactively configure surveillance cameras and may use predictions to “retro-feed certain cameras via the control plane with tailored configuration settings” for capturing appropriate information from the area covered by the cameras (para 0055). Carranza further teaches configuring cameras to capture, detect, and/or identify objects under optimal conditions (paras 0055-0056, 0079-0082).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Carranza’s surveillance/IoT camera system to include Momeyer’s selection from multiple image-processing operations and processing modes so that the selected other device is instructed to perform an appropriate processing operation, such as object detection, facial recognition, license plate recognition, object tracking, histogram analysis, edge detection, SIFT, shared processing, or remote processing, in order to tailor image/video processing to the particular task, improve processing efficiency, and reduce unnecessary processing and bandwidth usage.
Regarding claim 12, Carranza further teaches maintaining the sensor data and object detection data that indicates an object detected by the battery powered device using the sensor data by teaching that a smart camera includes vision sensors that generate video streams associated with the environment in which the smart camera is deployed, and that the smart camera may process the video streams using its own processor and/or transmit video streams and associated metadata to another component for processing and/or storage purposes (para 0049). Carranza further teaches detecting an object in a first video stream of a first camera by processing the first video stream using appropriate object detection algorithms, and generating metadata associated with an initial state of the object, where the metadata is generated by processing the first video stream using computer vision algorithms and indicates object type, identity, physical characteristics, orientation, distance from cameras, position, speed, direction of travel, path, and trajectory (paras 0074-0075).
Carranza further teaches wherein determining, for the sensor data maintained by the battery powered device, whether the battery powered device or the device should process the sensor data uses the object detection data by teaching that metadata associated with the current state of an object can be used to predict a future state of the object and proactively configure other cameras in the surveillance system to capture, detect, and/or identify the object under optimal conditions (paras 0021-0024). Carranza further teaches using the object metadata to predict the future state of the object, identify a second camera for capturing the object at the second point in time, and configure the second camera to capture the object based on the future state of the object (Abstract; paras 0076-0081).
Regarding claim 13, Carranza further teaches generating, by the battery powered device, the object detection data by detecting, using the sensor data, the object represented by the sensor data by teaching that smart cameras include processors, memory, communication interfaces, and vision sensors, where the vision sensors generate video streams associated with the environment in which the smart camera is deployed, and a smart camera may store video streams in memory, “process video streams using its own processor,” and/or transmit video streams and associated metadata to another component for processing and/or storage purposes (para 0049).
Carranza further teaches detecting an object in a first video stream of a first camera by processing the first video stream using appropriate object detection algorithms, where the object may be a person, car, or animal, and the first video stream may be processed further to continue tracking the object or to identify the object using facial recognition or license plate recognition (para 0074). Carranza also teaches generating metadata associated with an initial state of the object by processing the first video stream using computer vision algorithms, where the metadata can indicate object type, identity, physical characteristics, orientation, distance from cameras, position, speed, direction of travel, path, and trajectory (para 0075).
Regarding claim 15, Carranza in view of Momeyer teaches sending, by a third device, the instruction to the component of the battery powered device to cause the battery powered device to provide the sensor data to the other device because Carranza teaches a surveillance system including a surveillance orchestration device and smart cameras, where each smart camera includes processors, memory, communication interfaces, and vision sensors that generate video streams associated with the environment in which the smart camera is deployed (paras 0047-0049). Carranza teaches that a smart camera may “process video streams using its own processor” and/or “transmit video streams and/or associated metadata” over its communication interface to another component, such as the surveillance orchestration device, “for processing and/or storage purposes” (para 0049).
Carranza further teaches that the metadata processor evaluates metadata, generates insights and predictions, and may use those insights and predictions to “retro-feed certain cameras via the control plane with tailored configuration settings” (para 0055). Carranza also teaches configuring a second camera to capture an object at a predicted second point in time and to capture, detect, and/or identify the object under optimal conditions (paras 0079-0082). Thus, Carranza teaches a third device, such as the surveillance orchestration device or metadata processor, sending a control-plane instruction/configuration to a component of the smart camera to cause the smart camera to provide video/sensor data or associated metadata to another device/component for processing.
Momeyer further teaches selecting whether image data should be processed locally, shared, or remotely, including a shared processing mode in which a processing result is sent to the server and a remote processing mode in which “the captured clip is sent to the server” for processing (paras 0060-0062). Thus, Carranza in view of Momeyer teaches that the third-device instruction causes the battery-powered camera/device to provide sensor/video data to another device for processing.
Regarding claim 16, Carranza further teaches wherein the third device is the same device as the other device because Carranza teaches a surveillance orchestration device that communicates with smart cameras and receives video streams and/or associated metadata from the smart cameras for processing and/or storage purposes (paras 0047-0049). Carranza also teaches that the metadata processor may use insights and predictions to “retro-feed certain cameras via the control plane with tailored configuration settings” (para 0055). Thus, Carranza teaches that the surveillance orchestration device may act as the third device that sends control-plane instructions/configuration information to the smart camera, and may also act as the other device that receives and processes the video stream and/or associated metadata from the smart camera.
Momeyer further teaches selecting shared or remote processing modes, where image data or processing results are sent to another device/server for processing (paras 0060-0062). Therefore, Carranza in view of Momeyer teaches that the same other processing device may send the instruction to the battery-powered device and also receive the sensor data for processing.
Claim 8-9 are rejected under 35 U.S.C. 103 as being unpatentable over Carranza et al. (US 2019/0043207 A1) in view of Momeyer et al. (US 2012/0033876 A1), and further in view of Beach et al. (US 11,544,505 B1).
Regarding claim 8, Carranza in view of Momeyer teaches the method of claim 1 as discussed above.
Carranza in view of Momeyer teaches determining, for the sensor data maintained by the battery powered device at the property, whether the battery powered device or another device should process the sensor data as discussed above. Carranza teaches a surveillance/IoT camera system in which a smart camera may process video streams using its own processor and/or transmit video streams and associated metadata to another component, such as the surveillance orchestration device, for processing and/or storage purposes (paras 0047-0049). Momeyer teaches determining whether image data should be processed locally, shared, or remotely based on device performance constraints, bandwidth, cost, power limitations, and processing constraints (paras 0057-0062).
Beach teaches another device that maintains a machine learning model trained using data specific to the property by teaching a monitoring system configured to monitor a property, including a camera located at the property and configured to generate images, wherein the system updates an object recognition model based on user feedback from image clusters (Abstract; col. 1, lines 59-67; col. 2, lines 1-31). Beach expressly teaches that the techniques train a “site-specific object recognition model” and that the ground truth data may be “specific to a user’s particular environment” (col. 1, lines 15-29). Beach further teaches that a home monitoring system includes surveillance devices located in or surrounding a home, including cameras, motion detectors, window/door sensors, and keypad access door locks, and that the system builds a site-specific object recognition model for objects detected by surveillance devices of the home monitoring system (col. 3, lines 11-35).
Beach further teaches that the site-specific object recognition model is trained to recognize particular objects frequently present at a particular location, such as vehicles, humans, or animals at a home, and that site-specific object recognition models can vary between different locations, where each location has a site-specific model trained to recognize objects particular to that location, such as a homeowner’s car or a resident of the home (col. 9, lines 11-29). Beach also teaches training different object recognition models for different home monitoring systems so that a home monitoring system for a particular home is customized for the particular objects of interest for that home (col. 9, lines 30-47).
Carranza in view of Momeyer and Beach teaches sending the instruction to the component of the battery powered device to cause the battery powered device to send the sensor data to the other device for processing using the machine learning model trained using the data specific to the property because Carranza teaches that smart cameras may transmit video streams and/or associated metadata to another component for processing and/or storage purposes (para 0049), and Momeyer teaches selecting shared or remote processing modes in which image data or partial processing results are sent to another device/server for processing (paras 0060-0062). Beach further teaches that the object recognition model can be provided to one or more surveillance devices of the home monitoring system, such as a camera, to detect objects of interest, and that the monitoring server may store sensor and image data received from the monitoring system and perform analysis of sensor and image data (col. 9, lines 11-18; col. 22, lines 35-43).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Carranza’s surveillance/IoT camera system, as modified by Momeyer’s local/shared/remote processing selection, to use Beach’s site-specific object recognition model trained using data specific to the property so that sensor/image data sent from a battery-powered camera or sensor to another device is processed using a model customized to the particular monitored property, in order to improve object recognition accuracy, reduce false alarms, and enable the monitoring system to recognize objects that are particular to the property, such as residents, vehicles, animals, or other recurring objects.
Regarding claim 9, Carranza in view of Momeyer teaches sending the instruction to the component of the battery powered device at the property to cause the battery powered device to send the sensor data to the other device because Carranza teaches that smart cameras include communication interfaces and vision sensors, where the vision sensors generate video streams, and a smart camera may “transmit video streams and/or associated metadata” over its communication interface to another component, such as the surveillance orchestration device, “for processing and/or storage purposes” (para 0049). Momeyer further teaches determining whether captured image data should be processed locally, shared, or remotely, including shared processing in which a result is sent to the server and remote processing in which “the captured clip is sent to the server” for processing (paras 0057-0062).
Beach teaches processing using property specific data by teaching that clustering techniques are applied to objects detected by surveillance devices of a home monitoring system to build a “site-specific object recognition model,” which provides a refined interpretation of a camera video feed using the site-specific object recognition model (col. 3, lines 11-22). Beach further teaches that the home monitoring system includes surveillance devices located in or surrounding the home, including cameras, motion detectors, window/door sensors, and keypad access door locks, and that detection images include metadata such as date/time, camera location, scene location of the object, preliminary classification, and/or object identification (col. 3, line 23-col. 4, line 45).
Beach further teaches that an object recognition model can be site-specific, where the site-specific model is trained to recognize particular objects, such as vehicles, humans, or animals, that are frequently present at a particular location, such as a home. Beach teaches that site-specific object recognition models can vary between different locations and are trained to recognize objects particular to that location, such as a homeowner’s car or a resident of the home (col. 9, lines 11-29). Beach further teaches that different object recognition models can be trained for different home monitoring systems such that the model for a particular home is customized for the particular objects of interest for that home (col. 9, lines 30-47).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Carranza’s surveillance/IoT camera system, as modified by Momeyer’s local/shared/remote processing selection, to use Beach’s property/site-specific object recognition data when processing sensor/image data sent from a battery-powered camera or sensor to another device, in order to improve object recognition accuracy, reduce false alarms, and enable the monitoring system to recognize objects that are specific to the monitored property, such as residents, vehicles, animals, or recurring objects at the property.
Claim 10-11, 14 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Carranza et al. (US 2019/0043207 A1) in view of Momeyer et al. (US 2012/0033876 A1), and further in view of Saptharishi (US 2014/0085480 A1).
Regarding claim 10, Carranza in view of Momeyer teaches the method of claim 1 as discussed above.
Carranza in view of Momeyer teaches sending the instruction to cause the battery powered device to send the sensor data to the other device because Carranza teaches that smart cameras include communication interfaces and vision sensors, where the vision sensors generate video streams, and that a smart camera may “transmit video streams and/or associated metadata” over its communication interface to another component, such as the surveillance orchestration device, “for processing and/or storage purposes” (para 0049). Momeyer further teaches selecting shared or remote processing modes, where image data or processing results are sent to a server for processing, including a remote processing mode in which “the captured clip is sent to the server” for processing (paras 0060-0062).
Saptharishi teaches sending, to the other device, a second instruction that indicates a processing priority for the sensor data to cause the other device to process the sensor data according to the processing priority by teaching a video surveillance network camera system including network video cameras and content-aware computer networking devices that analyze video visual content using video analytics to provide managed video (Abstract; paras 0012-0019). Saptharishi teaches that rules and actions are propagated to sensors and infrastructure devices, and that, based on user-specified rules, analytics tasks to be performed on a given sensor’s data are determined and best routes for the sensor data packet are computed (para 0104).
Saptharishi further teaches that a sensor information packet header includes “Analytics tasks to be performed,” “Current value,” “Current priority,” and “Delivery Time” fields (para 0100). Saptharishi teaches that “Current priority” is determined based on packet value and the time within which the packet must be delivered, and that “Delivery Time” is computed based on user-specified rules and values (para 0100). Saptharishi further teaches that, based on user-supplied rules, associated value, and delivery-time constraints, the packet or set of packets is assigned a priority, placed in an appropriate interface FIFO buffer according to the assigned priorities, and transmitted such that packet delivery is scheduled based on bandwidth allocation and packet priority (para 0117).
Saptharishi also teaches that users can categorize different events or objects of interest by assigning priority values, and that video data associated with events or objects of interest can be stored intelligently for preset time periods that vary based on the priority values (para 0051). Saptharishi further teaches that configuration information can include a ranking of importance levels of multiple events of interest, where the importance levels define distribution or storage priorities for managed video (claim 20).
Thus, Carranza in view of Momeyer and Saptharishi teaches sending sensor/video data from a battery-powered device to another processing device, and sending priority/task information to the other device indicating a processing priority for the sensor data so that the other device processes, routes, stores, and/or transmits the sensor data according to the indicated priority.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Carranza’s surveillance/IoT camera system, as modified by Momeyer’s local/shared/remote processing selection, to include Saptharishi’s priority, delivery-time, and analytics-task information for sensor/video data sent to another processing device, in order to ensure that higher-value or time-sensitive security/surveillance data is processed and handled before lower-value or less urgent data, reduce bandwidth and storage waste, and improve timely handling of important surveillance events.
Regarding claim 11, Saptharishi further teaches determining, using data for the sensor data, the processing priority for the sensor data by teaching that content-aware computer networking devices analyze video visual content of sensor data using video analytics, generate metadata describing the content of the sensor data, and use user-specified rules to assign value and priority to sensor data packets (paras 0016-0019, 0088, 0092).
Saptharishi teaches that the value assessment mechanism synthesizes a summary of sensor data, detects events in the summary based on user-supplied rules, assigns value to detected events based on the rules, and assigns priority to the event based on the assigned value, the time when the event occurred, and the probability with which the event occurs (para 0088). Saptharishi further teaches that user-specified rules map synthesized summaries containing metadata to events and assign value to the event and optionally specify an action in response to the event (para 0092).
Saptharishi further teaches that the sensor information packet header includes “Analytics tasks to be performed,” “Current value,” “Current priority,” and “Delivery Time” fields, and that “Current priority” is determined based on the packet value and the time within which the packet must be delivered to its destination (para 0100). Saptharishi also teaches that, based on user-supplied rules, associated value, and delivery-time constraints, the packet or set of packets is assigned a priority and placed in the appropriate interface FIFO buffer according to the assigned priorities (para 0117).
Thus, Saptharishi teaches determining the processing priority for the sensor/video data using data for the sensor data, including metadata, detected event information, packet value, event time, delivery-time constraints, and user-specified rules.
Regarding claim 14, Carranza in view of Momeyer teaches sending the instruction because Carranza teaches that smart cameras include processors, memory, communication interfaces, and vision sensors, and that a smart camera may “process video streams using its own processor” and/or “transmit video streams and/or associated metadata” to another component for processing and/or storage purposes (para 0049). Momeyer teaches selecting local, shared, or remote processing modes, including a local processing mode, a shared processing mode, and a remote processing mode in which the captured clip is sent to the server for processing (paras 0060-0062).
Saptharishi teaches sending, by a first component of the battery powered device, the instruction to a second, different component of the battery powered device by teaching a network camera that includes, within the camera housing, an imaging system, a video processing system, a data storage system, a power system, and an input/output interface and control system (para 0035). Saptharishi further teaches that the video processing system includes a rules based engine, video analytics, and a storage management system (para 0037), and that video analytics analyzes video data and generates metadata based on content of the video data (paras 0039-0042).
Saptharishi further teaches that the metadata is communicated to the rules based engine, the rules based engine determines whether an event or object of interest requires action, and, based on the metadata and rules, the storage management system controls first and second encoders to supply high-quality video data and/or low-quality video data to the data storage system and also controls whether to send high-quality or low-quality video data to a central data storage unit (paras 0061-0065). Thus, Saptharishi teaches a first component of the camera, such as the rules based engine/video analytics/storage management system, sending control information or instructions to a second, different component of the camera, such as an encoder, data storage system, or streaming/archiving control unit, to process and/or send video sensor data.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Carranza’s surveillance/IoT camera system, as modified by Momeyer’s local/shared/remote processing selection, to include Saptharishi’s internal camera control architecture in which a first camera component instructs a second, different camera component to process, encode, store, or transmit video data based on video analytics and rules, in order to efficiently control local camera processing/transmission operations, reduce bandwidth and storage usage, and cause the appropriate internal camera component to perform the selected processing or transmission operation.
Regarding claim 17, Carranza in view of Momeyer teaches determining, for the sensor data maintained by the battery powered device, whether the battery powered device or another device should process the sensor data because Carranza teaches that a smart camera may “process video streams using its own processor” and/or “transmit video streams and/or associated metadata” over its communication interface to another component, such as the surveillance orchestration device, “for processing and/or storage purposes” (para 0049). Momeyer further teaches determining whether image data should be processed locally, shared, or remotely based on bandwidth, device performance constraints, cost, power limitations, and processing/time constraints (paras 0057-0062).
Saptharishi teaches receiving an advertisement signal that indicates whether the other device has computational capacity to process the sensor data by teaching that, according to a protocol, a network of sensors, switches, and routers may discover one another and the services, such as video analytics, that each device offers (para 0020). Saptharishi further teaches that every device broadcasts information about its capabilities and the capabilities of sensors connected directly to it, and that the broadcast is trapped by other devices in the network so that local directories are updated (para 0103). Saptharishi also teaches that routers and switches determine the services offered by other infrastructure devices, storage devices, and user workstations during service discovery (paras 0104, 0109).
Saptharishi further teaches that multiple instances of the same service may be offered by multiple nodes and infrastructure devices in the network, and that every node and device that advertises a service also includes information about its throughput and any other limitation of the service or the service provider (para 0110). Saptharishi also teaches that the infrastructure devices compute an optimal deployment strategy based on the nature of the service and the state of the network, and that new sensors or services can be added and the devices adjust service deployment metrics accordingly (paras 0120-0121). Thus, Saptharishi teaches receiving an advertised/broadcast indication of another device’s services, throughput, limitations, and resource availability, which indicates whether the other device has computational capacity to process the sensor data.
Saptharishi further teaches wherein determining, for the sensor data maintained by the battery powered device, whether the battery powered device or another device should process the sensor data uses the advertisement signal because Saptharishi teaches that, based on the services offered by devices and the task list in the sensor information packet header, a packet is either locally processed or forwarded, and a service directory is consulted to determine the next destination for the sensor data (para 0117). Saptharishi also teaches that, based on user-defined rules, analytics tasks to be performed on a given sensor’s data are determined, and, based on the task list, best routes for the sensor data packet are computed (para 0104). Therefore, Saptharishi teaches using the advertised services/capabilities/resource information of other devices to determine whether and where sensor data should be processed.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Carranza’s surveillance/IoT camera system, as modified by Momeyer’s local/shared/remote processing selection, to include Saptharishi’s service-discovery and advertisement/broadcast mechanism in which devices advertise their processing services, throughput, limitations, and resource availability, in order to select an available device having appropriate computational capacity for processing sensor/video data, improve load balancing among processing devices, and avoid forwarding sensor data to devices that lack sufficient processing capacity.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Knode (US 2021/0352002) paragraph 0009-0010, 0139 and 0158
Any inquiry concerning this communication or earlier communications from the examiner should be directed to OMEED ALIZADA whose telephone number is (571)270-5907. The examiner can normally be reached Monday-Friday, 9:30 am until 5:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Brian Zimmerman can be reached at 571-272-3059. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/OMEED ALIZADA/Primary Examiner, Art Unit 2686
Prosecution Insights