Detailed Action
This action is written in response to the remarks and amendments dated 8/11/26. This action is made final. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
The Examiner is persuaded by the Applicant’s arguments pertaining to the outstanding rejections under §101. Accordingly, these rejections are withdrawn.
The Applicants argue that the previous art of record does not anticipate or render obvious the claims as currently amended. The Examiner provides updated prior art rejections below necessitated by the current amendments.
Subject Matter Eligibility
In determining whether the claims are subject matter eligible, the examiner has considered and applied guidance from MPEP § 2106. The examiner finds that the independent claims are directed to the practical application of motion detection for a camera. Furthermore, the combination of steps performed in the recited method cannot be practically performed as a mental process.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
The following references are relied upon in the rejections below:
Calacci (Calacci, Dan, Jeffrey J. Shen, and Alex Pentland. "The cop in your neighbor's doorbell: Amazon ring and the spread of participatory mass surveillance." Proceedings of the ACM on Human-Computer Interaction 6.CSCW2 (2022): 1-47.)
Chandola (Chandola, Varun, Arindam Banerjee, and Vipin Kumar. "Anomaly detection: A survey." ACM computing surveys (CSUR) 41, no. 3 (2009): 1-58.)
Choi (US 2019/0042079 A1)
Jin (Jin, Xiaoying, and Scott Paswaters. "A fuzzy rule base system for object-based feature extraction and classification." Signal Processing, Sensor Fusion, and Target Recognition XVI. Vol. 6567. SPIE, 2007.)
Joshi (Joshi, Ajay J., Fatih Porikli, and Nikolaos Papanikolopoulos. "Breaking the interactive bottleneck in multi-class classification with active selection and binary feedback." 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, 2010.)
Mondal (Mondal, Sujoy, and Abhirup Das Barman. "Human auditory model based real-time smart home acoustic event monitoring." Multimedia Tools and Applications 81.1 (2022): 887-906. Published online 18 Sept 2021.)
Neoran (US 2010/0004926 A1)
Saeki (US 2021/0104240 A1)
Stolojescu-Crisan (Stolojescu-Crisan, Cristina, Calin Crisan, and Bogdan-Petru Butunoi. "Access control and surveillance in a smart home." High-Confidence Computing 2.1 (Mar 2022): 100036.)
Xu (US 11,935,377 B1)
Claims 1, 6, 16-17, 20 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Xu, Choi and Saeki.
Regarding claim 1, Xu discloses a method, comprising:
obtaining image data from a camera defining a field of view, wherein the image data includes one or more separate images taken at different points in time;
Col. 5, lines 18 et seq., “In some embodiments, the lenses 108a-108n may be directed, tilted, panned, zoomed and/or rotated to capture the environment surrounding the camera system 100 ( e.g., capture data from the field of view).”
Col. 1, lines 34 et seq., “video frames”.
using a control circuit of the camera to determine when an object has moved into the field of view of the camera;
Col. 11, lines 19 et seq., “The video pipeline may be configured to recognize objects. Objects may be recognized by interpreting numerical and/or symbolic information to determine that the visual data represents a particular type of object and/or feature. For example, the number of pixels and/or the colors of the pixels of the video data may be used to recognize portions of the video data as objects.”
Fig. 5 (reproduced below), depicting object detection functionality within a device processor.
PNG
media_image1.png
644
664
media_image1.png
Greyscale
determining a category for the object using the one or more rules … with criteria specifying multiple different categories of objects; and
Fig. 9, “Perform classification on objects based on characteristics detected”.
Col. 10, line 19 et seq., “may be configured to recognize gardener/pool maintenance person and inhibit triggering an alarm. …. In another example, the low cost structured light based 3D sensing system may be configured to trigger an alarm upon recognition of certain objects (e.g. restraining order is out against ex-spouse, alert 911 if that person is detected).”
sending an alert to a computing device when the object moves within the field of view indicating the category of the object.
Col. 10, line 14 et seq., “In an example, the low cost structured light based 3D sensing system may be used to lock/unlock a door, arm/disarm an alarm system, and/or allow "tripwire" control of access to a restricted region ( e.g., a garden, a vehicle, a garage, a house, etc.).”
Col. 22, line 59 et seq., “The user interface 730 may enable the users to provide 60 input to the access control device 50 and/or enable the access control device 50 to present output to the users. In one example, the user interface 730 may comprise a display and/or a keypad. For example, the user interface 730 may be one component of the access control device 50 that may be configured to receive the credentials presented by the user.”
Col. 22, line 40 et seq., “Based on the results of the computer vision operations, the processors 714 may generate a security measure ( e.g., sound an alarm, present a notification to a security guard, initiate a lockdown procedure, etc.).
Choi discloses the following further limitation which Xu does not disclose:
storing, in a memory of a control circuit of the camera, rule criteria for one or more rules, the rule criteria specifying multiple different categories of objects and including one or more threshold values;
[0127] “In particular, the object recognition device C may estimate the object information related to the object area using the detected object area as input data of the trained object recognition model. The trained object recognition model, for example, may be a neural network or deep learning-based recognition model. In addition, the object recognition device C may estimate the object information related to the object area by analyzing the detected object area through a rule-based object recognition model.”
accepting user input indicating whether the image data matches the category determined by the control circuit;
[0149] “The usage history information may include, for example, object search history of the user, feedback history regarding search results or product purchase history. The user feedback may include, for example, at least one of a user feedback regarding a search result related to the object and a user feedback regarding object information. …. The user feedback regarding the object information may include, for example, information regarding whether the user has confirmed the object information”. (Emphasis added.)
[0305] “As another example, the model learning unit 1910-4 may learn a recognition model through reinforcement learning which uses feedback on whether the result of the circumstances determination according to learning is correct.”
At the time of filing, it would have been obvious to a skilled machine learning engineer to combine the rule-based object recognition model taught by Choi with the motion detection system of Xu because the former can help make accurate motion detection predictions based on relevant context information, eg what the object is. Furthermore, the active learning component (ie accepting and using feedback on recognition predictions from a user) can yield improved predictive performance over time. Both disclosures pertain to camera / object recognition systems.
Saeki discloses the following further limitation which Xu/Choi do not disclose:
determining at least one updated threshold value for the rule criteria using the image data, the category determined by the control circuit, and the user input; and
[0118] “The description support device 2 of the present embodiment stores the information on the user evaluating boxes 73 into the user evaluating information of the history data D1. The user evaluating information in the history data D1 can be used to improve the detecting accuracy of the system 1. For example, the user evaluating information can be used to adjust the detection threshold V1 in the system 1, or can be used as training data for machine learning such as active learning of the learned model 35.”
automatically installing the at least one updated threshold value in the memory of the control circuit, wherein a subsequent determination of a category by the control circuit uses the at least one updated threshold value.
Id.
See also [0034] “The controller 20 may be a hardware circuit such as a dedicated electronic circuit or a reconfigurable electronic circuit designed to realize a predetermined function.“
At the time of filing, it would have been obvious to a skilled machine learning engineer to combine the technique disclosed by Saeki for updating detection thresholds for rules because they can yield improved performance for rule-based systems over time. Choi explicitly provides for rule-based detection decisions (see [0127], excerpt reproduced supra), however the details of that embodiment are beyond the scope of that disclosure; Saeki provides the recited details.
Regarding claim 6, Xu discloses the further limitation comprising:
comparing pixel data from one of the separate images to corresponding pixel data from another different one of the separate images.
Col. 11, lines 23 et seq., “For example, the number of pixels and/or the colors of the pixels of the video data may be used to recognize portions of the video data as objects.”
Regarding claim 16, Xu discloses the further limitation wherein the control circuit includes a processor, memory, and communication circuits operable to establish and maintain one or more communication links.
Col. 19, line 25 ‘processor’.
Col. 20, line 35 ‘memory’.
Col. 19, lines 41 et seq., “In some embodiments, the camera system lO0i may communicate with the cloud service 702 and/or the access control device 50 wirelessly. In some embodiments, the camera system l00i may communicate with the cloud service 702 and/or the access control device 50 via the physical 45 connection 722 (e.g., an ethernet connection, a USB connection, a wire, etc.).”
Regarding claim 17, Xu discloses the further limitation wherein the control circuit is operable to maintain one or more wireless or wired communication links with one or more other computing devices via a computer network.
Col. 19, lines 41 et seq., “In some embodiments, the camera system lO0i may communicate with the cloud service 702 and/or the access control device 50 wirelessly. In some embodiments, the camera system l00i may communicate with the cloud service 702 and/or the access control device 50 via the physical 45 connection 722 (e.g., an ethernet connection, a USB connection, a wire, etc.).”
Regarding claim 20, Xu discloses the further limitation wherein a data analytics service is in communication with the control circuit and is operable to analyze image or audio data provided by the camera using a neural network.
Col. 12, line 4 et seq., “One of the hardware modules 190a-190n (e.g., 190b) may 5 implement a convolutional neural network (CNN) module. The CNN module 190b may be configured to perform the computer vision operations on the video frames.”
Regarding claim 22, Xu discloses the further limitation wherein a data analytics service is in communication with the control circuit and is operable to analyze image or audio data provided by the camera using a convolutional neural network.
Col. 12, line 4 et seq., “One of the hardware modules 190a-190n (e.g., 190b) may 5 implement a convolutional neural network (CNN) module. The CNN module 190b may be configured to perform the computer vision operations on the video frames.”
Claims 2, 13-14 and 16-19 are rejected under 35 U.S.C. 103 as being unpatentable over Xu, Choi, Saeki and Stolojescu-Crisan.
Regarding claim 2, Stolojescu-Crisan discloses the following further limitation which Xu/Choi/Saeki do not disclose comprising:
sending the category and one or more of the separate images to a personal computing device, wherein the personal computing device is configured to present a user interface that provides access to the separate images and the category.
P. 8, fig. 12, “normal desktop view”, image captions: “Fat Cat”, “Fluffy Cat”, “The Dog”.
PNG
media_image2.png
277
974
media_image2.png
Greyscale
At the time of filing, it would have been obvious to a person of ordinary skill to combine the smart device GUI disclosed by Stolojescu-Crisan with the Xu/Choi/Saeki system because home security-related alerts may be time sensitive, and homeowners are likely to have a smartphone at hand at all times. Prompt notification will provide for escalation (eg calling 911) or de-escalation (eg disarming an alarm).
Regarding claim 13, Xu discloses the further limitation wherein the camera is mounted adjacent to a door.
Col. 2, line 27, “doorbell cameras”.
Regarding claim 14, Xu discloses the further limitation wherein the camera is mounted in a doorbell mechanism.
Col. 2, line 27, “doorbell cameras”.
Regarding claim 16, Stolojescu-Crisan discloses the further limitation wherein the control circuit includes a processor, memory, and communication circuits operable to establish and maintain one or more communication links.
P. 1, introduction, “IoT uses the Internet Protocol (IP) to connect devices, which include smart- phones, tablets and digital assistants to various types of sensors, appliances, and systems such as lighting, temperature, or security. The characteristics of the IoT are very wide and include a variety of physical elements, as shown in Fig. 1 [3] .”
The Examiner notes that a processor, memory and communication circuits are inherent throughout the IoT system described above.
The obviousness analysis of claim 2 applies equally here.
Regarding claim 17, Stolojescu-Crisan discloses the further limitation wherein the control circuit is operable to maintain one or more wireless or wired communication links with one or more other computing devices via a computer network.
P. 1, introduction, “IoT uses the Internet Protocol (IP) to connect devices, which include smart- phones, tablets and digital assistants to various types of sensors, appliances, and systems such as lighting, temperature, or security. The characteristics of the IoT are very wide and include a variety of physical elements, as shown in Fig. 1 [3] .”
P. 2, fig. 2 (reproduced below).
PNG
media_image3.png
440
536
media_image3.png
Greyscale
The obviousness analysis of claim 2 applies equally here.
Regarding claim 18, Stolojescu-Crisan discloses the further limitation wherein the camera is an IP camera.
P. 1, introduction, “IoT uses the Internet Protocol (IP) to connect devices, which include smart- phones, tablets and digital assistants to various types of sensors, appliances, and systems such as lighting, temperature, or security. The characteristics of the IoT are very wide and include a variety of physical elements, as shown in Fig. 1 [3] .”
The obviousness analysis of claim 2 applies equally here.
Regarding claim 19, Stolojescu-Crisan discloses the further limitation wherein a data analytics service is in communication with the control circuit and is operable to analyze image or audio data provided by the camera using artificial intelligence.
Abstract and passim: “smart home solutions” using “smart devices”.
See also p. 1, introduction, “Intelligent Building Management Systems”.
The obviousness analysis of claim 2 applies equally here.
Claims 3-4 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Xu, Choi, Saeki and Joshi.
Regarding claim 3, Joshi discloses the following further limitation which Xu/Choi/Saeki do not disclose comprising:
accepting user input from a user indicating that the image data matches the category determined by the control circuit.
P. 3, first paragraph, “Based on the estimated membership probabilities for the query image, the sample selection algorithm selects a sample image from the current training set. The query-sample pair is shown to the user for feedback. If a “match” response is obtained, indicating that the query and sample images belong to the same category, the query image is added to the current training set along with its category label. If a “no-match” response is obtained, the sample selection algorithm is again invoked to ask for a different sample image.”
At the time of filing, it would have been obvious to a person of ordinary skill to combine the active learning technique disclosed by Joshi with the Xu/Choi/Saeki system because this will increase the number (and reliability) of labeled samples for further training the object detection system.
Regarding claim 4, Joshi discloses the further limitation comprising:
sending one or more of the separate images, the category, and user input indicating whether the image data matches the category to a data analytics service via a communication link.
P. 3, first paragraph, “Based on the estimated membership probabilities for the query image, the sample selection algorithm selects a sample image from the current training set. The query-sample pair is shown to the user for feedback. If a “match” response is obtained, indicating that the query and sample images belong to the same category, the query image is added to the current training set along with its category label. If a “no-match” response is obtained, the sample selection algorithm is again invoked to ask for a different sample image.”
P. 2, fig. 2 (reproduced below), ‘communication links’ are indicated by the arrows between components.
PNG
media_image4.png
448
382
media_image4.png
Greyscale
The obviousness analysis of claim 3 applies equally here.
Regarding claim 10, Joshi discloses the following further limitation which Xu/Choi/Saeki does not disclose comprising:
accepting user input from a user indicating that sound data captured by the camera matches a category determined by the control circuit.
P. 3, first paragraph, “Based on the estimated membership probabilities for the query image, the sample selection algorithm selects a sample image from the current training set. The query-sample pair is shown to the user for feedback. If a “match” response is obtained, indicating that the query and sample images belong to the same category, the query image is added to the current training set along with its category label. If a “no-match” response is obtained, the sample selection algorithm is again invoked to ask for a different sample image.”
The obviousness analysis of claim 3 applies equally here.
Claims 5, 8 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Xu, Choi, Saeki and Chandola.
Regarding claim 5, Chandola discloses the following further limitation which Xu/Choi/Saeki do not disclose comprising:
using a data analytics service to determine updated rule criteria for at least one of the rules specifying different categories of objects, wherein the data analytics service uses the image data provided by the camera, the category determined by the control circuit, and user input to determine the updated rule criteria.
P. 21, “A basic multi-class rule-based technique consists of two steps. The first step is to learn rules from the training data using a rule learning algorithm, such as RIPPER, Decision Trees, and so on. Each rule has an associated confidence value that is proportional to ratio between the number of training instances correctly classified by the rule and the total number of training instances covered by the rule. The second step is to find, for each test instance, the rule that best captures the test instance. The inverse of the confidence associated with the best rule is the anomaly score of the test instance.”
See also p. 17, sec. 3.5, discussion applications to image processing.
At the time of filing, it would have been obvious to a person of ordinary skill to apply rule-based anomaly detection (as taught by Chandola) to the intrusion detection system of Xu/Choi/Saeki because intrusion detection is an anomaly detection task, meaning it is also a classification task. This combination would provide for alerts or alarms upon detection of unusual activity in the house, eg an intruder.
Regarding claim 8, Xu discloses the further limitation comprising:
obtaining sound data from an input device of the camera, wherein the sound data includes sound data obtained from the input device at different points in time; and
Col. 4, lines 44 et seq., “The circuit 114 may be implemented as one or more sensors (e.g., motion, ambient light, proximity, sound, etc.).”
Chandola discloses the following further limitation which Xu/Choi/Saeki do not disclose comprising:
using a control circuit of the camera to compare the sound data with rule criteria in the control circuit to determine an event category, wherein the rule criteria specifies one or more categories of events.
P. 18, “A single sensor network might be comprised of sensors that collect different types of data, such as binary, discrete, continuous, audio, video, and so forth. The data is generated in a streaming mode. Often times the environment in which the various sensors are deployed, and the communication channel, induce noise and missing values in the collected data.”
P. 21, “Variants of the basic technique have been proposed for anomaly detection in audio signal data”.
The obviousness analysis of claim 5 applies equally here.
Regarding claim 21, Xu discloses the further limitation wherein the rule criteria optionally includes one or more predetermined image patterns and/or one or more predetermined audio input patterns.
P. 12, line 19 et seq., “The CNN module 190b may be 20 configured to implement pattern and/or image recognition using a training process through multiple layers of feature detection. The CNN module 190b may be configured to conduct inferences against a machine learning model.”
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Xu, Choi, Saeki, Chandola and Calacci.
Regarding claim 9, Calacci discloses the following further limitation which Xu/Choi/Saeki/Chandola do not disclose comprising:
sending the event category and at least a portion of sound data captured by the camera to a personal computing device, wherein the personal computing device is configured to present a user interface that provides access to the sound data and the category.
P. 20, fig. 8 (a) and (b) (reproduced below). The Examiner notes that each video clip includes sound, as suggested by the speaker (volume) icon in the lower right corner of each video frame.
PNG
media_image5.png
290
1162
media_image5.png
Greyscale
At the time of filing, it would have been obvious to a person of ordinary skill to transmit video data including sound to a user’s personal computing device because this would provide for an immediate alert to the user / homeowner of activities taking place at their residence. Audio can give important additional information or context about persons or objects depicted in the frames.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Xu, Choi, Saeki and Neoran.
Regarding claim 11, Neoran discloses the following further limitation which Xu/Choi/Saeki do not disclose comprising:
using a data analytics service to determine updated rule criteria for at least one of the rules specifying different categories of objects, wherein the data analytics service uses sound data captured by the camera, the category determined by the control circuit, and user input to determine the updated rule criteria.
[0011] “In accordance with an aspect of an embodiment of the invention, classification and/or segmentation of the audio content by the apparatus includes obtaining an input audio signal; dividing the signal into one or more audio segments; classifying each segment of the audio signal, for example, using a multi-stage sieve-like approach and applying Bayesian and/or rule-based methods”.
At the time of filing, it would have been obvious to a person of ordinary skill to apply the rule-based audio classification techniques disclosed by Neoran with the intrusion detection system of Xu/Choi/Saeki because audio data is captured by the Xu system, and the analysis thereof would provide for more complete and accurate classification predictions by the Xu/Choi/Saeki system.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Xu, Choi, Saeki and Mondal.
Regarding claim 12, Mondal discloses the further limitation which Xu/Choi/Saeki do not disclose comprising:
comparing regions from sound data captured by the camera to one or more predetermined audio input patterns stored in a memory of the control circuit.
P. 892, fig. 3(c) (reproduced below).
PNG
media_image6.png
302
360
media_image6.png
Greyscale
At the time of filing, it would have been obvious to a person of ordinary skill to apply the audio pattern matching technique disclosed by Mondal with the intrusion detection system of Xu because the former can help inform better automated classification of intrusion events vs non-intrusion events.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Xu, Choi, Saeki, Chandola and Jin.
Regarding claim 15, Jin discloses the following further limitation which Xu/Choi/Saeki/Chandola do not disclose wherein the rule criteria include threshold values ranging between 0.0 and 1.0.
P. 9, sec. 4.1, “In the experiments, the confidence value thresholds were set slightly lower than the half magnitude of membership function which is 0.5.”
At the time of filing, it would have been obvious to a person of ordinary skill to apply the rule threshold criteria specified by Jin in the Xu//Choi/Saeki/Chandola system because membership functional are commonly normalized between 0 and 1 for interpretability.
Additional Relevant Prior Art
The following references were identified by the Examiner as being relevant to the disclosed invention, but are not relied upon in any rejection:
Edwards discloses a home / neighborhood security system featuring a plurality of linked cameras. (US 2016/0309123 A1)
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
Information regarding the status of an application may be found at the USPTO Patent Center at https://patentcenter.uspto.gov. Any inquiry concerning this communication should be directed to Vincent Gonzales at (571) 270-3837. The examiner can normally be reached Monday-Friday 7 am to 4 pm MT. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang, can be reached at (571) 270-7092.
/Vincent Gonzales/Primary Examiner, Art Unit 2124
