Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 02/20/2026 has been entered.
Claim Objections
Claim 22 is objected to because of the following informalities:
Claim 22 recites “to match” immediately followed by “to access”, which appears redundant/awkwardly phrased. The examiner recommends choosing one or the other to improve clarity.
Appropriate correction is required.
Specification
The lengthy specification has not been checked to the extent necessary to determine the presence of all possible minor errors. Applicant’s cooperation is requested in correcting any errors of which applicant may become aware in the specification (MPEP 608.01, ¶6.31).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 4-11, 13-15, 17, and 20-21 are rejected under 35 U.S.C. 103 as being unpatentable over Izhikevich (US9849588B2) in view of Wang (“Robot Button Pressing in Human Environments”, IEEE International Conference on Robotics and Automation, 2018).
Regarding claim 1, Izhikevich teaches;
A method comprising: accessing (i) first sensor data of a property captured at a first time and (ii) second sensor data of the property captured at a second time subsequent to the first time (taught as learning associations of context-action indication pairs with a time window, column 38 lines 5-16, indicating that separate frames, and thus times, of data are used to generate/learn associations);
detecting, using the first sensor data and the second sensor data, (i) a first state change of a manual interface at the property that controls a device at the property and (ii) a second state change of a region of the property affected by the first state change of the manual interface that controls the device (taught as learning associations of context-action indication pairs with a time window, column 38 lines 5-16, wherein that the control input [that determines the context-association pair] is communicated with interfaces, such as wired light switches, column 10 lines 9-17, or button presses of a remote controller, column 10 lines 20-24);
generating, using data that indicates the first state change of the manual interface and the second state change of the region of the property affected by the first state change, a mapping between (i) the manual interface at the property that controls the device at the property and (ii) an identifier for the device at the property (taught as generating action-indication associations, column 28 line 65- column 29 line 5; exemplified such that context associated from the environment, such as the state of other equipment in the premises, generates associations, column 32 lines 1-13; for example, the use of a button triggers a context, and an association with a TV on/off state may be formed, column 38 line 62 – column 39 line 11);
adding, to a database of a plurality of mappings, the generated mapping between (i) the manual interface that controls the device and (ii) the identifier for the device (taught as generating action-indication associations [mapping], column 28 line 65- column 29 line 5).
However, Izhikevich does not explicitly teach; in response to detecting a trigger for operating the device, controlling, using data from the database, a robot to operate the manual interface that controls the device.
Wang teaches; in response to detecting a trigger for operating the device, controlling, using data from the database (taught as using button recognition and localization, D. Button Panel Recognition and Localization, p7177, which is based on the calibration and storage of button panels, C. Environmental annotation and calibration), a robot to operate the manual interface that controls the device (taught as the controller of the robot initiating upon a condition, F. Control, p7178).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to explicitly control a robot based on learned associations as taught by Wang in the system taught by Izhikevich in order to enact automation functions. As taught by Wang, robots have to be able to interact with systems designed for human control (1. Introduction), in environments with legacy/older control systems. As Izhikevich already intends to implement the learning functions to control a robotic device present in a home automation system (column 8 lines 25-29), one of ordinary skill in the art would think to control a robot, such as taught by Wang, with manipulators to interact with a home environment that is not necessarily fully ‘smart’.
Regarding claim 4, Izhikevich as modified by Wang teaches;
The method of claim 1 (see claim 1 rejection). Izhikevich further teaches; wherein:
the first sensor data captured at the first time indicates a first status of the manual interface configured to control the device (taught as learning process including periodically acquiring a reference image and a current image, column 37 line 34-36; ), and
the second sensor data captured at the second time indicates (i) a second status, different than the first status, of the manual interface configured to control the device and (ii) an effect of the device being controlled by the manual interface (taught as a given [current] image being compared to the reference image in order to determine the context, column 38 lines 3-5, wherein the context determination is based on the detection of a sudden change between the given and reference image within a time window, column 37 lines 19-25).
Regarding claim 5, Izhikevich as modified by Wang teaches;
The method of claim 4 (see claim 4 rejection), wherein the effect of the device being controlled by the manual interface includes illumination of a portion of the property or area near the property (implied in detecting environmental conditions [part of context] including ambient light level, column 22 lines 48-54, with an exemplified learned association is turn lights on, column 38 lines 56-57).
Regarding claim 6, Izhikevich as modified by Wang teaches;
The method of claim 4 (see claim 4 rejection). Izhikevich further teaches; wherein detecting the first state change of the manual interface at the property that controls the device at the property comprises: comparing the first status of the manual interface configured to control the device with the second status of the manual interface configured to control the device (taught as a given [current] image being compared to the reference image in order to determine the context, column 38 lines 3-5, wherein the context determination is based on the detection of a sudden change between the given and reference image within a time window, column 37 lines 19-25).
Regarding claim 7, Izhikevich as modified by Wang teaches;
The method of claim 6 (see claim 6 rejection). Izhikevich further teaches; comprising: determining the first status of the manual interface configured to control the device is different than the second status (taught as a given [current] image being compared to the reference image in order to determine the context, column 38 lines 3-5, wherein the context determination is based on the detection of a sudden change between the given and reference image within a time window, column 37 lines 19-25).
Regarding claim 8, Izhikevich as modified by Wang teaches;
The method of claim 1 (see claim 1 rejection), wherein detecting the second state change of the region of the property affected by the first state change of the manual interface that controls the device comprises: comparing one or more values of pixels of an image of the first sensor data captured at the first time with one or more values of pixels of an image of the second sensor data captured at the second time (taught as a given [current] image being compared to the reference image in order to determine the context, column 38 lines 3-5, wherein the context determination is based on the detection of a sudden change between the given and reference image within a time window by analyzing characteristics including pixel brightness, color etc. column 37 lines 19-25).
Regarding claim 9, Izhikevich as modified by Wang teaches;
The method of claim 1 (see claim 1 rejection). Izhikevich further teaches; prior to controlling the robot to operate the manual interface that controls the device, comprising:
detecting an object in a first area at the property using at least one of the first sensor data or the second sensor data (taught as detecting objects [as part of context information], column 22 lines 48-52), wherein context data is acquired initially before controlling the robot); and
determining that the device is mapped to and affects the first area (indicated in that multiple monitoring areas/controllers are used and selectively interact with the robotic device based on the assigned area, column 4 lines 9-16; thus, a device within a first monitoring area would correspond to the universal controller associated with the first monitoring area, and a device within a second monitoring area would correspond to a universal controller associated with the second monitoring area).
Regarding claim 10, Izhikevich as modified by Wang teaches;
The method of claim 9 (see claim 9 rejection), comprising controlling the robot to operate the manual interface that controls the device in response to:
detecting the object in the first area using at least one of the first sensor data or the second sensor data (taught as detecting objects [as part of context information], column 22 lines 48-52), wherein context data is acquired initially before controlling the robot); and
determining that the device is mapped to and affects the first area (indicated in that multiple monitoring areas/controllers are used and selectively interact with the robotic device based on the assigned area, column 4 lines 9-16; and further indicated a location/area dependence in context information, column 10 lines 34-36).
Regarding claim 11, Izhikevich as modified by Wang teaches;
The method of claim 1 (see claim 1 rejection). Izhikevich further teaches; comprising: determining the device is not a smart device (while not explicit, the context association involved can include non-smart devices, such as TV remote buttons, column 33 lines 52-59) and is configured to be controlled by the manual interface (exemplified in one scenario as buttons that interact with a TV, column 33 lines 52-59).
Regarding claim 12, Izhikevich as modified by Wang teaches;
The method of claim 11 (see claim 11 rejection). Izhikevich further teaches;, wherein determining the device is not a smart device and is configured to be controlled by the manual interface includes: maintaining data indicating detection of movement of a person interacting with the manual interface (exemplified in one scenario as a user interacting with buttons that interact/control a TV for association development, column 33 lines 52-59).
Regarding claim 13, Izhikevich as modified by Wang teaches;
The method of claim 1 (see claim 1 rejection). Izhikevich further teaches; wherein the manual interface is a physical switch or physical button (exemplified in one scenario as buttons that interact with a TV, column 33 lines 52-59).
Regarding claim 14, Izhikevich as modified by Wang teaches;
The method of claim 1 (see claim 1 rejection). Izhikevich further teaches; wherein adding the generated mapping to the database of the plurality of mappings comprises: adding a location of the manual interface to the database of the plurality of mappings (indicated in that multiple monitoring areas/controllers are used and selectively interact with the robotic device based on the assigned area, column 4 lines 9-16; and further indicated a location/area dependence in context information, column 10 lines 34-36).
Regarding claim 15, Izhikevich as modified by Wang teaches;
The method of claim 1 (see claim 1 rejection). However, Izhikevich doesn’t explicitly teach; wherein controlling the robot to operate the manual interface that controls the device comprises:
providing instructions to the robot indicating how to access the manual interface controlling the device (indicated in that, upon determining an existing association, accessing an instruction associated with the action and communicating the instruction, column 31 lines 28-30).
Regarding claims 17 and 20-21, it has been determined that no further limitations exist apart from those previously addressed in claims 1. Therefore, claims 17-20 are rejected under the same rationale as claim 1.
Claim(s) 3 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Izhikevich (US9849588B2) as modified by Wang (“Robot Button Pressing in Human Environments”, IEEE International Conference on Robotics and Automation, 2018), and further in view of Nguyen (“PPS-Tags: Physical, Perceptual and Semantic Tags for Autonomous Mobile Manipulation, 2009).
Regarding claim 3, Izhikevich as modified by Wang teaches;
The method of claim 1 (see claim 1 rejection). Izhikevich further teaches; wherein detecting the second state change of the region of the property affected by the first state change of the manual interface that controls the device comprises;
determining a difference between the first sensor data captured at the first time and the second sensor data captured at the second time (taught as the learning process [for context association pairs] including periodically acquiring a reference image and a current image, column 37 line 34-36).
However, Izhikevich does not explicitly teach; comparing the difference to a threshold.
Nguyen teaches; determining a difference between the first sensor data captured at the first time and the second sensor data captured at the second time (taught as taking the average brightness/intensity of an image before and after performing an action, 2) Manipulating the PPS-tag, page 6, column 2); and
comparing the difference to a threshold (taught as comparing the brightnesses and testing if the change is greater than a threshold, 2) Manipulating the PPS-tag, page 6, column 2).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the comparison by Nguyen in the system taught by Izhikevich in order to identify differences in images accurately. General sensor noise, uncertainty, and error, without a threshold for activation, may trigger unintended activations or contexts, and thus should be accounted for to avoid such accidental activations.
Regarding claim 19, it has been determined that no further limitations exist apart from those previously addressed in claim 3. Therefore, claim 19 is rejected under the same rationale as claim 3.
Claim(s) 16 is rejected under 35 U.S.C. 103 as being unpatentable over Izhikevich (US9849588B2) in view of Wang (“Robot Button Pressing in Human Environments”, IEEE International Conference on Robotics and Automation, 2018), and further in view of Roberts (US20210383822A1).
Regarding claim 16, Wang as modified by Nguyen teaches;
The method of claim 15 (see claim 15 rejection). However, Wang does not explicitly teach; wherein the instructions include flight maneuvers for the robot to perform to access a portion of the manual interface.
Roberts teaches; wherein the instructions include flight maneuvers for the robot to perform to access a portion of the manual interface (taught as a drone sent by a system to investigate, such as sounds, paragraph 0046, using a navigational route to move around the property to reach a desired destination, paragraphs 0032-paragraph 0033).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate a robotic drone and flight planning as taught by Roberts in the system taught by Wang in order to improve navigation. Wang already discusses the need for future navigational methods and mapping, suggesting SLAM and more specific trajectories ( C. Environmental annotation and calibration, p7177 column 1). Roberts not only provides the suggested mapping and navigation (paragraphs 0032-0033), but includes the use of a drone (which would fly), which additionally improves potential reach and navigation [e.g avoiding ground obstacles more easily].
Claim(s) 22 is rejected under 35 U.S.C. 103 as being unpatentable over Izhikevich (US9849588B2) in view of Wang (“Robot Button Pressing in Human Environments”, IEEE International Conference on Robotics and Automation, 2018), and further in view of Muller (US20220106039A1).
Regarding claim 22, Izhikevich as modified by Wang teaches;
The method of claim 15 (see claim 15 rejection). Izhikevich further teaches; wherein providing the instructions to the robot indicating how to access the manual interface controlling the device comprises: providing, to the robot, instructions(taught as providing commands for simultaneous control of multiple controllable aspects, including speed and direction, column 12 lines 49-59).
However, Izhikevich does not explicitly teach; that indicate flight maneuvers including an orientation of the robot to substantially match to access a portion of the manual interface.
Muller teaches; that indicate flight maneuvers including an orientation of the robot to substantially match to access a portion of the manual interface (taught as a drone capable of physical interaction with the environment, including contact based functions, paragraph 0008, such as a surface in any orientation, paragraph 0072, and is controlled based on a guided or controlled navigation on a route to a contract point, paragraph 0069).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate a robotic drone as taught by Muller with a robotic brain/association learning system taught by Izhikevich in order to improve property automation. Izhikevich already teaches the intent to control a robotic device in a home automation system, including switches and actuators, (column 8 lines 25-29); more explicitly incorporating a robotic drone that specifically can manipulate interfaces, as taught by Mueller, [and would additionally fulfill the objectives taught in Wang, 1. Introduction) would allow for advanced manipulation capabilities, as taught by Muller (paragraph 0014).
Response to Arguments
Applicant argues on pages 8-11 of the remarks that the amendments to the claims sufficiently distinguish over the previously recited prior art.
The examiner agrees that the implications and arguments presented help distinguish over the combination of Wang and Nguyen (in particular the apparent use of labels/QR codes in Wang), and withdraws the previous rejection. However, a new rejection in view of Izhikevich is presented above, which further details the context mapping presented in the arguments.
As a note, the examiner recommends further integrating/detailing claim 15/22 to help overcome the new rejection; for example, further detail on providing instructions to ‘access’ the manual interface to prevent the broad interpretation implemented here.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
For further machine learning, such as reinforcement and demonstration learning, with regards to control home functions; US20160123619A1, US20210178585A1
For further smart home and robot mission integration; US20220151450A1
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GABRIEL ANFINRUD whose telephone number is (571)270-3401. The examiner can normally be reached M-F 9:30-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jelani Smith can be reached on (571)270-3969. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GABRIEL ANFINRUD/Examiner, Art Unit 3662
/JELANI A SMITH/Supervisory Patent Examiner, Art Unit 3662
Prosecution Insights