Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This is the first Office Action on the merits. Claims 1-18 are currently pending.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. KR10-2021-0060334, filed on 12/06/2023.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 11/10/2023 and 12/31/2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-4, 7, 12-14, and 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Gil et al. (US20180353042A1) in view of Lee (US20200004984A1), hereinafter Gil and Lee, respectively.
Regarding claim 1, Gil teaches of a robot device ("The cleaning robot 100", [0050]) comprising: a moving assembly configured to move the robot device ("The driving unit 250 can perform, for example, movement or cleaning operation of the cleaning robot 100", [0058]); a camera configured to generate an image signal by photographing surroundings of the robot device during driving of the robot device ("The camera module 220…to photograph the moving trace of the cleaning robot 100", [0053]); a communication interface ("a communication module 225…The cleaning robot 100 can communicate with an external server using a communication module", [0055]); and at least one processor ("The processor 210", [0051]) configured to: detect a person in a driving area of the robot device ("the processor 210 may generate skeletal information of a human using the photographed image and depth information of the subject using a skeleton extraction algorithm. For example, the processor 210 can detect an arm, a leg, a torso, a face, and the like constituting a human body by using a skeleton extraction algorithm", [0054]), based on a determination that no person is present in the driving area, recognize an object in an input image generated from the image signal using a cloud machine learning model ("Referring to operation 1130, when the object type is the object to be avoided, the cleaning robot 100 can reset the driving route in progress. Referring to operation 1035, when the type of the object is the object to be driven, the cleaning robot 100 can maintain the route in progress", [0163], "The object to be avoided may be, for example…people", [0164], "The external server 260 inputs the image data received from the cleaning robot 100 to the learning network model 270 to find an image similar to the inputted image", [0067]), based on a determination that a person is present in the driving area, recognize the object in the input image generated from the image signal using an on- device machine learning model ("Referring to operation 1130, when the object type is the object to be avoided, the cleaning robot 100 can reset the driving route in progress. Referring to operation 1035, when the type of the object is the object to be driven, the cleaning robot 100 can maintain the route in progress", [0163], "The object to be avoided may be, for example…people", [0164], "the learning network model 270 may be integrated in a hardware chip form and become a part of the processor 210", [0065]), and control the driving of the robot device through the moving assembly by using a result of recognizing the object ("Referring to operation 1130, when the object type is the object to be avoided, the cleaning robot 100 can reset the driving route in progress. Referring to operation 1035, when the type of the object is the object to be driven, the cleaning robot 100 can maintain the route in progress", [0163]), wherein the cloud machine learning model operates on a cloud server connected through the communication interface, and the on-device machine learning model operates on the robot device ("the learning network 270 may exist simultaneously in the memory (not shown) of the server 260 and the memory 230 of the cleaning robot 100", [0074]).
However, Gil does not teach of in a first mode and in a second mode.
Lee, in the same field of endeavor, teaches of in a first mode (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when privacy information is not included in the inputted content information, the processor may be configured to store the content information in the second region of the SoC memory”, [0019]), and in a second mode (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when the inputted content information includes image information, the apparatus may further include an object recognizer configured to recognize an object from the image information, and when privacy information is included in the image information or the recognized object, the processor may be configured to store the content information in the first region” [0020]).
Therefore, one of ordinary skill in the art, before the effective filing date of the claimed invention, would have modified the teachings of Gil with the teachings of Lee to have a first and second configuration of using machine learning on the cloud and locally with reasonable expectations of success. One of ordinary skill in the art would have been motivated to make this modification in order to protect the privacy of a person detected during object detection which may have included sensitive information (Lee, [0005]- [0006]).
Regarding claim 2, modified Gil teaches of all claim limitations of claim 1 as stated above, further comprising: an output interface (“The display 255 may be, for example, positioned on the upper surface of the cleaning robot 100 to display various states of the cleaning robot 100”, [0056]), wherein the at least one processor is further configured to provide a notification recommending through the output interface (“when the type of object detected by the cleaning robot 100 is estimated as an object to be avoided and the moving route is reset, a notification signal can be generated”, [0111]) when it is determined that the person is present in the driving area while operating in the first mode ("when the type of the object is estimated as a person among the object to be avoided”, [0109]), and provide a notification recommending through the output interface (“when the type of object detected by the cleaning robot 100 is estimated as an object to be avoided and the moving route is reset, a notification signal can be generated”, [0111]) when it is determined that no person is present in the driving area while operating in the second mode ("when the type of the object is estimated as a person among the object to be avoided”, [0109]).
However, modified Gil does not teach of changing an operation mode to the second mode and changing the operation mode to the first mode.
Lee, in the same field of endeavor, teaches of changing an operation mode to the second mode (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when the inputted content information includes image information, the apparatus may further include an object recognizer configured to recognize an object from the image information, and when privacy information is included in the image information or the recognized object, the processor may be configured to store the content information in the first region” [0020]), and changing the operation mode to the first mode (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when privacy information is not included in the inputted content information, the processor may be configured to store the content information in the second region of the SoC memory”, [0019]).
Therefore, one of ordinary skill in the art, before the effective filing date of the claimed invention, would have modified the teachings of modified Gil with the teachings of Lee to have a first and second configuration of using machine learning on the cloud and locally with reasonable expectations of success. One of ordinary skill in the art would have been motivated to make this modification in order to protect the privacy of a person detected during object detection which may have included sensitive information (Lee, [0005]- [0006]).
Regarding claim 3, modified Gil teaches of all claim limitations of claim 1 as stated above, additionally, wherein the at least one processor is further configured to determine whether the person is present in the driving area based on the object recognition result of the cloud machine learning model or the on-device machine learning model (“the processor 210 can detect an arm, a leg, a torso, a face, and the like constituting a human body by using a skeleton extraction algorithm”, [0054]), “The processor 280 may estimate the object as “human”, [0082]).
Regarding claim 4, modified Gil teaches of all claim limitations of claim 1 as stated above, additionally, wherein the communication interface is configured to communicate with an external device (“The communication module 225… The cleaning robot 100 can communicate with an external server using a communication module”, [0055]) including a first sensor configured to detect the person in the driving area (“the cleaning robot 100 may provide a sensor 180 (e.g., a 3D camera, a rotation-type light detection and ranging (LiDAR) sensor)”, [0048]), and the at least one processor is further configured to determine whether the person is present in the driving area based on a sensor detection value of the first sensor (“the processor 280 may determine (or estimate) an outer shape of the object as “human” “, [0081]).
Regarding claim 7, modified Gil teaches of all claim limitations of claim 1 as stated above, additionally, wherein the at least one processor is further configured to scan the entire driving area and determine whether the person is present in the driving area based on a scan result of the entire driving area ("Referring to FIG. 3A, the cleaning robot 100 may use a 3D depth camera (e.g., 3D depth camera 180 of FIGS. 1A and 1B) or rotating LiDAR sensor to obtain at least one image including distance information with the object located at an outside of the cleaning robot 100. For example, the cleaning robot 100 may obtain a distance value with respect to an object located on a driving route. The cleaning robot 100 can derive the shape of the object based on the obtained distance value and reflect it on the driving route", [0086], "The processor 280 may estimate the object as "human"", [0082]).
Regarding claim 12, modified Gil teaches of all claim limitations of claim 1 as stated above, further comprising: a cleaning assembly configured to perform at least one operation of sweeping, vacuum suction, or mop water supply ("The cleaning head mounting unit 130 includes a drum-shaped brush unit 131 installed on the suction hole 160 at a length corresponding to the suction hole 160 and rotated in a roller manner with respect to the bottom surface to sweep or scatter dust on the floor surface, and a brush motor (not shown) for rotating the brush unit 131", [0046], "The dust storage module 235 can perform cleaning according to the dry cleaning mode, thereby isolating the dust that is sucked in the sealed space. For example, the dust storage module 235 can control the operation of accommodating the dust sucked from the suction port (for example, the suction hole 160 in FIG. 1B) into the dust container during dry cleaning. The water supply module 240 may refer to a module that controls the supply of water to the cleaning head mount 130 when a wet cleaning is performed", [0059]), wherein the at least one processor is configured to operate the cleaning assembly while driving in the driving area (“The processor 210 may operate, for example, an operating system or application programs to control a plurality of hardware or software components”, [0051], “a cleaning head mounting unit 130 in which a cleaning head which sweeps or scatters dust existing in a cleaning space is mounted”, [0044]).
However, modified Gil does not teach of in the first mode and the second mode.
Lee, in the same field of endeavor, teaches of in the first mode (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when privacy information is not included in the inputted content information, the processor may be configured to store the content information in the second region of the SoC memory”, [0019]) and the second mode (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when the inputted content information includes image information, the apparatus may further include an object recognizer configured to recognize an object from the image information, and when privacy information is included in the image information or the recognized object, the processor may be configured to store the content information in the first region.” [0020]).
Therefore, one of ordinary skill in the art, before the effective filing date of the claimed invention, would have modified the teachings of modified Gil with the teachings of Lee to have a first and second configuration of using machine learning on the cloud and locally with reasonable expectations of success. One of ordinary skill in the art would have been motivated to make this modification in order to protect the privacy of a person detected during object detection which may have included sensitive information (Lee, [0005]- [0006]).
Regarding claim 13, Gil teaches of a method of controlling a robot device ("a controlling method for a cleaning robot", [0016]), the method comprising: generating an input image of the robot device's surroundings during driving of the robot device ("the cleaning robot 100 may transmit the input data for image processing…The input data may include, for example, images taken from a 3D depth camera", [0067]); detecting a person in a driving area of the robot device ("the processor 210 may generate skeletal information of a human using the photographed image and depth information of the subject using a skeleton extraction algorithm. For example, the processor 210 can detect an arm, a leg, a torso, a face, and the like constituting a human body by using a skeleton extraction algorithm", [0054]); based on a determination that no person is present in the driving area, recognizing an object in an input image generated from the image signal using a cloud machine learning model ("Referring to operation 1130, when the object type is the object to be avoided, the cleaning robot 100 can reset the driving route in progress. Referring to operation 1035, when the type of the object is the object to be driven, the cleaning robot 100 can maintain the route in progress", [0163], "The object to be avoided may be, for example…people", [0164], "The external server 260 inputs the image data received from the cleaning robot 100 to the learning network model 270 to find an image similar to the inputted image", [0067]); based on a determination that a person is present in the driving area, recognizing the object in the input image generated from the image signal using an on-device machine learning model ("Referring to operation 1130, when the object type is the object to be avoided, the cleaning robot 100 can reset the driving route in progress. Referring to operation 1035, when the type of the object is the object to be driven, the cleaning robot 100 can maintain the route in progress", [0163], "The object to be avoided may be, for example…people", [0164], "the learning network model 270 may be integrated in a hardware chip form and become a part of the processor 210", [0065]); and controlling the driving of the robot device by using a result of recognizing the object ("Referring to operation 1130, when the object type is the object to be avoided, the cleaning robot 100 can reset the driving route in progress. Referring to operation 1035, when the type of the object is the object to be driven, the cleaning robot 100 can maintain the route in progress", [0163]), wherein the cloud machine learning model operates on a cloud server communicating with the robot device, and the on-device machine learning model operates on the robot device ("the learning network 270 may exist simultaneously in the memory (not shown) of the server 260 and the memory 230 of the cleaning robot 100", [0074]).
However, Gil does not teach of in a first mode and in a second mode.
Lee, in the same field of endeavor, teaches of in a first mode (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when privacy information is not included in the inputted content information, the processor may be configured to store the content information in the second region of the SoC memory”, [0019]), and in a second mode (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when the inputted content information includes image information, the apparatus may further include an object recognizer configured to recognize an object from the image information, and when privacy information is included in the image information or the recognized object, the processor may be configured to store the content information in the first region” [0020]).
Therefore, one of ordinary skill in the art, before the effective filing date of the claimed invention, would have modified the teachings of Gil with the teachings of Lee to have a first and second configuration of using machine learning on the cloud and locally with reasonable expectations of success. One of ordinary skill in the art would have been motivated to make this modification in order to protect the privacy of a person detected during object detection which may have included sensitive information (Lee, [0005]- [0006]).
Regarding claim 14, modified Gil teaches of all claim limitations of claim 13 as stated above, further comprising: providing a notification recommending through the output interface (“when the type of object detected by the cleaning robot 100 is estimated as an object to be avoided and the moving route is reset, a notification signal can be generated”, [0111]) when it is determined that the person is present in the driving area while operating in the first mode ("when the type of the object is estimated as a person among the object to be avoided”, [0109]), and providing a notification recommending (“when the type of object detected by the cleaning robot 100 is estimated as an object to be avoided and the moving route is reset, a notification signal can be generated”, [0111]), when it is determined that no person is present in the driving area while operating in the second mode ("when the type of the object is estimated as a person among the object to be avoided”, [0109]).
However, modified Gil does not teach of changing an operation mode to the second mode and changing the operation mode to the first mode.
Lee, in the same field of endeavor, teaches of changing an operation mode to the second mode (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when the inputted content information includes image information, the apparatus may further include an object recognizer configured to recognize an object from the image information, and when privacy information is included in the image information or the recognized object, the processor may be configured to store the content information in the first region” [0020]), and changing the operation mode to the first mode (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when privacy information is not included in the inputted content information, the processor may be configured to store the content information in the second region of the SoC memory”, [0019]).
Therefore, one of ordinary skill in the art, before the effective filing date of the claimed invention, would have modified the teachings of modified Gil with the teachings of Lee to have a first and second configuration of using machine learning on the cloud and locally with reasonable expectations of success. One of ordinary skill in the art would have been motivated to make this modification in order to protect the privacy of a person detected during object detection which may have included sensitive information (Lee, [0005]- [0006]).
Regarding claim 16, Gil teaches of a non-transitory computer readable recording medium storing instructions that ([0178]), when executed by at least one processor ("The processor 210", [0051]), cause the at least one processor to: generate an input image of the robot device's surroundings during driving of the robot device ("the cleaning robot 100 may transmit the input data for image processing…The input data may include, for example, images taken from a 3D depth camera", [0067]); detect a person in a driving area of the robot device ("the processor 210 may generate skeletal information of a human using the photographed image and depth information of the subject using a skeleton extraction algorithm. For example, the processor 210 can detect an arm, a leg, a torso, a face, and the like constituting a human body by using a skeleton extraction algorithm", [0054]), based on a determination that no person is present in the driving area, recognize an object in an input image generated from the image signal using a cloud machine learning model ("Referring to operation 1130, when the object type is the object to be avoided, the cleaning robot 100 can reset the driving route in progress. Referring to operation 1035, when the type of the object is the object to be driven, the cleaning robot 100 can maintain the route in progress", [0163], "The object to be avoided may be, for example…people", [0164], "The external server 260 inputs the image data received from the cleaning robot 100 to the learning network model 270 to find an image similar to the inputted image", [0067]), based on a determination that a person is present in the driving area, recognize the object in the input image generated from the image signal using an on- device machine learning model ("Referring to operation 1130, when the object type is the object to be avoided, the cleaning robot 100 can reset the driving route in progress. Referring to operation 1035, when the type of the object is the object to be driven, the cleaning robot 100 can maintain the route in progress", [0163], "The object to be avoided may be, for example…people", [0164], "the learning network model 270 may be integrated in a hardware chip form and become a part of the processor 210", [0065]), and control the driving of the robot device through the moving assembly by using a result of recognizing the object ("Referring to operation 1130, when the object type is the object to be avoided, the cleaning robot 100 can reset the driving route in progress. Referring to operation 1035, when the type of the object is the object to be driven, the cleaning robot 100 can maintain the route in progress", [0163]), wherein the cloud machine learning model operates on a cloud server connected through the communication interface, and the on-device machine learning model operates on the robot device ("the learning network 270 may exist simultaneously in the memory (not shown) of the server 260 and the memory 230 of the cleaning robot 100", [0074]).
However, Gil does not teach of in a first mode and in a second mode.
Lee, in the same field of endeavor, teaches of in a first mode (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when privacy information is not included in the inputted content information, the processor may be configured to store the content information in the second region of the SoC memory”, [0019]), and in a second mode (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when the inputted content information includes image information, the apparatus may further include an object recognizer configured to recognize an object from the image information, and when privacy information is included in the image information or the recognized object, the processor may be configured to store the content information in the first region” [0020]).
Therefore, one of ordinary skill in the art, before the effective filing date of the claimed invention, would have modified the teachings of Gil with the teachings of Lee to have a first and second configuration of using machine learning on the cloud and locally with reasonable expectations of success. One of ordinary skill in the art would have been motivated to make this modification in order to protect the privacy of a person detected during object detection which may have included sensitive information (Lee, [0005]- [0006]).
Regarding claim 17, modified Gil teaches of all claim limitations of claim 16 as stated above, further comprising: an output interface (“The display 255 may be, for example, positioned on the upper surface of the cleaning robot 100 to display various states of the cleaning robot 100”, [0056]), wherein the at least one processor is further configured to provide a notification through the output interface (“when the type of object detected by the cleaning robot 100 is estimated as an object to be avoided and the moving route is reset, a notification signal can be generated”, [0111]) when it is determined that the person is present in the driving area while operating in the first mode ("when the type of the object is estimated as a person among the object to be avoided”, [0109]), and provide a notification through the output interface (“when the type of object detected by the cleaning robot 100 is estimated as an object to be avoided and the moving route is reset, a notification signal can be generated”, [0111]) when it is determined that no person is present in the driving area while operating in the second mode ("when the type of the object is estimated as a person among the object to be avoided”, [0109]).
However, modified Gil does not teach of recommending changing an operation mode to the second mode and recommending changing the operation mode to the first mode.
Lee, in the same field of endeavor, teaches of recommending changing an operation mode to the second mode (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when the inputted content information includes image information, the apparatus may further include an object recognizer configured to recognize an object from the image information, and when privacy information is included in the image information or the recognized object, the processor may be configured to store the content information in the first region” [0020]), and changing the operation mode to the first mode (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when privacy information is not included in the inputted content information, the processor may be configured to store the content information in the second region of the SoC memory”, [0019]).
Therefore, one of ordinary skill in the art, before the effective filing date of the claimed invention, would have modified the teachings of modified Gil with the teachings of Lee to have a first and second configuration of using machine learning on the cloud and locally with reasonable expectations of success. One of ordinary skill in the art would have been motivated to make this modification in order to protect the privacy of a person detected during object detection which may have included sensitive information (Lee, [0005]- [0006]).
Claims 5-6 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Gil and Lee as applied above, and further in view of Fadell et al. (US20160259308A1), hereinafter Fadell.
Regarding claim 5, modified Gil teaches of all claim limitations of claim 1 as stated above.
However, modified Gil does not teach of wherein the communication interface is configured to communicate with an area management system managing a certain area including the driving area, and the at least one processor is further configured to determine that no person is present in the driving area based on receiving going out information indicating that the area management system is set to a going out mode.
Fadell, in the same field of endeavor, teaches of wherein the communication interface is configured to communicate with an area management system managing a certain area including the driving area (“a smart-home environment may be provided with smart-device environment policies that use smart-devices to monitor activities within a smart-device environment”, [0006]), and the at least one processor is further configured to determine that no person is present in the driving area based on receiving going out information indicating that the area management system is set to a going out mode (“the low-power processor 22 may detect when a location (e.g., a house or room) is occupied (i.e., includes a presence of a human)”, [0072], describes the going out mode = no one home).
Therefore, one of ordinary skill in the art, before the effective filing date of the claimed invention, would have modified the teaching of modified Gil with the teaching of Fadell to use a communication interface to communicate with an area management system and a going out mode with reasonable expectations of success. One of ordinary skill in the art would have been motivated to make this modification in order to allow a device to do account-based operations within a shared household environment (Fadell, [0064]), and to automatically modify device operation when the household is unoccupied (Fadell, [0074]).
Regarding claim 6, modified Gil teaches of all claim limitations of claim 1 as stated above.
However, modified Gil does not teach of wherein the communication interface is configured to communicate with a device management server configured to control at least one electronic device registered in a user account, and the at least one processor is further configured to determine whether the person is present in the driving area based on user location information or going out mode setting information received from another electronic device registered in the user account of the device management server.
Fadell, in the same field of endeavor, teaches of wherein the communication interface is configured to communicate with a device management server configured to control at least one electronic device registered in a user account (“some or all of the occupants… can register their device 66 with the smart-device environment 30. Such registration can be made at a central server to authenticate the occupant and/or the device”, [0087]), and the at least one processor is further configured to determine whether the person is present in the driving area based on user location information or going out mode setting information received from another electronic device registered in the user account of the device management server (“the low-power processor 22 may detect when a location (e.g., a house or room) is occupied (i.e., includes a presence of a human)”, [0072], “based on the presence detection, the high-power processor 20 may adjust device settings to, e.g., conserve power when nobody is home or in a particular room with user preferences”, [0074], teaches detecting person presence at home using smart device = user location as well as going out mode settings).
Therefore, one of ordinary skill in the art, before the effective filing date of the claimed invention, would have modified the teaching of modified Gil with the teaching of Fadell to use a communication interface to communicate with an area management system, a going out mode, and a device management server with reasonable expectations of success. One of ordinary skill in the art would have been motivated to make this modification in order to allow a device to do account-based operations within a shared household environment (Fadell, [0064]), to automatically modify device operation when the household is unoccupied (Fadell, [0074]), and to centrally manage permissions and controls of specific devices in a shared environment (Fadell, [0087]).
Regarding claim 10, modified Gil teaches of all claim limitations of claim 1 as stated above, additionally, wherein the at least one processor is further configured to: provide a notification in response to determining that the person is present in the driving area while operating in the first mode (“when the type of object detected by the cleaning robot 100 is estimated as an object to be avoided and the moving route is reset, a notification signal can be generated”, [0111], "when the type of the object is estimated as a person among the object to be avoided”, [0109]), or provide a notification in response to determining that no person is present in the driving area while operating in the second mode (“when the type of object detected by the cleaning robot 100 is estimated as an object to be avoided and the moving route is reset, a notification signal can be generated”, [0111], "when the type of the object is estimated as a person among the object to be avoided”, [0109]).
However, modified Gil does not teach of recommending changing an operation mode to the second mode, or recommending changing the operation mode to the first mode, and the notification is output through at least one device registered in a user account of a device management server connected through the communication interface.
Lee, in the same field of endeavor, teaches of recommending changing an operation mode to the second mode (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when the inputted content information includes image information, the apparatus may further include an object recognizer configured to recognize an object from the image information, and when privacy information is included in the image information or the recognized object, the processor may be configured to store the content information in the first region” [0020]), or recommending changing the operation mode to the first mode (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when privacy information is not included in the inputted content information, the processor may be configured to store the content information in the second region of the SoC memory”, [0019]).
However, modified Lee does not teach of the notification is output through at least one device registered in a user account of a device management server connected through the communication interface.
Fadell, in the same field of endeavor, teaches of the notification is output through at least one device registered in a user account of a device management server connected through the communication interface (“some or all of the occupants… can register their device 66 with the smart-device environment 30. Such registration can be made at a central server to authenticate the occupant and/or the device”, [0087]).
Therefore, one of ordinary skill in the art, before the effective filing date of the claimed invention, would have modified the teaching of modified Gil with the teaching of Lee to have a first and second configuration of using machine learning on the cloud and locally and with the teaching of Fadell to have a device management server with reasonable expectations of success. One of ordinary skill in the art would have been motivated to make this modification in order to protect the privacy of a person detected during object detection which may have included sensitive information (Lee, [0005]- [0006]), and to centrally manage permissions and controls of specific devices in a shared environment (Fadell, [0087]).
Claim 8 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Gil and Lee as applied above, and further in view of Kleiner et al. (US20180284792A1), hereinafter Kleiner.
Regarding claim 8, modified Gil teaches of all claim limitations of claim 1 as stated above.
However, modified Gil does not teach of wherein the driving area comprises one or more sub driving areas, and the at least one processor is further configured to: recognize the object by operating in the first mode in a first sub driving area in which it is determined that no person is present, wherein the first sub driving area is among the one or more sub driving areas, and recognize the object by operating in the second mode in a second sub driving area in which it is determined that the person is present, wherein the second sub driving area is among the one or more sub driving areas.
Lee, in the same field of endeavor, teaches of the at least one processor is further configured to: recognize the object by operating in the first mode in a first sub driving area in which it is determined that no person is present (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when privacy information is not included in the inputted content information, the processor may be configured to store the content information in the second region of the SoC memory”, [0019]), and recognize the object by operating in the second mode in a second sub driving area in which it is determined that the person is present (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when privacy information is not included in the inputted content information, the processor may be configured to store the content information in the second region of the SoC memory”, [0019]).
However, Lee does not teach of wherein the driving area comprises one or more sub driving areas, wherein the first sub driving area is among the one or more sub driving areas, and wherein the second sub driving area is among the one or more sub driving areas.
Kleiner, in the same field of endeavor, teaches of wherein the driving area comprises one or more sub driving areas (“a segmentation map defining respective regions of a surface based on occupancy data that is collected by a mobile robot… classifying or otherwise identifying sub-regions of at least one of the respective regions as first and second areas”, [0004]), wherein the first sub driving area is among the one or more sub driving areas (“a segmentation map defining respective regions of a surface based on occupancy data that is collected by a mobile robot… classifying or otherwise identifying sub-regions of at least one of the respective regions as first and second areas”, [0004]), and wherein the second sub driving area is among the one or more sub driving areas (“a segmentation map defining respective regions of a surface based on occupancy data that is collected by a mobile robot… classifying or otherwise identifying sub-regions of at least one of the respective regions as first and second areas”, [0004]).
Therefore, one of ordinary skill in the art, before the effective filing date of the claimed invention, would have modified the teaching of modified Gil with the teaching of Lee to have a first and second configuration of using machine learning on the cloud and locally, and with the teaching of Kleiner to include a driving area comprising of sub driving areas with reasonable expectations of success. One of ordinary skill in the art would have been motivated to make this modification in order to protect the privacy of a person detected during object detection which may have included sensitive information (Lee, [0005]- [0006]), and to improve the efficiency of the robot by operating in one area at a time (Kleiner, [0189]).
Regarding claim 11, modified Gil teaches of all claim limitations of claim 1 as stated above.
However, modified Gil does not teach of wherein the at least one processor is further configured to operate in the second mode in a privacy area, regardless of whether the person is detected, when the privacy area is set in the driving area.
Lee, in the same field of endeavor, teaches of wherein the at least one processor is further configured to operate in the second mode regardless of whether the person is detected (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when privacy information is not included in the inputted content information, the processor may be configured to store the content information in the second region of the SoC memory”, [0019]).
However, Lee does not teach of in a privacy area, when the privacy area is set in the driving area.
Kleiner, in the same field of endeavor, teaches of in a privacy area, when the privacy area is set in the driving area (“a segmentation map defining respective regions of a surface based on occupancy data that is collected by a mobile robot… classifying or otherwise identifying sub-regions of at least one of the respective regions as first and second areas”, [0004]).
Therefore, one of ordinary skill in the art, before the effective filing date of the claimed invention, would have modified the teaching of modified Gil with the teaching of Lee to have a second configuration of using machine learning locally, and with the teaching of Kleiner to include a driving area comprising of sub driving areas with reasonable expectations of success. One of ordinary skill in the art would have been motivated to make this modification in order to protect the privacy of a person detected during object detection which may have included sensitive information (Lee, [0005]- [0006]), and to improve the efficiency of the robot by operating in one area at a time (Kleiner, [0189]).
Claims 9, 15, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Gil in view of Lee, and further in view of Desai et al. (US20180114334A1), hereinafter Desai.
Regarding claim 9, modified Gil teaches of all claim limitations of claim 1 as stated above.
However, modified Gil does not teach of wherein the on-device machine learning model operates in a normal mode in the second mode, and operates in a light mode with less throughput than the normal mode in the first mode, and the at least one processor is further configured to set the on-device machine learning model to the light mode while operating in the first mode, input the input image to the on-device machine learning model set to the light mode before inputting the input image to the cloud machine learning model, determine whether the person is detected based on an output of the on- device machine learning model set to the light mode, based on determining that no person is detected as an output of the on- device machine learning model set to the light mode, input the input image to the cloud machine learning model, and based on determining that the person is detected as an output of the on- device machine learning model set to the light mode, stop inputting the input image to the cloud machine learning model.
Lee, in the same field of endeavor, teaches of wherein the on-device machine learning model operates in a normal mode in the second mode (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when the inputted content information includes image information, the apparatus may further include an object recognizer configured to recognize an object from the image information, and when privacy information is included in the image information or the recognized object, the processor may be configured to store the content information in the first region” [0020]), based on determining that no person is detected (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when the inputted content information includes image information, the apparatus may further include an object recognizer configured to recognize an object from the image information, and when privacy information is included in the image information or the recognized object, the processor may be configured to store the content information in the first region” [0020]), and determine whether the person is detected (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when privacy information is not included in the inputted content information, the processor may be configured to store the content information in the second region of the SoC memory”, [0019]), based on determining that the person is detected (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when privacy information is not included in the inputted content information, the processor may be configured to store the content information in the second region of the SoC memory”, [0019]) stop inputting the input image to the cloud machine learning model (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when the inputted content information includes image information, the apparatus may further include an object recognizer configured to recognize an object from the image information, and when privacy information is included in the image information or the recognized object, the processor may be configured to store the content information in the first region” [0020]).
However, Lee does not teach of operates in a light mode with less throughput than the normal mode in the first mode, and the at least one processor is further configured to set the on-device machine learning model to the light mode while operating in the first mode, input the input image to the on-device machine learning model set to the light mode before inputting the input image to the cloud machine learning model, based on an output of the on- device machine learning model set to the light mode, as an output of the on- device machine learning model set to the light mode, and input the input image to the cloud machine learning model, as an output of the on- device machine learning model set to the light mode.
Desai, in the same field of endeavor, teaches of operates in a light mode with less throughput than the normal mode in the first mode (“Simplified versions of machine learning models have also been ported to run on mainstream mobile platforms… These models have less accuracy than their backend server counterparts and / or are designed to perform specialized object recognition tasks” [0029]), and the at least one processor is further configured to set the on-device machine learning model to the light mode while operating in the first mode (“Simplified versions of machine learning models have also been ported to run on mainstream mobile platforms… These models have less accuracy than their backend server counterparts and / or are designed to perform specialized object recognition tasks” [0029]), input the input image to the on-device machine learning model set to the light mode before inputting the input image to the cloud machine learning model (“existing general object recognition tasks typically send images captured by the mobile devices to backend systems such as powerful server machines and / or graphical processing units ( GPUs ) in a cloud environment that perform the compute - intensive image processing tasks”, [0026], for tasks that are compute-intensive send to the cloud, otherwise keep using the on-device models), based on an output of the on- device machine learning model set to the light mode (“Simplified versions of machine learning models have also been ported to run on mainstream mobile platforms… These models have less accuracy than their backend server counterparts and / or are designed to perform specialized object recognition tasks” [0029]) as an output of the on- device machine learning model set to the light mode (“Simplified versions of machine learning models have also been ported to run on mainstream mobile platforms… These models have less accuracy than their backend server counterparts and / or are designed to perform specialized object recognition tasks” [0029]), input the input image to the cloud machine learning model (“existing general object recognition tasks typically send images captured by the mobile devices to backend systems such as powerful server machines and / or graphical processing units ( GPUs ) in a cloud environment that perform the compute - intensive image processing tasks”, [0026], for tasks that are compute-intensive send to the cloud, otherwise keep using the on-device models), as an output of the on- device machine learning model set to the light mode (“Simplified versions of machine learning models have also been ported to run on mainstream mobile platforms… These models have less accuracy than their backend server counterparts and / or are designed to perform specialized object recognition tasks” [0029]).
Therefore, one of ordinary skill in the art, before the effective filing date of the claimed invention, would have modified the teaching of modified Gil with the teaching of Lee to have a first and second configuration of using machine learning on the cloud and locally, and with the teaching of Desai to utilize a light mode that can switch to the cloud machine learning model with reasonable expectations of success. One of ordinary skill in the art would have been motivated to make this modification in order to protect the privacy of a person detected during object detection which may have included sensitive information (Lee, [0005]- [0006]), and to scale computational power without having to change on-board hardware (Desai, [0161]).
Regarding claim 15, modified Gil teaches of all claim limitations of claim 13 as stated above.
However, modified Gil does not teach of wherein the on-device machine learning model operates in a normal mode in the second mode, and operates in a light mode with less throughput than the normal mode in the first mode, and the at least one processor is further configured to set the on-device machine learning model to the light mode while operating in the first mode, input the input image to the on-device machine learning model set to the light mode before inputting the input image to the cloud machine learning model, determine whether the person is detected based on an output of the on- device machine learning model set to the light mode, based on determining that no person is detected as an output of the on- device machine learning model set to the light mode, input the input image to the cloud machine learning model, and based on determining that the person is detected as an output of the on- device machine learning model set to the light mode, stop inputting the input image to the cloud machine learning model.
Lee, in the same field of endeavor, teaches of wherein the on-device machine learning model operates in a normal mode in the second mode (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when the inputted content information includes image information, the apparatus may further include an object recognizer configured to recognize an object from the image information, and when privacy information is included in the image information or the recognized object, the processor may be configured to store the content information in the first region” [0020]), based on determining that no person is detected (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when the inputted content information includes image information, the apparatus may further include an object recognizer configured to recognize an object from the image information, and when privacy information is included in the image information or the recognized object, the processor may be configured to store the content information in the first region” [0020]), and determine whether the person is detected (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when privacy information is not included in the inputted content information, the processor may be configured to store the content information in the second region of the SoC memory”, [0019]), based on determining that the person is detected (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when privacy information is not included in the inputted content information, the processor may be configured to store the content information in the second region of the SoC memory”, [0019]) stop inputting the input image to the cloud machine learning model (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when the inputted content information includes image information, the apparatus may further include an object recognizer configured to recognize an object from the image information, and when privacy information is included in the image information or the recognized object, the processor may be configured to store the content information in the first region” [0020]).
However, Lee does not teach of operates in a light mode with less throughput than the normal mode in the first mode, and the at least one processor is further configured to set the on-device machine learning model to the light mode while operating in the first mode, input the input image to the on-device machine learning model set to the light mode before inputting the input image to the cloud machine learning model, based on an output of the on- device machine learning model set to the light mode, as an output of the on- device machine learning model set to the light mode, and input the input image to the cloud machine learning model, as an output of the on- device machine learning model set to the light mode.
Desai, in the same field of endeavor, teaches of operates in a light mode with less throughput than the normal mode in the first mode (“Simplified versions of machine learning models have also been ported to run on mainstream mobile platforms… These models have less accuracy than their backend server counterparts and / or are designed to perform specialized object recognition tasks” [0029]), and the at least one processor is further configured to set the on-device machine learning model to the light mode while operating in the first mode (“Simplified versions of machine learning models have also been ported to run on mainstream mobile platforms… These models have less accuracy than their backend server counterparts and / or are designed to perform specialized object recognition tasks” [0029]), input the input image to the on-device machine learning model set to the light mode before inputting the input image to the cloud machine learning model (“existing general object recognition tasks typically send images captured by the mobile devices to backend systems such as powerful server machines and / or graphical processing units ( GPUs ) in a cloud environment that perform the compute - intensive image processing tasks”, [0026], for tasks that are compute-intensive send to the cloud, otherwise keep using the on-device models), based on an output of the on- device machine learning model set to the light mode (“Simplified versions of machine learning models have also been ported to run on mainstream mobile platforms… These models have less accuracy than their backend server counterparts and / or are designed to perform specialized object recognition tasks” [0029]) as an output of the on- device machine learning model set to the light mode (“Simplified versions of machine learning models have also been ported to run on mainstream mobile platforms… These models have less accuracy than their backend server counterparts and / or are designed to perform specialized object recognition tasks” [0029]), input the input image to the cloud machine learning model (“existing general object recognition tasks typically send images captured by the mobile devices to backend systems such as powerful server machines and / or graphical processing units ( GPUs ) in a cloud environment that perform the compute - intensive image processing tasks”, [0026], for tasks that are compute-intensive send to the cloud, otherwise keep using the on-device models), as an output of the on- device machine learning model set to the light mode (“Simplified versions of machine learning models have also been ported to run on mainstream mobile platforms… These models have less accuracy than their backend server counterparts and / or are designed to perform specialized object recognition tasks” [0029]).
Therefore, one of ordinary skill in the art, before the effective filing date of the claimed invention, would have modified the teaching of modified Gil with the teaching of Lee to have a first and second configuration of using machine learning on the cloud and locally, and with the teaching of Desai to utilize a light mode that can switch to the cloud machine learning model with reasonable expectations of success. One of ordinary skill in the art would have been motivated to make this modification in order to protect the privacy of a person detected during object detection which may have included sensitive information (Lee, [0005]- [0006]), and to scale computational power without having to change on-board hardware (Desai, [0161]).
Regarding claim 18, modified Gil teaches of all claim limitations of claim 16 as stated above.
However, modified Gil does not teach of wherein the instructions further cause the at least one processor to: set the on-device machine learning model to the light mode while operating in the first mode, input the input image to the on-device machine learning model set to the light mode before inputting the input image to the cloud machine learning model, determine whether the person is detected based on an output of the on- device machine learning model set to the light mode, based on determining that no person is detected as an output of the on- device machine learning model set to the light mode, input the input image to the cloud machine learning model, and based on determining that the person is detected as an output of the on- device machine learning model set to the light mode, stop inputting the input image to the cloud machine learning model.
Lee, in the same field of endeavor, teaches of wherein the instructions further cause the at least one processor to: based on determining that no person is detected (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when the inputted content information includes image information, the apparatus may further include an object recognizer configured to recognize an object from the image information, and when privacy information is included in the image information or the recognized object, the processor may be configured to store the content information in the first region” [0020]), and determine whether the person is detected (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when privacy information is not included in the inputted content information, the processor may be configured to store the content information in the second region of the SoC memory”, [0019]), based on determining that the person is detected (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when privacy information is not included in the inputted content information, the processor may be configured to store the content information in the second region of the SoC memory”, [0019]) stop inputting the input image to the cloud machine learning model (“tracks a moving person using a plurality of tracking modules, thereby increasing the accuracy of object recognition”, [0004], “when the inputted content information includes image information, the apparatus may further include an object recognizer configured to recognize an object from the image information, and when privacy information is included in the image information or the recognized object, the processor may be configured to store the content information in the first region” [0020]).
However, Lee does not teach of set the on-device machine learning model to the light mode while operating in the first mode, input the input image to the on-device machine learning model set to the light mode before inputting the input image to the cloud machine learning model, based on an output of the on- device machine learning model set to the light mode, as an output of the on- device machine learning model set to the light mode, and input the input image to the cloud machine learning model, as an output of the on- device machine learning model set to the light mode.
Desai, in the same field of endeavor, teaches of set the on-device machine learning model to the light mode while operating in the first mode (“Simplified versions of machine learning models have also been ported to run on mainstream mobile platforms… These models have less accuracy than their backend server counterparts and / or are designed to perform specialized object recognition tasks” [0029]), input the input image to the on-device machine learning model set to the light mode before inputting the input image to the cloud machine learning model (“existing general object recognition tasks typically send images captured by the mobile devices to backend systems such as powerful server machines and / or graphical processing units ( GPUs ) in a cloud environment that perform the compute - intensive image processing tasks”, [0026], for tasks that are compute-intensive send to the cloud, otherwise keep using the on-device models), based on an output of the on- device machine learning model set to the light mode (“Simplified versions of machine learning models have also been ported to run on mainstream mobile platforms… These models have less accuracy than their backend server counterparts and / or are designed to perform specialized object recognition tasks” [0029]) as an output of the on- device machine learning model set to the light mode (“Simplified versions of machine learning models have also been ported to run on mainstream mobile platforms… These models have less accuracy than their backend server counterparts and / or are designed to perform specialized object recognition tasks” [0029]), input the input image to the cloud machine learning model (“existing general object recognition tasks typically send images captured by the mobile devices to backend systems such as powerful server machines and / or graphical processing units ( GPUs ) in a cloud environment that perform the compute - intensive image processing tasks”, [0026], for tasks that are compute-intensive send to the cloud, otherwise keep using the on-device models), as an output of the on- device machine learning model set to the light mode (“Simplified versions of machine learning models have also been ported to run on mainstream mobile platforms… These models have less accuracy than their backend server counterparts and / or are designed to perform specialized object recognition tasks” [0029]).
Therefore, one of ordinary skill in the art, before the effective filing date of the claimed invention, would have modified the teaching of modified Gil with the teaching of Lee to have a first and second configuration of using machine learning on the cloud and locally, and with the teaching of Desai to utilize a light mode that can switch to the cloud machine learning model with reasonable expectations of success. One of ordinary skill in the art would have been motivated to make this modification in order to protect the privacy of a person detected during object detection which may have included sensitive information (Lee, [0005]- [0006]), and to scale computational power without having to change on-board hardware (Desai, [0161]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ABIGAIL LEE ESPINOZA whose telephone number is (571)272-4889. The examiner can normally be reached Monday - Friday 9:00 am - 5:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Adam Mott can be reached at (571) 270-5376. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
ABIGAIL LEE ESPINOZA
Examiner
Art Unit 3657
/ADAM R MOTT/Supervisory Patent Examiner, Art Unit 3657