DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 4 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding Claim 4, the claim depends on Claim 4 (itself).
This is likely due to a typographical error but nonetheless makes the scope of the claim indefinite, as it is unclear which claim is intended as the parent. For examining purposes, the claim will be treated as dependent on Claim 1.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 5, 8-9, 12-13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ha et al (US 20220371198, hereinafter Ha) in view of Kawamura et al (US 20190272580, hereinafter Kawamura).
Regarding Claim 1, Ha teaches:
a mobile autonomous device (see at least "The patrolling robot 300 according to one embodiment of the invention may include at least one of a module (e.g., a grab or a robotic arm module) for loading and unloading a serving object (e.g., a food tray), an imaging module (e.g., a visible light camera or an infrared camera) for acquiring images of surroundings" in par. 0030 and “For example, the present invention may also be applied to a robot patrolling in a common restaurant.” In par. 0043), comprising:
an autonomous vehicle unit comprising a propulsion system (see at least "Next, referring to FIG. 3 again, the drive unit 320 according to one embodiment of the invention may comprise a module for moving the main body 310 to other locations. For example, the drive unit 320 may include a module related to electrically, mechanically, or hydraulically driven wheels, propellers, or the like as the module for moving the main body 310 to other locations." in par. 0039) ;
a plurality of sensor devices comprising at least a camera device (see at least "an imaging module (e.g., a visible light camera or an infrared camera) for acquiring images of surroundings, a scanner module (e.g., a LIDAR sensor) for acquiring information on obstacles, a sound acquisition module (e.g., a microphone) for acquiring sounds of surroundings" in par. 0030) ;
at least one processor (see at least "processor 330" in par. 0040 ) ; and
memory that stores computer-executable instructions that, as a result of being executed by the at least one processor, cause the mobile autonomous device to (see memory storing instructions in par. 0086):
collect data of objects within the space using the plurality of sensor devices as the autonomous vehicle unit travels via the propulsion system (see at least "an imaging module (e.g., a visible light camera or an infrared camera) for acquiring images of surroundings, a scanner module (e.g., a LIDAR sensor) for acquiring information on obstacles, a sound acquisition module (e.g., a microphone) for acquiring sounds of surroundings" in par. 0030 and "For example, the present invention may also be applied to a robot patrolling in a common restaurant.” In par. 0043) , the data comprising:
a map of the space indicating locations of one or more of the objects within the space; (see at least "(see at least "According to one embodiment of the invention, the information on the location of the patrolling robot 300 may refer to, but is not limited to, information on coordinates of the patrolling robot 300 in a map for the patrolling place." in par. 0053 and “For example, when tables are arranged in the patrolling place, the situation information management unit 210 according to one embodiment of the invention may specify each of the tables (or its vicinity) as a fixed space, and when the patrolling place is formed such that the attendees can move freely as shown in FIG. 5, the spaces may be dynamically specified on the basis of states in which the attendees are gathered.” In par. 0064)
identifications of one or more of the objects within the space (see at least "As yet another example, the situation information management unit 210 according to one embodiment of the invention may specifically recognize a serving object placed on or removed from the support by processing the image information on the support using a machine learning-based object recognition model for serving objects. Here, according to one embodiment of the invention, the object recognition model may be implemented using an algorithm such as R-CNN (Region-based Convolutional Neural Network), YOLO (You Only Look Once), and SSD (Single Shot Detector)." in par. 0059 and “Further, the situation information management unit 210 according to one embodiment of the invention may acquire at least one of a number of attendees in each space of the patrolling place and a patrolling record at the patrolling place as second situation information on the patrolling place.” In par. 0060 ) ;
a condition associated with each of the one or more objects (see at least "Specifically, the situation information management unit 210 according to one embodiment of the invention may estimate at least one of demand for serving and demand for bussing in a specific space, on the basis of at least one of a ratio of an amount of serving objects served or bussed in the specific space and a number of attendees in the space, an amount of time having elapsed from the last visit of the patrolling robot 300 for serving or bussing in the specific space, an amount of time spent during the last visit (the number of attendees in the space may be considered together), an amount of time having elapsed from the last serving or bussing of the patrolling robot 300 in the specific space, a location of a space where serving objects served by the patrolling robot 300 have run out, and a location of a space full of serving objects to be retrieved (i.e., bussed) by the patrolling robot 300." in par. 0067) ;
compare the data to historical data (see at least "Specifically, the situation information management unit 210 according to one embodiment of the invention may estimate at least one of demand for serving and demand for bussing in a specific space, on the basis of at least one of a ratio of an amount of serving objects served or bussed in the specific space and a number of attendees in the space, an amount of time having elapsed from the last visit of the patrolling robot 300 for serving or bussing in the specific space, an amount of time spent during the last visit (the number of attendees in the space may be considered together), an amount of time having elapsed from the last serving or bussing of the patrolling robot 300 in the specific space, a location of a space where serving objects served by the patrolling robot 300 have run out, and a location of a space full of serving objects to be retrieved (i.e., bussed) by the patrolling robot 300." in par. 0067); and
based on the comparison between the data and the historical data, identify one or more discrepancies, wherein the one or more discrepancies comprise one or more of a change in location, a change in condition, and a change in presence of objects within the space; and (see at least "Specifically, the situation information management unit 210 according to one embodiment of the invention may estimate at least one of demand for serving and demand for bussing in a specific space, on the basis of at least one of a ratio of an amount of serving objects served or bussed in the specific space and a number of attendees in the space, an amount of time having elapsed from the last visit of the patrolling robot 300 for serving or bussing in the specific space, an amount of time spent during the last visit (the number of attendees in the space may be considered together), an amount of time having elapsed from the last serving or bussing of the patrolling robot 300 in the specific space, a location of a space where serving objects served by the patrolling robot 300 have run out, and a location of a space full of serving objects to be retrieved (i.e., bussed) by the patrolling robot 300." in par. 0067)
Ha does not appear to explicitly teach all of the following, but Kawamura does teach:
generate a notification indicative of the one or more discrepancies. (see at least " The second controller is configured to compare the image data received by the second communication at first timing with the image data received by the second communication interface at second timing after the first timing, determine a remaining amount of the consumable item based on a comparison result of the image data received at the first and second timing, and generate recommendation information upon the determining remaining amount decreasing to a predetermined threshold, and control the second communication interface to transmit the generated recommendation information. " in par. 0019 and “Meanwhile, according to the operations on the recommendation menu button D23, the menu addition button D24, and the last order button D25, a plurality of items according to the recommendation information corresponding to the respective operations may be displayed on a sub screen in a list form, and a recommendation target item may be selected by the employee. The order history button D26 is used to display the order history, for example, for today in a list form.” In par. 0115 and “On the other hand, if it is determined that one of the recommendation menu button D23, the menu addition button D24, and the last order button D25 is operated, the CPU 16A outputs the recommendation information of the item corresponding to the operated button to the order reception terminal 22 on the table selected as the display target of the video image, and notifies the table robot 24 of output of the recommendation information (Act D7).” In par. 0117)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method taught by Ha to incorporate the teachings of Kawamura wherein based on images from the table robot cameras, the system automatically populates a hall controller display to generate a list of menu items the employees can choose to recommend when a customer’s food or beverage runs out. The motivation to incorporate the teachings of Kawamura would be to enable more suitable items to be recommended to customers and at better timing (see par. 0039), which improves the chance of sales and the customer experience.
Regarding Claim 2, Ha as modified by Kawamura (references to Ha) teaches:
The mobile autonomous device of claim 1,
wherein the condition comprises one or more of cleanliness,
shape, (see at least " As yet another example, the situation information management unit 210 according to one embodiment of the invention may specifically recognize a serving object placed on or removed from the support by processing the image information on the support using a machine learning-based object recognition model for serving objects. Here, according to one embodiment of the invention, the object recognition model may be implemented using an algorithm such as R-CNN (Region-based Convolutional Neural Network), YOLO (You Only Look Once), and SSD (Single Shot Detector). However, the object recognition model is not necessarily limited to the foregoing and may be diversely changed as long as the objects of the invention may be achieved." in par. 0059)
whether the object has power or is operating.
Regarding Claim 5, Ha as modified by Kawamura (references to Ha) teaches:
the mobile autonomous device of claim 1,
wherein the at least one processor is further configured to trigger a corrective action based on the one or more discrepancies. (see at least and “Further, according to one embodiment of the invention, the task may be more specifically determined as, for example, departing from the kitchen for serving, continuing serving, returning to the kitchen for serving, switching from serving to bussing, continuing bussing, returning to the washing room for bussing, departing from the washing room for bussing, or switching from bussing to serving.” In par. 0074 and " Specifically, the task and travel management unit 220 according to one embodiment of the invention may determine the travel route of the patrolling robot 300 such that the patrolling robot 300 preferentially visits a space in which the demand for serving objects is higher." in par. 0076 )
Regarding Claim 8, Ha as modified by Kawamura (references to Ha) teaches:
the mobile autonomous device of claim 1,
wherein the space comprises at least one of a hotel room, a retail space, and a restaurant. (see restaurant in par. 0043)
Regarding Claim 9, Ha teaches:
a computer implemented method (see at least "method" in par. 0010 ) , comprising:
collecting, by an autonomous robot comprising at least one camera device, ground truth data of a restaurant space (see at least "The patrolling robot 300 according to one embodiment of the invention may include at least one of a module (e.g., a grab or a robotic arm module) for loading and unloading a serving object (e.g., a food tray), an imaging module (e.g., a visible light camera or an infrared camera) for acquiring images of surroundings" in par. 0030 and “For example, the present invention may also be applied to a robot patrolling in a common restaurant.” In par. 0043) , the ground truth data comprising:
information identifying a plurality of seating areas within the restaurant space; (see at least “For example, referring to FIG. 5, the situation information management unit 210 according to one embodiment of the invention may specify partial spaces of the patrolling place as a first space 510, a second space 520, and a third space 530 on the basis of states in which the attendees are gathered in the patrolling place, and may estimate the number of attendees contained in each of the spaces 510, 520, and 530 to acquire the numbers of attendees (i.e., 6, 4, and 2) in the respective spaces 510, 520, and 530 of the patrolling place.” In par. 0063 and " For example, when tables are arranged in the patrolling place, the situation information management unit 210 according to one embodiment of the invention may specify each of the tables (or its vicinity) as a fixed space, and when the patrolling place is formed such that the attendees can move freely as shown in FIG. 5, the spaces may be dynamically specified on the basis of states in which the attendees are gathered." in par. 0064)
occupancy status information for each of the plurality of seating areas indicating whether each seating area is empty, reserved, or occupied (see at least “For example, referring to FIG. 5, the situation information management unit 210 according to one embodiment of the invention may specify partial spaces of the patrolling place as a first space 510, a second space 520, and a third space 530 on the basis of states in which the attendees are gathered in the patrolling place, and may estimate the number of attendees contained in each of the spaces 510, 520, and 530 to acquire the numbers of attendees (i.e., 6, 4, and 2) in the respective spaces 510, 520, and 530 of the patrolling place.” In par. 0063 and " For example, when tables are arranged in the patrolling place, the situation information management unit 210 according to one embodiment of the invention may specify each of the tables (or its vicinity) as a fixed space, and when the patrolling place is formed such that the attendees can move freely as shown in FIG. 5, the spaces may be dynamically specified on the basis of states in which the attendees are gathered." in par. 0064); and
for occupied seating areas, customer satisfaction data comprising detected attributes of customers at the occupied seating areas; (see at least " For example, the situation information management unit 210 according to one embodiment of the invention may estimate demand for serving (or bussing) in a specific space to be higher as a value obtained by dividing an amount of serving objects served (or bussed) in the space by a number of attendees in the space is smaller; an amount of time having elapsed from the last visit of the patrolling robot 300 for serving or bussing in the space is greater; an amount of time spent by the patrolling robot 300 during the last visit is smaller (compared to the number of attendees in the space); or an amount of time having elapsed from the last serving (or bussing) of the patrolling robot 300 in the space is greater." in par. 0068)
generating a map representation of the restaurant space including locations of the plurality of seating areas and (see at least "According to one embodiment of the invention, the information on the location of the patrolling robot 300 may refer to, but is not limited to, information on coordinates of the patrolling robot 300 in a map for the patrolling place." in par. 0053 and “For example, when tables are arranged in the patrolling place, the situation information management unit 210 according to one embodiment of the invention may specify each of the tables (or its vicinity) as a fixed space, and when the patrolling place is formed such that the attendees can move freely as shown in FIG. 5, the spaces may be dynamically specified on the basis of states in which the attendees are gathered.” In par. 0064)
comparing the ground truth data to a set of service criteria to identify one or more seating areas requiring attention based on the customer satisfaction data; (see at least "Specifically, the situation information management unit 210 according to one embodiment of the invention may estimate at least one of demand for serving and demand for bussing in a specific space, on the basis of at least one of a ratio of an amount of serving objects served or bussed in the specific space and a number of attendees in the space, an amount of time having elapsed from the last visit of the patrolling robot 300 for serving or bussing in the specific space, an amount of time spent during the last visit (the number of attendees in the space may be considered together), an amount of time having elapsed from the last serving or bussing of the patrolling robot 300 in the specific space, a location of a space where serving objects served by the patrolling robot 300 have run out, and a location of a space full of serving objects to be retrieved (i.e., bussed) by the patrolling robot 300." in par. 0067)
determining that at least one seating area of the one or more seating areas requiring attention satisfies an urgency threshold; and (see at least "the patrolling record may include a serving record for serving objects (e.g., a location of a space that the patrolling robot 300 has visited for serving, a visit time, an amount of time spent during the visit, an amount of served serving objects, a serving time, an amount of time having elapsed from the last visit, an amount of time having elapsed from the last serving, a location of a space where serving objects have run out, and the like) and a bussing record for serving objects (e.g., a location of a space that the patrolling robot 300 has visited for bussing, a visit time, an amount of time spent during the visit, an amount of bussed serving objects, a bussing time, an amount of time having elapsed from the last visit, an amount of time having elapsed from the last bussing, a location of a space full of serving objects to be retrieved, and the like)." in par. 0065 and “Further, according to one embodiment of the invention, the task may be more specifically determined as, for example, departing from the kitchen for serving, continuing serving, returning to the kitchen for serving, switching from serving to bussing, continuing bussing, returning to the washing room for bussing, departing from the washing room for bussing, or switching from bussing to serving.” In par. 0074)
Ha does not appear to explicitly teach all of the following, but Kawamura does teach:
generating a map representation of the restaurant space including locations of the plurality of seating areas and visual indicators of the occupancy status information and customer satisfaction data (see at least " The hall controller 16 manages, for example, a situation of a table where the customer takes a seat, customer information (number of persons, gender, age, job, and the like), an order history, an occupied time (time elapsed from taking the seat, a remaining time up to a setting end time for a prix-fix menu), and the like. The hall controller 16 displays a screen for showing the situation of the customer. Employees, e.g., waitress and waiter, of the restaurant look at the screen of the hall controller 16. The hall controller 16 inputs imaging data indicating an area, including a table on which at least an item (beverage, food, and the like) is placed, as an image capturing area, and detects a remaining quantity of the item by performing an image processing based on the imaging data. The hall controller 16 has a function of outputting recommendation information used to recommend a suitable item to the customer at an appropriate timing based on the remaining quantity of the item determined according to a result of the image processing. The hall controller 16 inputs imaging data captured by a camera provided, for example, in the table robot 24 (or the order reception terminal 22) on the table. The imaging data may be any of a still image or a moving image (video). " in par. 0026 and “The hall controller 16 displays, for example, on a floor map displaying the table arrangement, information indicating the current location (operational situation) of the floor robot 26 on a path along which the floor robot 26 travels based on the locational data received from the floor robot 26.” In par. 0035 and “The detail screen D2 illustrated in FIG. 13 includes, for example, customer management information D21, a video image display area D22, and a plurality of function selection buttons D23 to D29. The customer management information D21 including the customer information shows information (an ordered course menu and remaining time determined for the course menu) indicative of a current situation of the customer. In the video image display area D22, a video image according to the imaging data received from the table robot 24 is displayed.” In par. 0114 and Fig. 12-13)
generating, based on the determining, a notification in a graphical user interface indicating the at least one seating area requiring urgent attention. (see at least " The second controller is configured to compare the image data received by the second communication at first timing with the image data received by the second communication interface at second timing after the first timing, determine a remaining amount of the consumable item based on a comparison result of the image data received at the first and second timing, and generate recommendation information upon the determining remaining amount decreasing to a predetermined threshold, and control the second communication interface to transmit the generated recommendation information. " in par. 0019 and “Meanwhile, according to the operations on the recommendation menu button D23, the menu addition button D24, and the last order button D25, a plurality of items according to the recommendation information corresponding to the respective operations may be displayed on a sub screen in a list form, and a recommendation target item may be selected by the employee. The order history button D26 is used to display the order history, for example, for today in a list form.” In par. 0115 and “On the other hand, if it is determined that one of the recommendation menu button D23, the menu addition button D24, and the last order button D25 is operated, the CPU 16A outputs the recommendation information of the item corresponding to the operated button to the order reception terminal 22 on the table selected as the display target of the video image, and notifies the table robot 24 of output of the recommendation information (Act D7).” In par. 0117)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method taught by Ha to incorporate the teachings of Kawamura wherein based on images from the table robot cameras, the system automatically populates a hall controller display to show employees a map with the status of each seat around the restaurant tables and generates a list of menu items the employees can choose to recommend when a customer’s food or beverage runs out. The motivation to incorporate the teachings of Kawamura would be to enable more suitable items to be recommended to customers and at better timing (see par. 0039), which improves the chance of sales and the customer experience.
Regarding Claim 12, Ha as modified by Kawamura (references to Ha) teaches:
the computer implemented method of claim 9, further comprising:
comparing the ground truth data to historical ground truth data associated with the plurality of seating areas; and identifying at least one discrepancy. (see at least " Further, the situation information management unit 210 according to one embodiment of the invention may acquire the patrolling record at the patrolling place as the second situation information on the patrolling place. The situation information management unit 210 according to one embodiment of the invention may acquire the patrolling record on the basis of the above-described first situation information on the patrolling robot 300, and the patrolling record may include a serving record for serving objects (e.g., a location of a space that the patrolling robot 300 has visited for serving, a visit time, an amount of time spent during the visit, an amount of served serving objects, a serving time, an amount of time having elapsed from the last visit, an amount of time having elapsed from the last serving, a location of a space where serving objects have run out, and the like) and a bussing record for serving objects (e.g., a location of a space that the patrolling robot 300 has visited for bussing, a visit time, an amount of time spent during the visit, an amount of bussed serving objects, a bussing time, an amount of time having elapsed from the last visit, an amount of time having elapsed from the last bussing, a location of a space full of serving objects to be retrieved, and the like). The situation information management unit 210 according to one embodiment of the invention may acquire the patrolling record for each space of the patrolling place, and may acquire the patrolling record for each serving object when the serving object is specifically recognized." in par. 0065)
Regarding Claim 13, Ha as modified by Kawamura (references to Ha) teaches:
the computer implemented method of claim 9, further comprising:
determining that the at least one discrepancy satisfies the urgency threshold; (see at least "the patrolling record may include a serving record for serving objects (e.g., a location of a space that the patrolling robot 300 has visited for serving, a visit time, an amount of time spent during the visit, an amount of served serving objects, a serving time, an amount of time having elapsed from the last visit, an amount of time having elapsed from the last serving, a location of a space where serving objects have run out, and the like) and a bussing record for serving objects (e.g., a location of a space that the patrolling robot 300 has visited for bussing, a visit time, an amount of time spent during the visit, an amount of bussed serving objects, a bussing time, an amount of time having elapsed from the last visit, an amount of time having elapsed from the last bussing, a location of a space full of serving objects to be retrieved, and the like)." in par. 0065)
triggering a corrective action based on the at least one discrepancy. (see at least and “Further, according to one embodiment of the invention, the task may be more specifically determined as, for example, departing from the kitchen for serving, continuing serving, returning to the kitchen for serving, switching from serving to bussing, continuing bussing, returning to the washing room for bussing, departing from the washing room for bussing, or switching from bussing to serving.” In par. 0074 and " Specifically, the task and travel management unit 220 according to one embodiment of the invention may determine the travel route of the patrolling robot 300 such that the patrolling robot 300 preferentially visits a space in which the demand for serving objects is higher." in par. 0076)
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ha et al (US 20220371198, hereinafter Ha) in view of Kawamura et al (US 20190272580, hereinafter Kawamura) and Francis et al (US 8447863, hereinafter Francis)
Regarding Claim 3, Ha as modified by Kawamura teaches:
The mobile autonomous device of claim 1,
Ha and Kawamura do not appear to explicitly teach all of the following, but Francis does teach:
wherein the at least one processor is further configured to determine distances between at least some of the one or more objects. (see at least "Contextual information may include information that has been observed via one or more sensors and that relates to an object's location, environment, and/or surroundings. In some embodiments, contextual information may include (1) a map of one or more areas and objects associated with the map; (2) previously recognized objects and/or uses of the objects; and/or (3) an inventory of one or more objects in the area. The cloud may analyze the contextual information to infer, predict, or otherwise recognize an object. For example, the robot may send a query to the cloud that includes the location of the object, location of the robot, images or the identity of objects within a predetermined distance of the object, the time of day, a description of the area in which the object is located, etc. " in col. 18 lines 52-65 and “For example, the server may use a machine learning algorithm, or any probability algorithm in general, to estimate the probability that the object is a toolbox. The estimation may be based at least in part on images of the object and/or an identifier associated with the object, for example. In embodiments, the server may also estimate the probability that the object is a toolbox based on an inventory of objects in the area, a historical record or log of the objects that were located at or near the current location of the object, etc. One or more of these probabilities may be weighted and combined to determine the probability that the recognized object in this example is a toolbox rather than another type of box.” In col. 25 lines 22-32 )
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method taught by Ha as modified by Kawamura to incorporate the teachings of Francis wherein the mobile robot recognizes objects within a predetermined distance of an unidentified object to provide context for identifying the unidentified object. The motivation to incorporate the teachings of Francis would be to improve the likelihood of correctly identifying the object and how the robot should interact with it (see col. 24 line 65 to col. 25 line 6)
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ha et al (US 20220371198, hereinafter Ha) in view of Kawamura et al (US 20190272580, hereinafter Kawamura) and Okamoto et al (JP 2007111854, hereinafter Okamoto)
Regarding Claim 4, Ha as modified by Kawamura teaches:
the mobile autonomous device of claim 1,
Ha and Kawamura do not appear to explicitly teach all of the following, but Okamoto does teach:
wherein the historical data comprises a desired state for the one or more objects, wherein the desired state of an object comprises a minimum or maximum distance between the object and another object of the one or more objects. (see at least " Further, when the father goes out of the room, the father operates the operation terminal 103 to return the can juice 21 moved by his son to P1 (x1, y1, z1) on the original table. An instruction is given (the instruction for the robot 102 will be described later). The robot 102 that has received the instruction moves to the can juice position P2 (x2, y2, z2), and grips the can juice at time t4." in par. and “Equipment attribute data related to equipment, such as furniture, is recorded on the data (surface 1, surface 2) of each surface constituting the facility, the type of facility, and when the facility has a surface on which an article can be placed. This includes the shape and posture of the main article placed. Specifically, for example, the data of the surfaces constituting the equipment are expressed as follows. ((X1, Y1, Z1), (X2, Y2, Z2), (X3, Y3, Z3), 1,400)
Here, the first three sets of coordinate values represent the real world coordinates of each vertex constituting the surface. The next value (1) is a flag indicating whether or not an article can be placed on the surface. “1” indicates that the article can be placed and “0” indicates that the article cannot be placed. The last value (400) indicates the upper limit height (mm) of an article that can be placed when the article can be placed on the surface. For example, when the surface is a table top, the distance from the top to the ceiling is the upper limit height, and when the surface is a shelf on the bookshelf, The distance is the upper limit height.”) and “Further, when moving to the location of the facility specified by the facility ID, a route that approaches the facility to a predetermined distance is created.” On page 8 And “Therefore, a posture (posture 3) that matches the other book 57 and fits in the installation space of the bookshelf 55 is selected, and the gripped book 50 is changed to that posture and installed. Whether or not the installation space is limited can be determined by, for example, the space where the installation space is partitioned or the maximum length of the installation space being shorter than the maximum length of the article. In addition to installing books on a bookshelf, the present invention can be applied to the case where dishes are efficiently installed on a cupboard or a dishwasher.” On page 29)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method taught by Ha as modified by Kawamura to incorporate the teachings of Okamoto wherein the robot saves the previous locations of items relative to storage locations and minimum spacing needed so that it can return the items in a way that fits into the storage locations when prompted. The motivation to incorporate the teachings of Okamoto would be to more efficiently store items in their expected locations like shelves or cupboards. (see page 29)
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ha et al (US 20220371198, hereinafter Ha) in view of Kawamura et al (US 20190272580, hereinafter Kawamura) and Buibas et al (US 20200380701, hereinafter Buibas)
Regarding Claim 6, Ha as modified by Kawamura teaches:
the mobile autonomous device of claim 1,
Ha and Kawamura do not appear to explicitly teach all of the following, but Buibas does teach:
wherein the corrective action comprises employing at least one ultraviolet lamp of the autonomous vehicle unit. (see at least " Cleaning actions may have any effects including for example cosmetic effects or sanitizing, sterilizing, or disinfecting effects. Processor 130 may then transmit commands 7025 to one or more cleaning actuators to perform the identified cleaning actions 7024. Cleaning actuators may be fully or partially automated. They may use any technology or technologies to perform cleaning. The actuators may be in fixed locations within the store, or they may be moveable devices that may be manually or automatically positioned during cleaning, including for example mobile robotic cleaning devices." in par. 0327 and “Zone 7031 corresponds to item storage area 7003b; this specific item storage area may for example be selected for cleaning because both shoppers 7011a and 7011b have touched items in this area. For zone 7032, two illustrative cleaning actuators are used to clean the zone: a ventilation system 7034 that forces air into, through, or out of the zone, and a chemical fogger 7035 that emits a disinfecting gas, solution, or vapor into the zone. For zone 7031, an ultraviolet light 7033 is used to irradiate the zone, as described for example with respect to FIG. 69.” In par. 0328)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method taught by Ha as modified by Kawamura to incorporate the teachings of Buibas wherein mobile robotic cleaning devices with UV light actuators are used to clean certain areas of a store. The motivation to incorporate the teachings of Buibas would be to efficiently clean only the spots that need it to reduce energy usage (see par. 0322).
Claim(s) 7, 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ha et al (US 20220371198, hereinafter Ha) in view of Kawamura et al (US 20190272580, hereinafter Kawamura) and Okada et al (US 20220297308, hereinafter Okada)
Regarding Claim 7, Ha as modified by Kawamura teaches:
the mobile autonomous device of claim 1,
Ha and Kawamura do not appear to explicitly teach all of the following, but Okada does teach:
wherein the notification indicative of the one or more discrepancies is displayed in a graphical user interface of the autonomous vehicle unit. (see at least " Subsequently, in S10, in a case where the visitor DB 443 of the storage unit 44 stores information with which the required time for providing a product or a service can be specified, the control server 4 may transmit the information to the guide robot 3 to notify the visitor of the remaining required time via the display unit 331 or the speaker. This enables the visitor to know the remaining required time, so the anxiety of the visitor can be mitigated." in par. 0108)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method taught by Ha as modified by Kawamura to incorporate the teachings of Okada wherein the mobile robot displays how long the visitor can expect before they are served. The motivation to incorporate the teachings of Okada would be to mitigate the anxiety of the visitor (see par. 0108)
Regarding Claim 14, Ha as modified by Kawamura teaches:
the computer implemented method of claim 9,
Ha and Kawamura do not appear to explicitly teach all of the following, but Okada does teach:
wherein collecting customer satisfaction data comprising detected attributes of customers at the occupied seating areas comprises detecting conditions including one or more of smiling, laughing, eating, talking, looking impatient, speaking in a raised voice, using words associated with dissatisfaction, and anger of the customers. (see at least " The state estimation unit 456 estimates a state of the visitor that has been identified by the visitor identifying unit 454, including a visitor's emotion. For example, the visitor's emotion such as smiling, being angry, or the like is estimated, based on the face image of the visitor that has been acquired from at least one of the imaging apparatus 11 and the guide robot 3. Further, the state estimation unit 456 detects that the visitor is taking a gesture such as looking around or acting restless, based on the moving image of the visitor, and estimates that the visitor is in a state of needing to be served, such as anxiety or anger. Further, the state estimation unit 456 detects that a remaining amount of a drink held by the visitor is small, and estimates that the visitor is thirsty, that is, the drink is additionally needed." in par. 0062 and “Therefore, the necessity of serving takes a negative value. On the other hand, in a case of crying with tears, looking around, shaking legs, having a remaining amount of the drink equal to or smaller than the threshold, or having an angry face, it is better for the guide robot 3 to serve the visitor. Therefore, the necessity of serving takes a positive value. Among these items, when the sum of the items corresponding to the visitor is 10 or more, it may be determined that the visitor needs to be served. Note that the user's states and scores illustrated in FIG. 7 and the threshold used for determining that the visitor needs to be served can be appropriately set in accordance with the type of facility and the service to be provided.” In par. 0094)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method taught by Ha as modified by Kawamura to incorporate the teachings of Okada wherein the mobile robot recognizes a customer in need of service based on their emotion recognized in images. The motivation to incorporate the teachings of Okada would be to prevent the emotion’s of customers from getting worse due to long waiting (see par. 0020, 0065)
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ha et al (US 20220371198, hereinafter Ha) in view of Kawamura et al (US 20190272580, hereinafter Kawamura) and Johnson et al (US 11034027, hereinafter Johnson)
Regarding Claim 10, Ha as modified by Kawamura teaches:
the computer implemented method of claim 9, further comprising
Ha and Kawamura do not appear to explicitly teach all of the following, but Johnson does teach:
generating the map representation of the restaurant space comprises using one or more SLAM techniques. (see at least " Using one or more robots 18, a map of the warehouse 10 must be created and the location of various fiducial markers dispersed throughout the warehouse must be determined. To do this, one or more of the robots 18 as they are navigating the warehouse they are building/updating a map 10a, FIG. 4, utilizing its laser-radar 22 and simultaneous localization and mapping (SLAM), which is a computational problem of constructing or updating a map of an unknown environment. Popular SLAM approximate solution methods include the particle filter and extended Kalman filter. The SLAM GMapping approach is the preferred approach, but any suitable SLAM approach can be used." in col. 6 lines 35-46 )
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method taught by Ha as modified by Kawamura to incorporate the teachings of Johnson where SLAM is used for the robots to navigate in an unknown indoor space. The motivation to incorporate the teachings of Johnson would be to help the robots more efficiently and effectively navigate the environment (see col. 8 lines 32-37)
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ha et al (US 20220371198, hereinafter Ha) in view of Kawamura et al (US 20190272580, hereinafter Kawamura) and Kim et al (KR 20180054505, hereinafter Kim)
Regarding Claim 11, Ha as modified by Kawamura teaches:
the computer implemented method of claim 9,
wherein the ground truth data is collected, at least in part, while the robot is exploring the restaurant space (see at least " More specifically, the situation information management unit 210 according to one embodiment of the invention may estimate a degree to which the attendees are gathered in the vicinity of the robot and/or at least a partial space of the patrolling place, using at least one of an imaging module (e.g., a visible light camera or an infrared camera), a scanner module (e.g., a LIDAR sensor), and a sound acquisition module (e.g., a microphone) that may be included or installed in the patrolling robot 300 and/or the patrolling place, and may acquire the number of attendees in each space of the patrolling place on the basis of the degree." in par. 0062 ) , and
Ha and Kawamura do not appear to explicitly teach all of the following, but Kim does teach:
identification of the plurality of seating areas is determined via neural network algorithms. (see at least "The burial state recognition unit 103 learns the image output from the image capturing unit 101 using the neural network model of the deep learning technology, in particular, the Convolutional Neural Network (CNN) model, Lt; / RTI > Thereafter, the burial state recognition unit 105 can recognize the burial state using the CNN model as the input image from the image capturing unit 101 based on the learning information about the burial state. For example, a human face is recognized through an image in a video, and the number of persons is counted to distinguish whether the table is an empty table or an occupied table, and it is possible to recognize whether or not there is a person in the chair." On page 3)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method taught by Ha as modified by Kawamura to incorporate the teachings of Kim wherein neural networks are used to process images of a restaurant scene and identify tables as well as which chairs are occupied. The motivation to incorporate the teachings of Kim would be to improve customer convenience (see page 8).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DYLAN M KATZ whose telephone number is (571)272-2776. The examiner can normally be reached Mon-Thurs. 8:00-6:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abby Lin can be reached on (571) 270-3976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DYLAN M KATZ/Primary Examiner, Art Unit 3657
Prosecution Insights