DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 12/26/2025 has been entered.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1, 3-5, and 12-19 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 4-6, and 12-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Edwards et al. (US 20200319627 A1, hereinafter Edwards) in view of Sugahara et al. (US 20190087976 A1, hereinafter Sugahara), and further in view of Mizukami (US 20200311965 A1).
Regarding claim 1, Edwards teaches:
A work system (Fig. 1, robotics picking environment 100), comprising:
a camera configured to acquire an object image by photographing an object from a work direction (at least as in paragraph 0036, “the picking system 106 may obtain image data (e.g., a 2-D or 3-D image) for a set of objects (e.g., one or more objects) located within a selection area that may be reached by the robotics device 118… The image may be obtained using an imaging system (not shown) that includes one or more cameras”);
processing circuitry which includes a machine learning model, and is configured to acquire a work position based on a work region of the object obtained from the machine learning model (at least as in paragraph 0037, “Upon identifying each object depicted in the image, the picking system 106 may identify a set of pick point selection factors associated with each object by applying an algorithm developed based on the training model in the first dataset and/or the second dataset”; at least as in paragraph 0035, “the dataset refers to a training model, which would be familiar to a person of ordinary skill in the art, for generating an algorithm that predicts pick points for objects that may be depicted in an image associated with a pick request”); and
an automated machine configured to execute work on the object based on a work position obtained by inputting the object image to the processing circuitry (at least as in paragraph 0032, “The robot or robotics device 118 may pick up an object based, at least in part, on the pick point selection information”),
wherein the camera is configured to photograph a plurality of the objects (at least as in paragraph 0036, “the picking system 106 may obtain image data (e.g., a 2-D or 3-D image) for a set of objects (e.g., one or more objects)”),
wherein the processing circuitry is further configured to identify the work region of each of the plurality of objects (at least as in paragraph 0067, “In the example illustrated in FIG. 4, one or more of a robotics AI engine (not shown) or a remote AI engine (not shown) may select a respective pick point 404a, 404b, 404c, 404d, 404e for each of objects 402a, 402b, 402c, 402d”; at least as in paragraph 0037, “Upon identifying each object depicted in the image, the picking system 106 may identify a set of pick point selection factors associated with each object by applying an algorithm developed based on the training model in the first dataset and/or the second dataset”; see also [0121]),
wherein the processing circuitry is further configured to identify, based on at least one of an area or a shape of the work region of each of the plurality of objects obtained from the machine learning model, a target object for which the work region is not covered by another object when viewed from the work direction as a work subject, and to acquire the work position for the target object (at least as in paragraph 0067, “because debris 405 obscures a portion of the top surface of object 401, the robotics AI engine and/or remote AI engine may be unable to select a pick point 403 for object 401. In other words, the dataset used by the robotics AI engine and/or remote AI engine may omit information related to selecting a pick point for an object covered with debris”; at least as in paragraph 0083, “The robotics AI engine 502 may send (at 541) pick point information for an object (e.g., a box or a sphere) to the robotics device 518”)…
But Edwards does not explicitly teach:
wherein the machine learning model is obtained by operating a computer to execute:
arranging a virtual object in a virtual space;
generating, in the virtual space, a virtual object image which is an image of the virtual object viewed from an imaging direction;
generating, based on information on the virtual object in the virtual space, an image indicating a work region of the virtual object viewed from the imaging direction; and
learning the work region of the virtual object in the virtual object image by using the virtual object image and the image indicating the work region to train the machine learning model…
wherein a plurality of virtual objects is arranged in the virtual space…
wherein the image indicating the work region indicates, when viewed from the imaging direction, a designated region designated in advance in a part of the virtual object.
However, Sugahara, in the same field of endeavor of a robot control system configured to conduct gripping operations through the use of machine-learning models trained by generating three-dimensional models of target objects and generating extracted images indicating a specific region of the object, specifically teaches:
wherein the machine learning model is obtained by operating a computer to execute:
arranging a virtual object in a virtual space (at least as in paragraph 0075, “The CAD model generator 22 generates a three-dimensional model representing the three-dimensional shape of the object by combining multiple distance images taken from multiple angles in the three-dimensional imaging device 10”);
generating, in the virtual space, a virtual object image which is an image of the virtual object viewed from an imaging direction (at least as in paragraph 0081, “The extracted images generated by the gripping location specifying unit 22a can be combined with the distance images, forming teacher data (ground truth) for learning the image recognition models. Each image used as the teacher data is called the teacher image in the descriptions below. The teacher data could be a pair of an extracted image and the corresponding distance image. The teacher data could be distance images that are marked making the regions specified in the extracted images distinguishable from other regions”; at least as in paragraph 0179, “If the number of the distance images that were actually taken is limited, it is possible to generate new distance images by editing existing distance images”);
generating, based on information on the virtual object in the virtual space, an image indicating a work region of the virtual object viewed from the imaging direction (at least as in paragraph 0080, “The gripping location specifying unit 22a can automatically specify the location of specific regions within each three-dimensional model, including the gripping locations… The gripping location specifying unit 22a according to the embodiment generates images that specify the gripping location (region) for each three-dimensional model… called the extracted images”); and
learning the work region of the virtual object in the virtual object image by using the virtual object image and the image indicating the work region to train the machine learning model (at least as in paragraph 0104, “The learning unit 31 generates a model used for recognition of images by learning. In the learning process, multiple distance images from multiple angles and the extracted images for the corresponding distance images are used as the teacher data… If the extracted images are scaled to the same size as the distance image for the corresponding angle, it is possible to distinguish the regions corresponding to the edges or the handles to the other regions, in the distance images”)…
wherein the image indicating the work region indicates, when viewed from the imaging direction, a designated region designated in advance in a part of the virtual object (at least as in paragraph 0080, “The gripping location specifying unit 22a according to the embodiment generates images that specify the gripping location (region) for each three-dimensional model. The images which indicate specific regions of the object, such as the gripping location are called the extracted images in the following. As mentioned below, the specified regions could be any region as long as the region is within the object. Thus, specified regions could be regions other than the gripping locations. The type of target object and the specified areas (locations) could be determined based on the purpose and scope of image recognition. As long as image recognition can be applied to the object, any type of object could be the target object. Objects could be living creatures or organisms”).
Mizukami, disclosing a robot system for performing various work operations through target object detection and a method of generating learning data for detecting the target object from an image obtained by imaging target objects stacked in bulk, specifically teaches:
wherein a plurality of virtual objects is arranged in the virtual space (at least as in paragraph 0044, “The first bulk data generation section 423 generates first bulk data by stacking a plurality of object data by simulation”; at least as in paragraph 0092, “By generating the learning data 900 including a large number of data such as the object data 901 to 907, a scale of the learning data can be increased, and the detection accuracy of the target object 91 in the object detection apparatus 4 can be further increased”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Edward, to include Mizukami’s teaching of a system conducting work operations on bulk imaging data through stacking a plurality of objects in simulation and Sugahara's teaching of a system extracting images indicating specific regions of the target object from three-dimensional models, since Mizukami teaches wherein the system improves detection rate by increasing training data, thus increasing efficiency and accuracy of the operation on the target object and reduces the load on the computational hardware when large-scale learning data is generated and Sugahara teaches wherein the system reduces the computational load and the cost and time required for training the models as training data may be generated automatically.
Regarding claim 4, in view of the above combination of Edward, Sugahara, and Mizukami, Edward further discloses:
The work system according to claim 1, wherein the work is picking of the object (at least as in paragraph 0032, wherein the robotic picking environment conducts a picking operation based on a pick point).
Regarding claim 5, in view of the above combination of Edward, Sugahara, and Mizukami, Edward further discloses:
The work system according to claim 4, wherein the picking is performed by holding a surface of the object (at least as in paragraph 0038, wherein “A signal instructing the robot or robotics device 118 to pick up object A using an 8 inch.sup.2 pick point centered around the center point of object A's top surface may be sent from the picking system 106”).
Regarding claim 6, the above combination of Edward, Sugahara, and Mizukami discloses the work system according to claim 1, but does not explicitly teach wherein the machine learning model is an instance segmentation model.
However, Sugahara, in the same field of endeavor of a robot control system configured to conduct gripping operations through the use of machine-learning models trained by generating three-dimensional models of target objects and generating extracted images indicating a specific region of the object, specifically teaches:
wherein the machine learning model is an instance segmentation model (at least as in paragraph 0080, “The task of specifying the location of an area within the object is a form of regional segmentation”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Edward, to include Sugahara's teaching of a system extracting images indicating specific regions of the target object from three-dimensional models, since Sugahara teaches wherein the system reduces the computational load and the cost and time required for training the models as training data may be generated automatically.
Regarding claim 12, the above combination of Edward, Sugahara, and Mizukami discloses the work system according to claim 1, but does not explicitly teach wherein the machine learning model is obtained by operating the computer to execute:
arranging the virtual object in the virtual space;
generating, in the virtual space, the virtual object image which is an image of the virtual object viewed from the imaging direction;
generating, based on information on the designated region of the virtual object in the virtual space, an image indicating a work region of the virtual object viewed from the imaging direction; and
learning the work region of the virtual object in the virtual object image by using the virtual object image and the image indicating the work region.
However, Sugahara, in the same field of endeavor of a robot control system configured to conduct gripping operations through the use of machine-learning models trained by generating three-dimensional models of target objects and generating extracted images indicating a specific region of the object, specifically teaches:
wherein the machine learning model is obtained by operating a computer to execute:
arranging the virtual object in the virtual space (at least as in paragraph 0075, “The CAD model generator 22 generates a three-dimensional model representing the three-dimensional shape of the object by combining multiple distance images taken from multiple angles in the three-dimensional imaging device 10”);
generating, in the virtual space, the virtual object image which is an image of the virtual object viewed from the imaging direction (at least as in paragraph 0081, “The extracted images generated by the gripping location specifying unit 22a can be combined with the distance images, forming teacher data (ground truth) for learning the image recognition models. Each image used as the teacher data is called the teacher image in the descriptions below. The teacher data could be a pair of an extracted image and the corresponding distance image. The teacher data could be distance images that are marked making the regions specified in the extracted images distinguishable from other regions”; at least as in paragraph 0179, “If the number of the distance images that were actually taken is limited, it is possible to generate new distance images by editing existing distance images”);
generating, based on information on the designated region of the virtual object in the virtual space, an image indicating a work region of the virtual object viewed from the imaging direction (at least as in paragraph 0080, “The gripping location specifying unit 22a can automatically specify the location of specific regions within each three-dimensional model, including the gripping locations… The gripping location specifying unit 22a according to the embodiment generates images that specify the gripping location (region) for each three-dimensional model… called the extracted images”); and
learning the work region of the virtual object in the virtual object image by using the virtual object image and the image indicating the work region (at least as in paragraph 0104, “The learning unit 31 generates a model used for recognition of images by learning. In the learning process, multiple distance images from multiple angles and the extracted images for the corresponding distance images are used as the teacher data… If the extracted images are scaled to the same size as the distance image for the corresponding angle, it is possible to distinguish the regions corresponding to the edges or the handles to the other regions, in the distance images”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Edward, to include Sugahara's teaching of a system extracting images indicating specific regions of the target object from three-dimensional models, since Sugahara teaches wherein the system reduces the computational load and the cost and time required for training the models as training data may be generated automatically.
Regarding claim 13, the above combination of Edward, Sugahara, and Mizukami discloses the work system according to claim 1, but does not explicitly disclose wherein the image indicating the work region indicates, when viewed from the imaging direction, a designated region designated in advance in a part of the virtual object.
However, Sugahara, in the same field of endeavor of a robot control system configured to conduct gripping operations through the use of machine-learning models trained by generating three-dimensional models of target objects and generating extracted images indicating a specific region of the object, specifically teaches:
wherein the image indicating the work region indicates, when viewed from the imaging direction, a designated region designated in advance in a part of the virtual object (at least as in paragraph 0080, “The gripping location specifying unit 22a according to the embodiment generates images that specify the gripping location (region) for each three-dimensional model. The images which indicate specific regions of the object, such as the gripping location are called the extracted images in the following. As mentioned below, the specified regions could be any region as long as the region is within the object. Thus, specified regions could be regions other than the gripping locations. The type of target object and the specified areas (locations) could be determined based on the purpose and scope of image recognition. As long as image recognition can be applied to the object, any type of object could be the target object. Objects could be living creatures or organisms”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Edward, to include Sugahara's teaching of a system extracting images indicating specific regions of the target object from three-dimensional models, since Sugahara teaches wherein the system reduces the computational load and the cost and time required for training the models as training data may be generated automatically.
Regarding claim 14, the above combination of Edward, Sugahara, and Mizukami discloses the work system according to claim 13, but does not explicitly teach wherein the machine learning model is obtained by operating the computer to execute:
arranging the virtual object in the virtual space;
generating, in the virtual space, the virtual object image which is an image of the virtual object viewed from the imaging direction;
generating, based on information on the designated region of the virtual object in the virtual space, an image indicating a work region of the virtual object viewed from the imaging direction; and
learning the work region of the virtual object in the virtual object image by using the virtual object image and the image indicating the work region.
However, Sugahara, in the same field of endeavor of a robot control system configured to conduct gripping operations through the use of machine-learning models trained by generating three-dimensional models of target objects and generating extracted images indicating a specific region of the object, specifically teaches:
wherein the machine learning model is obtained by operating a computer to execute:
arranging the virtual object in the virtual space (at least as in paragraph 0075, “The CAD model generator 22 generates a three-dimensional model representing the three-dimensional shape of the object by combining multiple distance images taken from multiple angles in the three-dimensional imaging device 10”);
generating, in the virtual space, the virtual object image which is an image of the virtual object viewed from the imaging direction (at least as in paragraph 0081, “The extracted images generated by the gripping location specifying unit 22a can be combined with the distance images, forming teacher data (ground truth) for learning the image recognition models. Each image used as the teacher data is called the teacher image in the descriptions below. The teacher data could be a pair of an extracted image and the corresponding distance image. The teacher data could be distance images that are marked making the regions specified in the extracted images distinguishable from other regions”; at least as in paragraph 0179, “If the number of the distance images that were actually taken is limited, it is possible to generate new distance images by editing existing distance images”);
generating, based on information on the designated region of the virtual object in the virtual space, an image indicating a work region of the virtual object viewed from the imaging direction (at least as in paragraph 0080, “The gripping location specifying unit 22a can automatically specify the location of specific regions within each three-dimensional model, including the gripping locations… The gripping location specifying unit 22a according to the embodiment generates images that specify the gripping location (region) for each three-dimensional model… called the extracted images”); and
learning the work region of the virtual object in the virtual object image by using the virtual object image and the image indicating the work region (at least as in paragraph 0104, “The learning unit 31 generates a model used for recognition of images by learning. In the learning process, multiple distance images from multiple angles and the extracted images for the corresponding distance images are used as the teacher data… If the extracted images are scaled to the same size as the distance image for the corresponding angle, it is possible to distinguish the regions corresponding to the edges or the handles to the other regions, in the distance images”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Edward, to include Sugahara's teaching of a system extracting images indicating specific regions of the target object from three-dimensional models, since Sugahara teaches wherein the system reduces the computational load and the cost and time required for training the models as training data may be generated automatically.
Regarding claim 15, the above combination of Edward, Sugahara, and Mizukami discloses the work system according to claim 3, however, but does not explicitly disclose wherein the image indicating the work region indicates, when viewed from the imaging direction, a designated region designated in advance in a part of the virtual object.
However, Sugahara, in the same field of endeavor of a robot control system configured to conduct gripping operations through the use of machine-learning models trained by generating three-dimensional models of target objects and generating extracted images indicating a specific region of the object, specifically teaches:
wherein the image indicating the work region indicates, when viewed from the imaging direction, a designated region designated in advance in a part of the virtual object (at least as in paragraph 0080, “The gripping location specifying unit 22a according to the embodiment generates images that specify the gripping location (region) for each three-dimensional model. The images which indicate specific regions of the object, such as the gripping location are called the extracted images in the following. As mentioned below, the specified regions could be any region as long as the region is within the object. Thus, specified regions could be regions other than the gripping locations. The type of target object and the specified areas (locations) could be determined based on the purpose and scope of image recognition. As long as image recognition can be applied to the object, any type of object could be the target object. Objects could be living creatures or organisms”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Edward, to include Sugahara's teaching of a system extracting images indicating specific regions of the target object from three-dimensional models, since Sugahara teaches wherein the system reduces the computational load and the cost and time required for training the models as training data may be generated automatically.
Regarding claim 16, the above combination of Edward, Sugahara, and Mizukami discloses the work system according to claim 15, but does not explicitly teach wherein the machine learning model is obtained by operating the computer to execute:
arranging the virtual object in the virtual space;
generating, in the virtual space, the virtual object image which is an image of the virtual object viewed from the imaging direction;
generating, based on information on the designated region of the virtual object in the virtual space, an image indicating a work region of the virtual object viewed from the imaging direction; and
learning the work region of the virtual object in the virtual object image by using the virtual object image and the image indicating the work region.
However, Sugahara, in the same field of endeavor of a robot control system configured to conduct gripping operations through the use of machine-learning models trained by generating three-dimensional models of target objects and generating extracted images indicating a specific region of the object, specifically teaches:
wherein the machine learning model is obtained by operating a computer to execute:
arranging the virtual object in the virtual space (at least as in paragraph 0075, “The CAD model generator 22 generates a three-dimensional model representing the three-dimensional shape of the object by combining multiple distance images taken from multiple angles in the three-dimensional imaging device 10”);
generating, in the virtual space, the virtual object image which is an image of the virtual object viewed from the imaging direction (at least as in paragraph 0081, “The extracted images generated by the gripping location specifying unit 22a can be combined with the distance images, forming teacher data (ground truth) for learning the image recognition models. Each image used as the teacher data is called the teacher image in the descriptions below. The teacher data could be a pair of an extracted image and the corresponding distance image. The teacher data could be distance images that are marked making the regions specified in the extracted images distinguishable from other regions”; at least as in paragraph 0179, “If the number of the distance images that were actually taken is limited, it is possible to generate new distance images by editing existing distance images”);
generating, based on information on the designated region of the virtual object in the virtual space, an image indicating a work region of the virtual object viewed from the imaging direction (at least as in paragraph 0080, “The gripping location specifying unit 22a can automatically specify the location of specific regions within each three-dimensional model, including the gripping locations… The gripping location specifying unit 22a according to the embodiment generates images that specify the gripping location (region) for each three-dimensional model… called the extracted images”); and
learning the work region of the virtual object in the virtual object image by using the virtual object image and the image indicating the work region (at least as in paragraph 0104, “The learning unit 31 generates a model used for recognition of images by learning. In the learning process, multiple distance images from multiple angles and the extracted images for the corresponding distance images are used as the teacher data… If the extracted images are scaled to the same size as the distance image for the corresponding angle, it is possible to distinguish the regions corresponding to the edges or the handles to the other regions, in the distance images”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Edward, to include Sugahara's teaching of a system extracting images indicating specific regions of the target object from three-dimensional models, since Sugahara teaches wherein the system reduces the computational load and the cost and time required for training the models as training data may be generated automatically.
Regarding claim 17, the above combination of Edward, Sugahara, and Mizukami discloses the work system according to claim 1, but does not explicitly disclose “further comprising a region designator configured to operate the computer to execute:
arranging a user interface object in the virtual space together with the virtual object;
receiving from a user a change in a position of the user interface object relative to the virtual object; and
identifying the designated region by projecting the user interface object onto the virtual object.”
However, Sugahara, in the same field of endeavor of a robot control system configured to conduct gripping operations through the use of machine-learning models trained by generating three-dimensional models of target objects and generating extracted images indicating a specific region of the object, specifically teaches:
further comprising a region designator (at least as in paragraph 0073, wherein “The image processing device 20 includes a view synchronizing unit 21, a CAD model generator 22, a CAD model storage 23, a displaying unit 24 and an input unit 25. The CAD model generator 22 has a gripping location specifying unit 22a as an internal component”) configured to operate the computer to execute:
arranging a user interface object in the virtual space together with the virtual object (at least as in paragraph 0082, wherein “the marking of the regions could be done by setting attribute information to the corresponding pixels in the image… It is possible to set specific colors or tones to the corresponding points or pixels. Any method can be used for marking regions… The feature of the gripping location specifying unit 22a could be implemented using three-dimensional CAD software running on the image processing device 20”; at least as in paragraph 0083, wherein “FIG. 7 shows the task for specifying the gripping location manually. There have been cases where the region corresponding to the gripping location was specified manually by hand for multiple images, as shown in FIG. 7. In the example shown in FIG. 7, the boundaries of the regions are specified by hand-written lines”; at least as in paragraph 0101, wherein “the displaying unit 24 shows the images of the three-dimensional model, extracted images, distance images or the like… helps the generation of models and configuration changes of the models by visual images [and] show a GUI (Graphical User Interface) which accepts operations from the users”);
receiving from a user a change in a position of the user interface object relative to the virtual object (at least as in paragraph 0082, wherein “Any method can be used for marking regions”; at least as in paragraph 0083, wherein “FIG. 7 shows the task for specifying the gripping location manually… the boundaries of the regions are specified by hand-written lines”; at least as in paragraph 0087, wherein “The task of specifying the gripping location (area) for the first time could be done manually, by having the user manipulate the three-dimensional CAD software”; at least as in paragraph 0102, wherein “the input unit 25 enables manipulations of the three-dimensional CAD software, selections of images shown in the displaying unit 24, changes in the presentation of images, instructions to the three-dimensional imaging device 10”); and
identifying the designated region by projecting the user interface object onto the virtual object (at least as in paragraph 0101, wherein “the displaying unit 24 shows the images of the three-dimensional model, extracted images, distance images or the like”; at least as in paragraph 0080, wherein “The gripping location specifying unit 22a according to the embodiment generates images that specify the gripping location (region) for each three-dimensional model. The images which indicate specific regions of the object, such as the gripping location are called the extracted images in the following”; at least as in paragraph 0087, wherein “The task of specifying the gripping location (area) for the first time could be done manually, by having the user manipulate the three-dimensional CAD software”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Edward, to include Sugahara's teaching of a system extracting images indicating specific regions of the target object from three-dimensional models, since Sugahara teaches wherein the system reduces the computational load and the cost and time required for training the models as training data may be generated automatically.
Regarding claim 18, the above combination of Edward, Sugahara, and Mizukami discloses the work system according to claim 12, but does not explicitly disclose “further comprising a region designator configured to operate the computer to execute:
arranging a user interface object in the virtual space together with the virtual object;
receiving from a user a change in a position of the user interface object relative to the virtual object; and
identifying the designated region by projecting the user interface object onto the virtual object.”
However, Sugahara, in the same field of endeavor of a robot control system configured to conduct gripping operations through the use of machine-learning models trained by generating three-dimensional models of target objects and generating extracted images indicating a specific region of the object, specifically teaches:
further comprising a region designator (at least as in paragraph 0073, wherein “The image processing device 20 includes a view synchronizing unit 21, a CAD model generator 22, a CAD model storage 23, a displaying unit 24 and an input unit 25. The CAD model generator 22 has a gripping location specifying unit 22a as an internal component”) configured to operate the computer to execute:
arranging a user interface object in the virtual space together with the virtual object (at least as in paragraph 0082, wherein “the marking of the regions could be done by setting attribute information to the corresponding pixels in the image… It is possible to set specific colors or tones to the corresponding points or pixels. Any method can be used for marking regions… The feature of the gripping location specifying unit 22a could be implemented using three-dimensional CAD software running on the image processing device 20”; at least as in paragraph 0083, wherein “FIG. 7 shows the task for specifying the gripping location manually. There have been cases where the region corresponding to the gripping location was specified manually by hand for multiple images, as shown in FIG. 7. In the example shown in FIG. 7, the boundaries of the regions are specified by hand-written lines”; at least as in paragraph 0101, wherein “the displaying unit 24 shows the images of the three-dimensional model, extracted images, distance images or the like… helps the generation of models and configuration changes of the models by visual images [and] show a GUI (Graphical User Interface) which accepts operations from the users”);
receiving from a user a change in a position of the user interface object relative to the virtual object (at least as in paragraph 0082, wherein “Any method can be used for marking regions”; at least as in paragraph 0083, wherein “FIG. 7 shows the task for specifying the gripping location manually… the boundaries of the regions are specified by hand-written lines”; at least as in paragraph 0087, wherein “The task of specifying the gripping location (area) for the first time could be done manually, by having the user manipulate the three-dimensional CAD software”; at least as in paragraph 0102, wherein “the input unit 25 enables manipulations of the three-dimensional CAD software, selections of images shown in the displaying unit 24, changes in the presentation of images, instructions to the three-dimensional imaging device 10”); and
identifying the designated region by projecting the user interface object onto the virtual object (at least as in paragraph 0101, wherein “the displaying unit 24 shows the images of the three-dimensional model, extracted images, distance images or the like”; at least as in paragraph 0080, wherein “The gripping location specifying unit 22a according to the embodiment generates images that specify the gripping location (region) for each three-dimensional model. The images which indicate specific regions of the object, such as the gripping location are called the extracted images in the following”; at least as in paragraph 0087, wherein “The task of specifying the gripping location (area) for the first time could be done manually, by having the user manipulate the three-dimensional CAD software”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Edward, to include Sugahara's teaching of a system extracting images indicating specific regions of the target object from three-dimensional models, since Sugahara teaches wherein the system reduces the computational load and the cost and time required for training the models as training data may be generated automatically.
Claim(s) 7-10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Edwards et al. (US 20200319627 A1, hereinafter Edwards) in view of Sugahara et al. (US 20190087976 A1, hereinafter Sugahara) and Mizukami (US 20200311965 A1), and further in view of Weinzaepfel et al. (US 20200364509 A1, hereinafter Weinzaepfel).
Regarding claim 7, the above combination of Edward, Sugahara, and Mizukami discloses the work system according to claim 6, but does not explicitly teach wherein the instance segmentation model is Mask R-CNN.
However, Weinzaepfel, disclosing a method for training a neural network to provide visual localization by detecting and segmenting an object-of-interest in the training image and generating training data through simulations, specifically teaches “wherein the instance segmentation model is Mask R-CNN” (at least as in paragraph 0063, wherein “Branch A, which predicts segmentation and match regression, follows mask regional convolution neural network architecture”; at least as in paragraph 0134, wherein “the above described training and use of the visual localization network of FIGS. 1-8 emulates DensePose, an extension of mask regional convolution neural network”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Edward, to include Weinzaepfel’s teaching of training a visual localization neural network, since Weinzaepfel teaches wherein the method allows the generation of more viewpoints of the objects-of-interest, and thus, enabling improved detection and matching of objects-of-interest at test time with novel viewpoints.
Regarding claim 8, the above combination of Edward, Sugahara, and Mizukami discloses the work system according to claim 1, but does not explicitly teach wherein the image indicating the work region is a mask image indicating an existence region of the virtual object when viewed from the imaging direction and corresponding to the virtual object image.
However, Weinzaepfel, disclosing a method for training a neural network to provide visual localization by detecting and segmenting an object-of-interest in the training image and generating training data through simulations, specifically teaches “wherein the image indicating the work region is a mask image indicating an existence region of the virtual object when viewed from the imaging direction and corresponding to the virtual object image” (at least as in paragraph 0059, wherein “Branch A predicts binary segmentation for each object-of-interest and x and y reference image coordinates regression”; at least as in paragraph 0063, wherein “Branch A, which predicts segmentation and match regression, follows mask regional convolution neural network architecture”; at least as in paragraph 0069, wherein “a bounding box of the manual mask annotation 1301 defines a reference image 1500 (with bounding box 1501) for an object-of-interest 1350 in a first frame of training data 1300”; at least as in paragraph 0071, wherein “Upon obtaining the three-dimensional positions of key points 1351 in the manual mask annotation 1301 for the object-of-interest 1350 in the first frame of training data 1300, the bounding box of the manual mask annotation 1301 for the object-of-interest 1350 can be propagated to a second frame of training data 1100 to obtain a manual mask annotation 1101 for an object-of-interest 1150 in the second frame of training data 1100 if there are enough matches between key points 1351 of the first frame of training data 1300 and key points 1051 of the second frame of training data 1100”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Edward, to include Weinzaepfel’s teaching of training a visual localization neural network, since Weinzaepfel teaches wherein the method allows the generation of more viewpoints of the objects-of-interest, and thus, enabling improved detection and matching of objects-of-interest at test time with novel viewpoints.
Regarding claim 9, the above combination of Edward, Sugahara, and Mizukami discloses the work system according to claim 6, but does not explicitly teach wherein the image indicating the work region is a mask image indicating an existence region of the virtual object when viewed from the imaging direction and corresponding to the virtual object image.
However, Weinzaepfel, disclosing a method for training a neural network to provide visual localization by detecting and segmenting an object-of-interest in the training image and generating training data through simulations, specifically teaches “wherein the image indicating the work region is a mask image indicating an existence region of the virtual object when viewed from the imaging direction and corresponding to the virtual object image” (at least as in paragraph 0059, wherein “Branch A predicts binary segmentation for each object-of-interest and x and y reference image coordinates regression”; at least as in paragraph 0063, wherein “Branch A, which predicts segmentation and match regression, follows mask regional convolution neural network architecture”; at least as in paragraph 0069, wherein “a bounding box of the manual mask annotation 1301 defines a reference image 1500 (with bounding box 1501) for an object-of-interest 1350 in a first frame of training data 1300”; at least as in paragraph 0071, wherein “Upon obtaining the three-dimensional positions of key points 1351 in the manual mask annotation 1301 for the object-of-interest 1350 in the first frame of training data 1300, the bounding box of the manual mask annotation 1301 for the object-of-interest 1350 can be propagated to a second frame of training data 1100 to obtain a manual mask annotation 1101 for an object-of-interest 1150 in the second frame of training data 1100 if there are enough matches between key points 1351 of the first frame of training data 1300 and key points 1051 of the second frame of training data 1100”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Edward, to include Weinzaepfel’s teaching of training a visual localization neural network, since Weinzaepfel teaches wherein the method allows the generation of more viewpoints of the objects-of-interest, and thus, enabling improved detection and matching of objects-of-interest at test time with novel viewpoints.
Regarding claim 10, the above combination of Edward, Sugahara, and Mizukami discloses the work system according to claim 9, but does not explicitly teach wherein the mask image indicates the existence region of at least one virtual object involved in the plurality of virtual objects, based on information on the plurality of virtual objects.
However, Weinzaepfel, disclosing a method for training a neural network to provide visual localization by detecting and segmenting an object-of-interest in the training image and generating training data through simulations, specifically teaches “wherein the mask image indicates the existence region of at least one virtual object involved in the plurality of virtual objects, based on information on the plurality of virtual objects” (at least as in paragraph 0059, wherein “Branch A predicts binary segmentation for each object-of-interest and x and y reference image coordinates regression”; at least as in paragraph 0063, wherein “Branch A, which predicts segmentation and match regression, follows mask regional convolution neural network architecture”; at least as in paragraph 0069, wherein “a bounding box of the manual mask annotation 1301 defines a reference image 1500 (with bounding box 1501) for an object-of-interest 1350 in a first frame of training data 1300”; at least as in paragraph 0071, wherein “Upon obtaining the three-dimensional positions of key points 1351 in the manual mask annotation 1301 for the object-of-interest 1350 in the first frame of training data 1300, the bounding box of the manual mask annotation 1301 for the object-of-interest 1350 can be propagated to a second frame of training data 1100 to obtain a manual mask annotation 1101 for an object-of-interest 1150 in the second frame of training data 1100 if there are enough matches between key points 1351 of the first frame of training data 1300 and key points 1051 of the second frame of training data 1100”; at least as in paragraph 0081, “to test the visual localization process and to measure the impact of varying lighting conditions and occlusions on different localization methods, a virtual model was used”; at least as in paragraph 0082, “The virtual model consisted of a scene containing three to four rooms, in which a total of forty-two images of paintings (objects-of-interest) are placed on the virtual walls”; at least as in paragraph 0083, “To train the visual localization network, training data is acquired by simulating the scene being captured by a robot and test data is acquired by simulating photos taken by visitors”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Edward, to include Weinzaepfel’s teaching of training a visual localization neural network, since Weinzaepfel teaches wherein the method allows the generation of more viewpoints of the objects-of-interest, and thus, enabling improved detection and matching of objects-of-interest at test time with novel viewpoints.
Claim(s) 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sugahara et al. (US 20190087976 A1, hereinafter Sugahara) in view of Edwards et al. (US 20200319627 A1, hereinafter Edwards), and further in view of Mizukami (US 20200311965 A1).
Regarding claim 19, Sugahara teaches:
A machine learning device, comprising a central processing unit and a memory (at least as in paragraph 0050-0054, wherein the characteristic learning device 30 generates a model for recognizing gripping locations and are information processing systems with CPUs and storage devices) which are configured to:
arrange a virtual object in a virtual space (at least as in paragraph 0075, “The CAD model generator 22 generates a three-dimensional model representing the three-dimensional shape of the object by combining multiple distance images taken from multiple angles in the three-dimensional imaging device 10”);
generate, in the virtual space, a virtual object image which is an image of the virtual object viewed from an imaging direction (at least as in paragraph 0081, “The extracted images generated by the gripping location specifying unit 22a can be combined with the distance images, forming teacher data (ground truth) for learning the image recognition models. Each image used as the teacher data is called the teacher image in the descriptions below. The teacher data could be a pair of an extracted image and the corresponding distance image. The teacher data could be distance images that are marked making the regions specified in the extracted images distinguishable from other regions”; at least as in paragraph 0179, “If the number of the distance images that were actually taken is limited, it is possible to generate new distance images by editing existing distance images”);
generate, based on information on the virtual object in the virtual space, an image indicating a work region of the virtual object viewed from the imaging direction (at least as in paragraph 0080, “The gripping location specifying unit 22a can automatically specify the location of specific regions within each three-dimensional model, including the gripping locations… The gripping location specifying unit 22a according to the embodiment generates images that specify the gripping location (region) for each three-dimensional model… called the extracted images”); and
cause a machine learning model to learn the work region of the virtual object in the virtual object image by using the virtual object image and the image indicating the work region (at least as in paragraph 0104, “The learning unit 31 generates a model used for recognition of images by learning. In the learning process, multiple distance images from multiple angles and the extracted images for the corresponding distance images are used as the teacher data… If the extracted images are scaled to the same size as the distance image for the corresponding angle, it is possible to distinguish the regions corresponding to the edges or the handles to the other regions, in the distance images”);
acquire a work position based on the work region (at least as in paragraph 0135, “By using the learned models, the image recognition unit 44 can detect objects and specific regions within the objects”);
generate a working instruction from the work region, the working instruction being configured to direct an automated machine to execute work on an object (at least as in paragraph 0135, “based on the result of the detection of regions within the object, the robotic hand 41 is controlled, enabling the gripping and transportation of objects”); and
send the working instruction to the automated machine to cause the automated machine to execute the work on the object (at least as in paragraph 0139, “If the robot 40 can detect regions within the object, it is possible to obtain images shown in FIG. 12, and estimate the gripping locations of the coffee cup”),
wherein the image indicating the work region indicates, when viewed from the imaging direction, a designated region designated in advance in a part of the virtual object (at least as in paragraph 0080, “The gripping location specifying unit 22a according to the embodiment generates images that specify the gripping location (region) for each three-dimensional model. The images which indicate specific regions of the object, such as the gripping location are called the extracted images in the following. As mentioned below, the specified regions could be any region as long as the region is within the object. Thus, specified regions could be regions other than the gripping locations. The type of target object and the specified areas (locations) could be determined based on the purpose and scope of image recognition. As long as image recognition can be applied to the object, any type of object could be the target object. Objects could be living creatures or organisms”).
But Sugahara does not explicitly teach:
wherein a plurality of virtual objects is arranged in the virtual space,
wherein the machine learning device is further configured to identify the work region of each of the plurality of virtual objects,
wherein the machine learning device is configured to identify, based on at least one of an area or a shape of the work region of each of the plurality of virtual objects, a target virtual object for which the work region is not covered by another virtual object when viewed from a work direction as a work subject, and to acquire the work position for the target virtual object.
However, Mizukami, disclosing a robot system for performing various work operations through target object detection and a method of generating learning data for detecting the target object from an image obtained by imaging target objects stacked in bulk, specifically teaches:
wherein a plurality of virtual objects is arranged in the virtual space (at least as in paragraph 0044, “The first bulk data generation section 423 generates first bulk data by stacking a plurality of object data by simulation”; at least as in paragraph 0092, “By generating the learning data 900 including a large number of data such as the object data 901 to 907, a scale of the learning data can be increased, and the detection accuracy of the target object 91 in the object detection apparatus 4 can be further increased”).
Edwards, in the same field of endeavor of controlling robotic picking systems using AI, specifically teaches:
wherein the machine learning device is further configured to identify the work region of each of the plurality of virtual objects (at least as in paragraph 0067, “In the example illustrated in FIG. 4, one or more of a robotics AI engine (not shown) or a remote AI engine (not shown) may select a respective pick point 404a, 404b, 404c, 404d, 404e for each of objects 402a, 402b, 402c, 402d”; at least as in paragraph 0037, “Upon identifying each object depicted in the image, the picking system 106 may identify a set of pick point selection factors associated with each object by applying an algorithm developed based on the training model in the first dataset and/or the second dataset”; see also [0121]),
wherein the machine learning device is configured to identify, based on at least one of an area or a shape of the work region of each of the plurality of virtual objects, a target virtual object for which the work region is not covered by another virtual object when viewed from a work direction as a work subject, and to acquire the work position for the target virtual object (at least as in paragraph 0067, “because debris 405 obscures a portion of the top surface of object 401, the robotics AI engine and/or remote AI engine may be unable to select a pick point 403 for object 401. In other words, the dataset used by the robotics AI engine and/or remote AI engine may omit information related to selecting a pick point for an object covered with debris”; at least as in paragraph 0083, “The robotics AI engine 502 may send (at 541) pick point information for an object (e.g., a box or a sphere) to the robotics device 518”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Sugahara, to include Mizukami’s teaching of a system conducting work operations on bulk imaging data through stacking a plurality of objects in simulation and Edwards’ teaching of an AI based robotic picking system extracting pick points for a plurality of objects, since Mizukami teaches wherein the system improves detection rate by increasing training data, thus increasing efficiency and accuracy of the operation on the target object and reduces the load on the computational hardware when large-scale learning data is generated and Edwards teaches wherein the system reduces the need for human oversight and the learning time required.
Claim(s) 20-21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sugahara et al. (US 20190087976 A1, hereinafter Sugahara) in view of Edwards et al. (US 20200319627 A1, hereinafter Edwards) and Mizukami (US 20200311965 A1), and further in view of Weinzaepfel et al. (US 20200364509 A1, hereinafter Weinzaepfel).
Regarding claim 20, Sugahara discloses:
A machine learning method of causing a computer to execute:
arranging virtual objects in a virtual space (at least as in paragraph 0075, “The CAD model generator 22 generates a three-dimensional model representing the three-dimensional shape of the object by combining multiple distance images taken from multiple angles in the three-dimensional imaging device 10”);
generating, in the virtual space, a virtual object image which is an image of the virtual objects viewed from an imaging direction (at least as in paragraph 0081, “The extracted images generated by the gripping location specifying unit 22a can be combined with the distance images, forming teacher data (ground truth) for learning the image recognition models. Each image used as the teacher data is called the teacher image in the descriptions below. The teacher data could be a pair of an extracted image and the corresponding distance image. The teacher data could be distance images that are marked making the regions specified in the extracted images distinguishable from other regions”; at least as in paragraph 0179, “If the number of the distance images that were actually taken is limited, it is possible to generate new distance images by editing existing distance images”);
generating, based on information on the virtual object in the virtual space, an image indicating a work region of the virtual object viewed from the imaging direction (at least as in paragraph 0080, “The gripping location specifying unit 22a can automatically specify the location of specific regions within each three-dimensional model, including the gripping locations… The gripping location specifying unit 22a according to the embodiment generates images that specify the gripping location (region) for each three-dimensional model… called the extracted images”); and
causing a machine learning model to learn an (at least as in paragraph 0104, “The learning unit 31 generates a model used for recognition of images by learning. In the learning process, multiple distance images from multiple angles and the extracted images for the corresponding distance images are used as the teacher data… If the extracted images are scaled to the same size as the distance image for the corresponding angle, it is possible to distinguish the regions corresponding to the edges or the handles to the other regions, in the distance images”; at least as in paragraph 0080, “As mentioned below, the specified regions could be any region as long as the region is within the object. Thus, specified regions could be regions other than the gripping locations”);
generating a working instruction from the work region, the working instruction being configured to direct an automated machine to execute work on an object at least as in paragraph 0135, “based on the result of the detection of regions within the object, the robotic hand 41 is controlled, enabling the gripping and transportation of objects”);
sending the working instruction to the automated machine to cause the automated machine to execute the work on the object (at least as in paragraph 0139, “If the robot 40 can detect regions within the object, it is possible to obtain images shown in FIG. 12, and estimate the gripping locations of the coffee cup”),
acquiring a work position based on the work region (at least as in paragraph 0135, “By using the learned models, the image recognition unit 44 can detect objects and specific regions within the objects”);
generating a working instruction from the work region, the working instruction being configured to direct an automated machine to execute work on an object (at least as in paragraph 0135, “based on the result of the detection of regions within the object, the robotic hand 41 is controlled, enabling the gripping and transportation of objects”; Examiner notes wherein this limitation is repeated); and
sending the working instruction to the automated machine to cause the automated machine to execute the work on the object at least as in paragraph 0139, “If the robot 40 can detect regions within the object, it is possible to obtain images shown in FIG. 12, and estimate the gripping locations of the coffee cup”; Examiner notes wherein this limitation is repeated),
wherein the image indicating the work region indicates, when viewed from the imaging direction, a designated region designated in advance in a part of the virtual object (at least as in paragraph 0080, “The gripping location specifying unit 22a according to the embodiment generates images that specify the gripping location (region) for each three-dimensional model. The images which indicate specific regions of the object, such as the gripping location are called the extracted images in the following. As mentioned below, the specified regions could be any region as long as the region is within the object. Thus, specified regions could be regions other than the gripping locations. The type of target object and the specified areas (locations) could be determined based on the purpose and scope of image recognition. As long as image recognition can be applied to the object, any type of object could be the target object. Objects could be living creatures or organisms”).
But Sugahara does not explicitly teach:
existence region…
wherein a plurality of virtual objects is arranged in the virtual space,
wherein the machine learning device is further configured to identify the work region of each of the plurality of virtual objects,
wherein the machine learning device is configured to identify, based on at least one of an area or a shape of the work region of each of the plurality of virtual objects, a target virtual object for which the work region is not covered by another virtual object when viewed from a work direction as a work subject, and to acquire the work position for the target virtual object.
However, Weinzaepfel discloses a method for training a neural network to provide visual localization by detecting and segmenting an object-of-interest in the training image and generating training data through simulations. Weinzaepfel specifically discloses “an existence region” (at least as in paragraph 0059, wherein “Branch A predicts binary segmentation for each object-of-interest and x and y reference image coordinates regression”; at least as in paragraph 0063, wherein “Branch A, which predicts segmentation and match regression, follows mask regional convolution neural network architecture”; at least as in paragraph 0069, wherein “a bounding box of the manual mask annotation 1301 defines a reference image 1500 (with bounding box 1501) for an object-of-interest 1350 in a first frame of training data 1300”; at least as in paragraph 0071, wherein “Upon obtaining the three-dimensional positions of key points 1351 in the manual mask annotation 1301 for the object-of-interest 1350 in the first frame of training data 1300, the bounding box of the manual mask annotation 1301 for the object-of-interest 1350 can be propagated to a second frame of training data 1100 to obtain a manual mask annotation 1101 for an object-of-interest 1150 in the second frame of training data 1100 if there are enough matches between key points 1351 of the first frame of training data 1300 and key points 1051 of the second frame of training data 1100”).
Mizukami, disclosing a robot system for performing various work operations through target object detection and a method of generating learning data for detecting the target object from an image obtained by imaging target objects stacked in bulk, specifically teaches:
wherein a plurality of virtual objects is arranged in the virtual space (at least as in paragraph 0044, “The first bulk data generation section 423 generates first bulk data by stacking a plurality of object data by simulation”; at least as in paragraph 0092, “By generating the learning data 900 including a large number of data such as the object data 901 to 907, a scale of the learning data can be increased, and the detection accuracy of the target object 91 in the object detection apparatus 4 can be further increased”).
Edwards, in the same field of endeavor of controlling robotic picking systems using AI, specifically teaches:
wherein the machine learning device is further configured to identify the work region of each of the plurality of virtual objects (at least as in paragraph 0067, “In the example illustrated in FIG. 4, one or more of a robotics AI engine (not shown) or a remote AI engine (not shown) may select a respective pick point 404a, 404b, 404c, 404d, 404e for each of objects 402a, 402b, 402c, 402d”; at least as in paragraph 0037, “Upon identifying each object depicted in the image, the picking system 106 may identify a set of pick point selection factors associated with each object by applying an algorithm developed based on the training model in the first dataset and/or the second dataset”; see also [0121]),
wherein the machine learning device is configured to identify, based on at least one of an area or a shape of the work region of each of the plurality of virtual objects, a target virtual object for which the work region is not covered by another virtual object when viewed from a work direction as a work subject, and to acquire the work position for the target virtual object (at least as in paragraph 0067, “because debris 405 obscures a portion of the top surface of object 401, the robotics AI engine and/or remote AI engine may be unable to select a pick point 403 for object 401. In other words, the dataset used by the robotics AI engine and/or remote AI engine may omit information related to selecting a pick point for an object covered with debris”; at least as in paragraph 0083, “The robotics AI engine 502 may send (at 541) pick point information for an object (e.g., a box or a sphere) to the robotics device 518”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Sugahara, to include Mizukami’s teaching of a system conducting work operations on bulk imaging data through stacking a plurality of objects in simulation and Edwards’ teaching of an AI based robotic picking system extracting pick points for a plurality of objects, since Mizukami teaches wherein the system improves detection rate by increasing training data, thus increasing efficiency and accuracy of the operation on the target object and reduces the load on the computational hardware when large-scale learning data is generated and Edwards teaches wherein the system reduces the need for human oversight and the learning time required.
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Sugahara, to include Weinzaepfel’s teaching of training a visual localization neural network, Mizukami’s teaching of a system conducting work operations on bulk imaging data through stacking a plurality of objects in simulation, and Edwards’ teaching of an AI based robotic picking system extracting pick points for a plurality of objects, since Weinzaepfel teaches wherein the method allows the generation of more viewpoints of the objects-of-interest thus enabling improved detection and matching of objects-of-interest at test time with novel viewpoints, Mizukami teaches wherein the system improves detection rate by increasing training data, thus increasing efficiency and accuracy of the operation on the target object and reduces the load on the computational hardware when large-scale learning data is generated, and Edwards teaches wherein the system reduces the need for human oversight and the learning time required.
Regarding Claim 21, the above combination of Sugahara, Weinzaepfel, Edwards, and Mizukami teaches the work system according to claim 7, but does not explicitly teach wherein the processing circuitry further determines the work region is not covered by another object by using a label output from the Mask R-CNN.
However, Weinzaepfel discloses a method for training a neural network to provide visual localization by detecting and segmenting an object-of-interest in the training image and generating training data through simulations. Weinzaepfel specifically discloses “by using a label output from the Mask R-CNN” (at least as in paragraph 0044, wherein “Given a query image, a convolutional neural network outputs a list of detections. Each detection consists of: (a) a bounding box with a class label o, i.e., the identifier of the detected object-of-interest and a confidence score; (b) a segmentation mask; and (c) a set of two-dimensional to two-dimensional matches {q.fwdarw.q.sup.I} between pixels q in the query image and pixels q.sup.I in the reference image I.sub.o of the object-of-interest o”; at least as in paragraph 0063, “at least as in paragraph 0063, wherein “Branch A, which predicts segmentation and match regression, follows mask regional convolution neural network architecture”; at least as in paragraph 0134, wherein “the above described training and use of the visual localization network of FIGS. 1-8 emulates DensePose, an extension of mask regional convolution neural network”)
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Sugahara, to include Weinzaepfel’s teaching of training a visual localization neural network, since Weinzaepfel teaches wherein the method allows the generation of more viewpoints of the objects-of-interest thus enabling improved detection and matching of objects-of-interest at test time with novel viewpoints.
Allowable Subject Matter
Claim 3 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RICARDO ICHIKAWA VISCARRA whose telephone number is (571)270-0154. The examiner can normally be reached M-F 9-12 & 2-4 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Adam Mott can be reached on (571) 270-5376. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RICARDO I VISCARRA/Examiner, Art Unit 3657
/ADAM R MOTT/ Supervisory Patent Examiner, Art Unit 3657
Prosecution Insights