DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-28 are pending.
Priority
This application claims the priority of International Application No. PCT/JP2022/047928 filed Dec. 26, 2022.
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 04/30/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Specification
Applicant is reminded of the proper content of an abstract of the disclosure.
A patent abstract is a concise statement of the technical disclosure of the patent and should include that which is new in the art to which the invention pertains. The abstract should not refer to purported merits or speculative applications of the invention and should not compare the invention with the prior art.
If the patent is of a basic nature, the entire technical disclosure may be new in the art, and the abstract should be directed to the entire disclosure. If the patent is in the nature of an improvement in an old apparatus, process, product, or composition, the abstract should include the technical disclosure of the improvement. The abstract should also mention by way of example any preferred modifications or alternatives.
Where applicable, the abstract should include the following: (1) if a machine or apparatus, its organization and operation; (2) if an article, its method of making; (3) if a chemical compound, its identity and use; (4) if a mixture, its ingredients; (5) if a process, the steps.
Extensive mechanical and design details of an apparatus should not be included in the abstract. The abstract should be in narrative form and generally limited to a single paragraph within the range of 50 to 150 words in length.
See MPEP § 608.01(b) for guidelines for the preparation of patent abstracts.
The abstract of the disclosure is objected to because the abstract’s length is less than 50 words. A corrected abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text. See MPEP § 608.01(b).
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-5, 7-9, 12-14, 22, and 25-28 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Ye et al. (US 20220355474 A1, hereinafter “Ye”).
Regarding claim 1, Yes discloses a controller (Ye, see at least Fig. 1A, system 1500) comprising:
an image acquisition unit (Ye, see at least Figs. 1A, 2D, computing system 1100/1100C) configured to acquire a depth map captured by a visual sensor (Ye, see at least Figs. 1A-B, par. [0021-0022], “The system 1500 may be configured to generate, receive, and/or process the image information … the camera 1200 (which may also be referred to as an image sensing device) may be a 2D camera and/or a 3D camera … The camera 1200B may be a 3D camera (also referred to as a spatial structure sensing camera or spatial structure sensing device) that is configured to generate 3D image information which includes or forms spatial structure information regarding an environment in the camera's field of view. The spatial structure information may include depth information (e.g., a depth map) which describes respective depth values of various locations relative to the camera 1200B, such as locations on surfaces of various objects in the camera 1200's field of view …“); and
a determination unit (Ye, see at least Figs. 1A, 2D, computing system 1100/1100C) configured to determine whether an object is placed within a range specified by predetermined input information, based on the depth map (Ye, see at least Figs. 2F, par. [0040-0041], the computing system is configured to identify an object in a camera field of view based on image information 2700, e.g., a depth map or a point cloud that indicates respective depth values of various locations on one or more surfaces (e.g., top surface or other outer surface) of the imaged objects/repositories).
Regarding claim 2, Ye teaches all the limitations of claim 1. Ye further teaches comprising a search range setting unit configured to set the range specified by the predetermined input information as a search range (Ye, see at least par. [0084], “… if the computing system determines that a target object is at a particular region/cell of the repository, it may attempt to crop the image information so as to extract a specific portion corresponding to that cell … In some instances, such a region may be determined based on known information regarding a setup of a particular repository … If the computing system determines that the AGV has an error in terms of the precision of its movement (e.g., 2 cm), it may broaden the image region in which it searches for image features, so as to account for this possible error. For example, it may broaden the search region by 2 cm in one or more directions”).
Regarding claim 3, Ye teaches all the limitations of claims 1 and 2. Ye further teaches wherein the predetermined input information is information specifying the search range as a range on an image (Ye, see at least par. [0084], “… if the computing system determines that a target object is at a particular region/cell of the repository, it may attempt to crop the image information so as to extract a specific portion corresponding to that cell … In some instances, such a region may be determined based on known information regarding a setup of a particular repository … If the computing system determines that the AGV has an error in terms of the precision of its movement (e.g., 2 cm), it may broaden the image region in which it searches for image features, so as to account for this possible error. For example, it may broaden the search region by 2 cm in one or more directions”).
Regarding claim 4, Ye teaches all the limitations of claims 1 and 2. Ye further teaches wherein the predetermined input information is information specifying the search range as a range in a real space (Ye, see at least Fig. 2F, par. [0041, 0084], search range is specified based on physical edges of objects appear in the image information of a real space or known information regarding a setup of a particular object).
Regarding claim 5, Ye teaches all the limitations of claim 1. Ye further teaches wherein the predetermined input information is described in a robot program (Ye, see at least Figs. 1C, 3D, par. [0023-0025], “The robot operation system 1500B may include the computing system 1100, the camera 1200, and a robot 1300”; par. [0024], “the computing system 1100 of FIGS. 1A-1C may form or be integrated into the robot 1300, which may also be referred to as a robot controller”).
Regarding claim 7, Ye teaches all the limitations of claims 1 and 2. Ye further teaches wherein the search range setting unit sets the search range as a range defined in a real space (Ye, see at least Fig. 2F, par. [0041, 0084], search range is specified based on physical edges of objects appear in the image information of a real space or known information regarding a setup of a particular object).
Regarding claim 8, Ye teaches all the limitations of claims 1 and 2. Ye further teaches wherein the search range setting unit sets the search range as a range associated with an image capture range of the visual sensor (Ye, see at least par. [0021], search range is the camera’s field of view; par. [0084], the computing system is configured to position the camera farther from the shelf and from the robot, so as to ensure that a relevant cell of the shelf falls within the camera's field of view).
Regarding claim 9, Ye teaches all the limitations of claims 1-3. Ye further teaches wherein the range on the image is represented as numerical value information based on a coordinate on the image (Ye, see at least Fig. 2E, par. [0039-0040], “…In the example of FIG. 2E, the 2D image information 2600 may include respective portions 2000A/2000B/2000C/2000D/2550, also referred to as image portions, that represent respective surfaces of objects imaged by the camera 3200 … In FIG. 2E, each image portion 2000A/2000B/2000C/2000D/2550 of the 2D image information 2600 may be an image region, or more specifically a pixel region (if the image is formed by pixels). Each pixel in the pixel region of the 2D image information 2600 may be characterized as having a position that is described by a set of coordinates [U, V] and may have values that are relative to a camera coordinate system …the computing system 1100 may be configured to extract an image portion 2000A from the 2D image information 2600 to obtain only the image information associated with a corresponding object …”).
Regarding claim 12, Ye teaches all the limitations of claim 1. Ye further teaches further comprising an image capture range setting unit configured to set the range specified by the predetermined input information as an image capture range within which the visual sensor captures an image, wherein the image capture range setting unit transmits, to the visual sensor, a signal instructing the visual sensor to perform image capture in the set image capture range (Ye, see at least par. [0084], “the computing system is configured to move the camera to be positioned farther from the shelf and from the AGV, so as to ensure that a relevant cell of the shelf falls within the camera's field of view”; par. [0095], the computing system is configured to zoom into and extract a portion of the image information to focus on a particular portion of the image information corresponding to the particular object).
Regarding claim 13, Ye teaches all the limitations of claim 1. Ye further teaches wherein the determination unit determines whether an object is placed on an inspection target surface within the specified range (Ye, see at least Figs. 2E-F, objects such as boxes stacked on a pallet within the camera’s field of view).
Regarding claim 14, Ye teaches all the limitations of claims 1 and 13. Ye further teaches wherein the determination unit determines whether an object is placed on the inspection target surface, based on a frequency distribution related to distance information of each point in the acquired depth map (Ye, see at least par. [0106], “…the computing system may use a statistical element, such as a histogram of pixel intensity or a histogram of depth values, to determine a location of an edge or feature in the image information. As discussed above, the image information may be captured in a 2D image, depth map, or point cloud indicating the coordinates and intensities of pixels representing a target repository or object”).
Regarding claim 22, Ye teaches all the limitations of claims 1, 13, and 14. Ye further teaches further comprising a workpiece transfer execution unit configured to execute workpiece transfer work of picking up a workpiece from a feeding device and placing the workpiece on a predetermined placement surface by a robot (Ye, see at least Fig. 3A, par. [0053], “the system 3000 may include a robot 3300 configured for placing an object into or onto a destination repository, wherein the object may be retrieved by the robot 3300 from a source repository, such as the conveyor”), wherein the determination unit determines whether an object is present on the placement surface, based on a depth map in which the placement surface is captured before the workpiece transfer execution unit places the workpiece on the placement surface (Ye, see at least Fig. 4, par. [0039, 0113], the computing system is configured to use image information captured by the camera of the source repository/conveyor, i.e. depth map that indicates respective depth values of various locations on one or more surfaces (e.g., top surface or other outer surface) of the imaged objects/repositories, subsequent to execution of the source repository approach command, that include information describing the source repository and to determine source repository content information for describing whether one or more objects are present in the source).
Regarding claim 25, Ye teaches all the limitations of claims 1, 13, and 14. Ye further teaches further wherein the depth map is an image representing the distance information of each point by brightness (Ye, see at least Fig. 2E, par. [0039, 0106], the image information is a depth map indicating the coordinates, grayscale (or intensities) of pixels representing an object from a viewpoint of the camera), and the frequency distribution represents a distribution of brightness of each point in the depth map (Ye, see at least par/ [0106], “the computing system may use a statistical element, such as a histogram of pixel intensity or a histogram of depth values, to determine a location of an edge or feature in the image information”).
Regarding claim 26, Ye teaches all the limitations of claim 1. Ye further teaches a robot system (Ye, see at least Fig. 1C, system 1500B) comprising:
a robot (Ye, see at least Figs. 1C, 3A, robot 1300);
a visual sensor (Ye, see at least Figs. 1C, 3A, camera 1200); and
the controller according to claim 1, wherein the controller controls the robot (Ye, see at least Figs. 1C, 3A, par. [0023-0025], “The robot operation system 1500B may include the computing system 1100, the camera 1200, and a robot 1300”; par. [0024], “the computing system 1100 of FIGS. 1A-1C may form or be integrated into the robot 1300, which may also be referred to as a robot controller”).
Regarding claim 27, Ye discloses an object presence determination method executed on an information processing device (Ye, see at least Figs. 2B, 4, par. [0060], method 4000 related to object recognition method execute, the at least one processing circuit 1100 may perform the method 4000 by executing instructions stored on a non-transitory computer-readable medium (e.g., 1120)), the method comprising:
acquiring a depth map captured by a visual sensor (Ye, see at least Figs. 1C, 2E-F, 4, par. [0022, 0077], step 4004, acquiring image information captured by the camera 1200, i.e. depth map which describes respective depth values of various locations relative to the camera 1200B, such as locations on surfaces of various objects in the camera 1200's field of view); and
determining whether an object is placed within a range specified by predetermined input information, based on the depth map (Ye, see at least Figs. 1C, 2E-F, 4, par. [0022, 0113], step 4009, detecting whether one or more objects are present in the source repository within the camera 1200's field of view, based on the depth map describing the source repository).
Regarding claim 28, Ye discloses a non-transitory computer readable storage medium storing instructions that, when executed by a processor of a computer, cause the processor to perform (Ye, see at least Figs. 2B, 4, par. [0060], method 4000 related to object recognition method execute, the at least one processing circuit 1100 may perform the method 4000 by executing instructions stored on a non-transitory computer-readable medium (e.g., 1120)):
acquiring a depth map captured by a visual sensor (Ye, see at least Figs. 1C, 2E-F, 4, par. [0022, 0077], step 4004, acquiring image information captured by the camera 1200, i.e. depth map which describes respective depth values of various locations relative to the camera 1200B, such as locations on surfaces of various objects in the camera 1200's field of view); and
determining whether an object is placed within a range specified by predetermined input information, based on the depth map (Ye, see at least Figs. 1C, 2E-F, 4, par. [0022, 0113], step 4009, detecting whether one or more objects are present in the source repository within the camera 1200's field of view, based on the depth map describing the source repository).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 6, 10, and 24 are rejected under 35 U.S.C. 103 as being unpatentable over Ye et al. (US 20220355474 A1, hereinafter “Ye”) as applied to claims 1-3, 13-14 and 22 above, and further in view of Siguta (US 20200033109 A1).
Regarding claim 6, Ye teaches all the limitations of claim 1. Ye further teaches a user interface, i.e. any suitable display medium such as a liquid crystal display, is configured to present information handled by the computing system 1500 (Ye, see at least par. [0018]). However, Ye fails to specifically teach the predetermined input information is information input through a user interface.
Siguta, in the same field of endeavor, teaches a controller comprising a user interface, i.e. UI display control unit 11a displays a user interface screen, is configured to allow a user to input instruction to specify a range for detecting an object in the range, for example, the user draws a circle or a line on the user interface to specify a portion serving as a measurement object in the image of the workpiece and the controller is configured to detect the structure of the specified portion (Siguta, see at least Figs. 2-10, 19, par. [0043, 0105-0106]).
It would have been obvious to one or ordinary skill in the art at the time of filing of the invention to modify the controller as taught by Ye with the technique of providing a user interface to allow a user to input instruction to specify a range for detecting an object in the range as taught by Siguta in order to reduce the work load required to measure a workpiece (Siguta, par. [0007]).
Regarding claim 10, Ye teaches all the limitations of claims 1-3. Ye further teaches a user interface, i.e. any suitable display medium such as a liquid crystal display, is configured to present information handled by the computing system 1500 (Ye, see at least par. [0018]). However, Ye fails to specifically teach wherein the search range setting unit provides a graphical user interface for specifying the range on the image by a graphical operation.
Siguta, in the same field of endeavor, teaches a controller is configured to provide a graphical user interface, i.e. UI display control unit 11a displays a user interface screen, for specifying the range on the image by a graphical operation for detecting an object in the range, for example, the user draws a circle or a line on an image of workpiece to specify a portion serving as a measurement object in the image of the workpiece and the controller is configured to detect the structure of the specified portion (Siguta, see at least Figs. 2-10, 19, par. [0043, 0105-0106]).
It would have been obvious to one or ordinary skill in the art at the time of filing of the invention to modify the controller as taught by Ye with the technique of providing a graphical user interface for specifying the range on the image by a graphical operation as taught by Siguta in order to reduce the work load required to measure a workpiece (Siguta, par. [0007]).
Regarding claim 24, Ye teaches all the limitations of claims 1, 13-14 and 22. Ye further teaches the controller further comprising the non-transitory computer-readable medium 1120 is configured to store image information generated by the camera 1200 and received by the computing system 1100 (Ye, see at least Fig. 2D, par. [0031]). However, Ye fails to specifically teach a history image storage unit configured to store a history image when an object is determined to be present by the determination unit.
Siguta, in the same field of endeavor, teaches the controller is further configured to store past measurement history information in a measurement history database 16b in the storage unit 16 on the basis of the measurement object structure detected by the measurement object acquisition unit 11c (Siguta, see at least Fig. 1, par. [0053]).
It would have been obvious to one or ordinary skill in the art at the time of filing of the invention to modify the controller as taught by Ye with the technique of providing a history image storage unit configured to store a history image when an object is determined to be present by the determination unit as taught by Siguta in order to provide a ranking for a list of candidate measurement items to the user via the user interface (Siguta, par. [0053-0054]).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Ye et al. (US 20220355474 A1, hereinafter “Ye”) as applied to claims 1, 2 and 4 above, and further in view of Kamoi (US 10421193 B2).
Regarding claim 11, Ye teaches all the limitations of claims 1, 2 and 4. Ye further teaches the search range setting unit acquires, as the information specifying the search range as a range in a real space, three-dimensional coordinates of a plurality of points in a real space (Ye, see at least Fig. 2F, par. [0041-0042], “… an image segmentation operation for extracting image information may involve detecting image locations at which physical edges of objects appear (e.g., edges of a box) in the 3D image information 2700 and using such image locations to identify an image portion (e.g., 2730) that is limited to representing an individual object in a camera field of view (e.g., 3000A) … In the example of FIG. 2F, the point cloud may include respective sets of coordinates that describe the location of the respective surfaces of the imaged objects/repositories. The coordinates may be 3D coordinates, such as [X Y Z] coordinates, and may have values that are relative to a camera coordinate system, or some other coordinate system). However, Ye fails to specifically teach to acquire three-dimensional coordinates of a plurality of points in a real space by causing a robot to execute an operation of touching up the plurality of points.
Kamoi, in the same field of endeavor, teaches to set a setting operator sets numerical values, i.e. an X coordinate, a Y coordinate and a Z coordinate in the case of a three dimensional work space, of operable-inoperable area in a search area based on causing the robot to execute an operation of touching up a plurality of points P1 to P2 along a scheduled search route 6 (Kamoi, see at least Figs. 1, 2, cols. 1, 4, 5).
It would have been obvious to one or ordinary skill in the art at the time of filing of the invention to modify the controller as taught by Ye with the technique of acquiring three-dimensional coordinates of a plurality of points in a real space by causing a robot to execute an operation of touching up the plurality of points as taught by Kamoi in order to present the operable-inoperable area in the search area to a user via a display device so that whether to make the operable-inoperable area effective becomes selectable (Kamoi, cols. 2, 9).
Claims 20-21 are rejected under 35 U.S.C. 103 as being unpatentable over Ye et al. (US 20220355474 A1, hereinafter “Ye”) as applied to claims 1, 13 and 14 above, and further in view of Higo et al. (US 20200279388 A1, hereinafter “Higo”).
Regarding claim 20, Ye teaches all the limitations of claims 1, 13 and 14. Ye further teaches a user interface, i.e. any suitable display medium such as a liquid crystal display, is configured to present information handled by the computing system 1500 (Ye, see at least par. [0018]). However, Ye fails to specifically teach a histogram creation unit configured to generate an image representing the frequency distribution, based on distance information of each point in the acquired depth map, and display the image on a display screen.
Higo, in the same field of endeavor, teaches a controller comprising generation unit 103 configured to generate a histogram image U103 representing the frequency distribution based on the depth image obtained from the result of the measurement of the depth values from the measurement apparatus to a target object by the measurement apparatus (Higo, see at least Figs. 2, 7, par. [0030-0033, 0036-0041]), and display the histogram image U103 on a display screen (Higo, see at least Fig. 7).
It would have been obvious to one or ordinary skill in the art at the time of filing of the invention to modify the controller as taught by Ye with the technique of generating an image representing the frequency distribution, based on distance information of each point in the acquired depth map, and displaying the image on a display screen as taught by Kamoiin order to generate a histogram based on user input using various graphical interface (Higo, par. [0034]).
Regarding claim 21, the combination of Ye and Higo teaches all the limitations of claims 1, 13, 14, and 20 as discussed above. The combination of Ye and Higo further teaches the histogram creation unit is configured to generate an image representing the frequency distribution within a specified range in the depth map (Hiro, see at least Fig. 21, par. [0094], the generation unit 103 is configured to modify the histogram G211 by highlighting part W210 of the histogram G211 when the user selects the recessed portion in the center of the part W210 in the image I210 by a GUI).
Allowable Subject Matter
Claims 15-19 and 23 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TRANG DANG whose telephone number is (703)756-1049. The examiner can normally be reached Monday-Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Khoi Tran can be reached at (571)272-6919. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TRANG DANG/ Examiner, Art Unit 3656 /KHOI H TRAN/Supervisory Patent Examiner, Art Unit 3656
Prosecution Insights