Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This action is in response to the applicant’s filing on April 17,2026. Claims 1, 2, 4-6, 8, 9, 18-30 are pending.
Response to Amendment and Arguments
In respond to applicant's arguments based on the filed amendment with respect to 35 U.S.C. 102 rejections of said previous office action have been fully considered; however, upon further consideration, a new ground(s) of rejection is made.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1,5-6, 8, 9, 18-28, 30 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kojima US2021/0312661 (“Kojima”) in view of Sussman et al. US2019/0092184 (“Sussman”).
Regarding claim(s) 1, 19, 28. Kojima discloses a working machine to perform one or more work tasks in a work area, the working machine comprising:
a machine localization system to localize the working machine based on perception sensor observations captured by one or more perception sensors of the working machine, the perception sensor observations being indicative of data embedded on one or more markers placed in the work area or proximate to the work area, wherein the working machine ([0022] FIG. 11 is a flowchart showing a subroutine of step S3 (absolute position calculation process) of FIG. 8. [0023] FIG. 12 is a diagram showing coordinates of vertices of the marker 4 in a marker coordinate system. [0024] FIG. 13 is a diagram showing coordinates of vertices of the marker 4 in the image 40 captured by the image capturing apparatus 11 of FIG. 1.);
obtains localization data responsive to reading one or more machine-readable optical images on the one or more markers ([0064] FIG. 5 is a diagram showing an example of the marker 4 of FIG. 4. In the example of FIG. 5, the marker 4 is configured as a square flat plate. On one side of the marker 4, the marker 4 has a visually distinguishable pattern, into which an identifier of the marker 4 itself is encoded. In the example of FIG. 5, the marker 4 has a pattern constituted of 7×7 white or black square cells in the longitudinal and lateral directions. T), respectively;-
determines, using the obtained localization data, an absolute position of the working machine or one or more absolute positions of the one or more markers ([0071] The image recognizer 33 extracts one of the plurality of markers 4 disposed at predetermined positions and visually distinguishable from each other, from an image captured by the image capturing apparatus 11. The absolute position calculator 34 calculates the absolute position and the absolute attitude of the vehicle 1 indicating the position and the attitude of the vehicle 1 in the map (i.e., world coordinate system), by referring to the information on the markers 4 and the map information, both stored in the storage apparatus 35, based on the position and the attitude of the one extracted marker 4. In addition, the absolute position calculator 34 provides the absolute position and the absolute attitude with a timestamp of the image associated with calculation of the absolute position and the absolute attitude.), respectively; and
performs the one or more work tasks based on a least one of the determined absolute position of the working machine or the determined one or more absolute positions of the one or more markers ([0193] In the first to third embodiments, the positioning apparatus may be provided on a four-wheel vehicle, such as a forklift or a truck, or may be provided on vehicles with one to three, five or more wheel. In addition, in the first to third embodiments, the positioning apparatus may be provided on a moving body without wheels, such as an airplane, a helicopter, a drone, and a hovercraft, regardless of the number of wheels and/or the presence/absence of wheels. The positioning apparatus according to the present embodiments can estimate a position of a moving body not based on a number of rotation of wheels, but based on an image captured by an image capturing apparatus.)
Kojima does not explicitly disclose by reading one or more machine-readable optical images on the one or more markers, localization data and information regarding one or more work tasks and control operation of the vehicle based on localization data and information regarding one or more work tasks.
Sussman teaches another robotic system and method that read one or more machine-readable optical images on the one or more markers, localization data and information regarding one or more work tasks and control operation of the vehicle based on localization data and information regarding one or more work tasks ([0042, 0044, 0056] control tasks associated with such a warehouse system, such as, consolidation, counting, verification, inspection and clean-up of products… a processor (not shown) that receives data from the laser-radar and cameras 24a and 24b to capture information representative of the robot's environment. There is a memory (not shown) that operates with the processor to carry out various tasks associated with navigation within the warehouse 10, as well as to navigate to fiducial marker 30 placed on shelves 12, as shown in FIG. 3. Fiducial marker 30 (e.g. a two-dimensional bar code) corresponds to bin/location of an item ordered. The navigation approach of this invention is described in detail below with respect to FIGS. 4-8. ).
It would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify the system and method of Kojima by incorporating the applied teaching of Sussman to improve the accuracy device positioning determination and vehicle task management and one of ordinary skill before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Regarding claim(s) 5. Kojima in view of Sussman further teaches wherein the machine- readable optical images are reflective in a spectrum outside the a human-visible spectrum ([0021] FIG. 10 shows feature points extracted by an image processor 31 of FIG. 3; (a) shows feature points F1 and F2 extracted from an image 40(n) at time moment n; and (b) shows feature points F1′ and F2′ extracted from an image 40(n) at time moment n′.)
Regarding claim(s) 6. Kojima in view of Sussman further teaches wherein the spectrum comprises an IR (infrared) spectrum ([0105] In step S26, the absolute position calculator 34 reads out the position and the attitude of the marker 4 in the world coordinate system (i.e., the absolute position and the absolute attitude of the marker 4) from the storage apparatus 35, based on the identifier of the marker 4 detected in step S21.)
Regarding claim(s) 8. Kojima in view of Sussman further teaches wherein the one or more machine-readable optical images comprise an infrared IR) quick response (QR) ((fig. 5). [0105] In step S26, the absolute position calculator 34 reads out the position and the attitude of the marker 4 in the world coordinate system (i.e., the absolute position and the absolute attitude of the marker 4) from the storage apparatus 35, based on the identifier of the marker 4 detected in step S21.)
Regarding claim(s) 9. Kojima in view of Sussman further teaches wherein the one or more work tasks are part of a mission definition, wherein the embedded data embedded on the one or more markers includes detour information delineating one or more deviations from the mission definition ([0149] FIG. 30 is a diagram showing a trajectory 103 of the vehicle 1 calculated by executing a correction process according to a comparison example of the first embodiment. FIG. 31 is a diagram showing a trajectory 104 of the vehicle 1 calculated by executing the marker evaluation process of FIG. 17. Each of FIGS. 30 and 31 shows a protrusion provided on one side of each of the markers 4 to indicate a front surface of the marker 4 (Zm axis in FIG. 5) for convenience of explanation. Actually, such protrusion is not provided. Referring to FIG. 30, since the positions of the image capturing apparatus 11 in the marker coordinate systems have not been correctly determined in regions 111, 113, and 114, the positions of the image capturing apparatus 11 are inverted with respect to the normal lines of the markers 4 as show in FIG. 22, and therefore, errors occur in calculated positions of the vehicle 1.)
Regarding claim(s) 18. Kojima in view of Sussman further teaches one or more perception sensors, wherein: at least one of the one or more perception sensors is configured to operate in a first mode for discovering the one or more markers in the work area; and at least one of the one or more perception sensors is configured to operate in a second mode to read the one or more machine-readable optical images on the one or more markers ([0008] According to an aspect of the present disclosure, a positioning apparatus is provided with a first calculator, a storage apparatus, a second calculator, and a corrector. The first calculator calculates a first position and a first attitude of a moving body indicating a relative position and a relative attitude of the moving body with respect to a reference position and a reference attitude, based on a plurality of images captured by an image capturing apparatus mounted on the moving body. The storage apparatus stores information on identifiers, positions, and attitudes of a plurality of markers disposed at predetermined positions and visually distinguishable from each other, and information on a map containing a passageway for the moving body.)
Regarding claim(s) 20. Kojima in view of Sussman further teaches setting up the one or more markers for localization within an environment ([0005] For example, there is a technology called Visual Simultaneous Localization and Mapping (Visual-SLAM) as).
Regarding claim(s) 21. Kojima in view of Sussman further teaches generating and storing a landmark map including the one or more markers ([0005], a moving body provided with an image capturing apparatus moves and captures images around the moving body, and then, an amount of movement of the moving body is calculated based on amounts of movement of feature points in the captured images. Thus, it is possible to estimate a current position of the moving body, and generate a map based on a trajectory of the moving body.)
Regarding claim(s) 22. Kojima in view of Sussman further teaches wherein the steps are performed by the working machine ([0193] In the first to third embodiments, the positioning apparatus may be provided on a four-wheel vehicle, such as a forklift or a truck, or may be provided on vehicles with one to three, five or more wheel. In addition, in the first to third embodiments, the positioning apparatus may be provided on a moving body without wheels, such as an airplane, a helicopter, a drone, and a hovercraft, regardless of the number of wheels and/or the presence/absence of wheels. The positioning apparatus according to the present embodiments can estimate a position of a moving body not based on a number of rotation of wheels, but based on an image captured by an image capturing apparatus.)
Regarding claim(s) 23. Kojima in view of Sussman further teaches wherein communicating the intermediary value to the resource comprises communicating the intermediary value via a wireless connection (Sussman: [0043] Referring to FIG. 1, a typical order-fulfillment warehouse 10 includes shelves 12 filled with the various items that could be included in an order 16. In operation, the order 16 from warehouse management server 15 arrives at an order-server 14.).
Regarding claim(s) 24. Kojima in view of Sussman further teaches further comprising updating, at the resource, information associated with the intermediary value (Sussman: [0052] Using one or more robots 18, a map of the warehouse 10 must be created and dynamically updated to determine the location of objects, both static and dynamic, as well as the locations of various fiducial markers dispersed throughout the warehouse. To do this, one of the robots 18 navigate the warehouse and build/update a map 10a,).
Regarding claim(s) 25. Kojima in view of Sussman further teaches wherein the work area comprises a field (Sussman: [0012] FIG. 1 is a top plan view of an order-fulfillment warehouse;).
Regarding claim(s) 26. Kojima in view of Sussman further teaches wherein at least one of the one or more machine-readable optical images encodes an intermediary value usable to retrieve the localization data or the information regarding one or more work tasks (Sussman: [0042] While the description provided herein is focused on picking items from bin locations in the warehouse to fulfill an order for shipment to a customer, the system is equally applicable to the storage or placing of items received into the warehouse in bin locations throughout the warehouse for later retrieval and shipment to a customer. The invention is also applicable to inventory control tasks associated with such a warehouse system, such as, consolidation, counting, verification, inspection and clean-up of products.).
Regarding claim(s) 27. Kojima in view of Sussman further teaches wherein the first mode comprises a scan with a first resolution and the second mode comprises a scan with a second resolution that is higher than the first resolution (Sussman: [0053] Robot 18 utilizes its laser-radar 22 to create/update map 10a of warehouse 10 as robot 18 travels throughout the space identifying open space 112, walls 114, objects 116, and other static obstacles such as shelves 12a in the space, based on the reflections it receives as the laser-radar scans the environment.).
Regarding claim(s) 30. Kojima in view of Sussman further teaches wherein the working machine is configured to autonomously control operation of the working machine based on the localization data (Sussman: [0044] In a preferred embodiment, a robot 18, shown in FIGS. 2A and 2B, includes an autonomous wheeled base 20 having a laser-radar 22. The base 20 also features a transceiver (not shown) that enables the robot 18 to receive instructions from the order-server 14, and a pair of digital optical cameras 24a and 24b.).
Claim(s) 2, 4, 29 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kojima US2021/0312661 (“Kojima”) in view of Sussman et al. US2019/0092184 (“Sussman”) further in view of Boyless et al. US2020/0389601 (“Boyless”).
Regarding claim(s) 2, 4, 29. Kojima in view of Sussman further silent to the machine localization system comprises one or more global navigation satellite system (GNSS) receivers configured to receive receives GNSS localization data, and the machine localization system is configured to combine the GNSS localization data with the localization data obtained by reading the one or more machine-readable optical images on the one or more markers to determine the position of the working machine or the one or more positions of the one or more markers.
Boyless teaches global navigation satellite system (GNSS) receivers configured to receive receives GNSS localization data, and the machine localization system is configured to combine the GNSS localization data with the localization data obtained by reading the one or more machine-readable optical images on the one or more markers to determine the position of the working machine or the one or more positions of the one or more markers ([0011-0014] receiving GNSS-measured device state data comprising device orientation and device position; receiving device sensor orientation data from the device orientation sensor; performing fiduciary recognition with respect to the spherical camera images; performing estimation of a pose of the at least one fiduciary marker with respect to the spherical camera images; determining a transformation of the device display with respect to the spherical camera using at least the spherical camera images, the GNSS-measured device state data, and the device sensor orientation data; and determining an absolute device state using the transformation; wherein: the absolute device state comprises absolute device orientation and absolute device position; and spherical image based self-localization of the electronic device is determined.).
It would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to further modify the combination the system and method of Kojima in view Sussman with Boyless’s teaching above to improve the accuracy device positioning determination and one of ordinary skill before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Inquiry
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TRUC M DO whose telephone number is (571)270-5962. The examiner can normally be reached on 9AM-6PM.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ramón Mercado, Ph.D. can be reached on (571) 270-5744. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TRUC M DO/Primary Examiner, Art Unit 3658
Prosecution Insights