DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention
Claims 1–14 and 17–19 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which Applicant regards as the invention.
The purpose of 35 U.S.C. 112(b) is to ensure that the scope of the claims is reasonably certain to one of ordinary skill in the art. See Nautilus, Inc. v. Biosig Instruments, Inc., 572 U.S. 898 (2014).
The following claim terminology renders the scope of these claims indefinite.
Independent claims 1 and 17 recite: "annotation data indicative of the debris."
The claim does not define what constitutes "annotation data." It is unclear whether the annotation data comprises: a bounding box, a segmentation mask, a confidence score, pixel coordinates, metadata, text, vectors, polygons, heat maps, labels, object identifiers, or any other representation.
Although dependent claim 4 later recites that the annotation data comprises a bounding box, independent claims 1 and 17 remain broad enough to encompass innumerable forms of annotation.
Accordingly, one of ordinary skill cannot determine the metes and bounds of the claimed annotation data.
Claims 6, 10, and 11 recite: determining a target image portion.
The claims fail to define: what constitutes the target image portion; how it is selected;
whether it is a pixel, region, object boundary, bounding box, segmentation mask, crop, or arbitrary image region.
Because no objective boundaries are provided, the scope of the limitation is uncertain.
Claims 6, 10, and 11 recite: mapping the target image portion to a target location.
These claims fail to specify: what mapping algorithm is used, what coordinate systems are employed, whether calibration data is required, whether the mapping is approximate or exact,
whether multiple image locations correspond to one workspace location, or what degree of mapping accuracy satisfies the claim.
Accordingly, the metes and bounds of the claimed mapping step cannot be reasonably determined.
Claims 6, 10, and 11 recite: target location.
These claims fail to define whether the target location corresponds to: a grid cell, an intersection, a track segment, an image coordinate, a physical coordinate, a container, or any other workspace feature. Thus, the scope of the limitation is uncertain.
Claims 6–8 recite: exclusion zone.
These claims fail to define: the boundaries of the exclusion zone; how large the exclusion zone is; how it is determined; whether it includes one grid cell, multiple grid cells, track segments,
or arbitrary workspace regions.
Dependent claim 7 merely states that the exclusion zone comprises adjacent grid cells but provides no objective standard regarding how many adjacent cells are required.
Accordingly, the scope of the exclusion zone remains indefinite.
Claim 7 recites: adjacent.
This claim does not specify whether "adjacent" includes: orthogonally adjacent cells,
diagonally adjacent cells, cells within one transport move, or any neighboring region.
Because no objective standard is provided, the limitation is indefinite.
Claim 8 recites: the exclusion zone is lifted.
The claim does not explain: what constitutes lifting, whether the exclusion zone is deleted,
disabled, ignored, overwritten, or otherwise modified. Therefore the scope of the limitation is uncertain.
Claim 9 recites: master controller.
The claim fails to identify: whether the master controller is centralized, distributed,
hardware, software, cloud-based, local, or part of the transport device. Accordingly, the limitation lacks reasonably certain scope.
Claims 10 and 13 recite: service device.
The claim merely states that the service device comprises a cleaning mechanism.
These claims do not define: what constitutes a service device; whether it is autonomous;
manually operated; permanently installed; or movable. Thus, the metes and bounds are uncertain.
Claims 12 and 13 recite: classification data representative of a class of debris.
These claims do not define: what classes exist; how many classes exist; how the classes are determined; or what characteristics distinguish one class from another. Consequently, the limitation lacks objective boundaries.
Claim 13 recites: first class and second class.
The claims provide no objective criteria distinguishing one class from another.
Accordingly, these limitations are indefinite.
Claims 10 and 13 recite: cleaning mechanism with means for removing debris.
The phrase "means for removing debris" invokes 35 U.S.C. 112(f) because the claim uses the term "means" together with purely functional language and recites insufficient structure for performing the claimed function. The specification must therefore disclose corresponding structure linked to the claimed function. If corresponding structure is absent or insufficiently disclosed, the claims are indefinite under 35 U.S.C. 112(b).
See MPEP §2181.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1–14 and 17-19 are rejected under 35 U.S.C. §101 because the claimed invention is directed to a judicial exception without reciting significantly more than the judicial exception.
Step 1
Claims 1–14 and 17-19 are directed to one of the four statutory categories of invention, namely, a process (claims 1–13), a machine (claims 17–19), and a manufacture/apparatus (claim 14).
Accordingly, the analysis proceeds to Step 2A.
Step 2A, Prong One
Claims 1–14 and 17-19 recite judicial exceptions.
Representative independent claim 1 recites, in substance:
obtaining image data;
processing the image data using an object detection model;
determining whether image data includes annotation data indicative of debris.
These limitations recite abstract ideas.
More particularly, the claims recite mental processes, including: observation of image information; evaluation of image information; recognizing objects; identifying debris;
determining whether debris exists; generating annotation information; classifying debris; determining locations; determining exclusion zones; determining control actions.
These are acts capable of being performed in the human mind or with pen and paper, even if the claims recite generic computer implementation.
See MPEP §2106.04(a)(2).
The claims further recite mathematical concepts, including: object detection models; convolutional neural networks; image processing; mapping image coordinates to workspace coordinates; determining classification data; generating bounding boxes;
determining target image portions; generating exclusion-zone data.
These limitations employ mathematical relationships, mathematical calculations, mathematical modeling, or mathematical algorithms used to analyze data.
See MPEP §2106.04(a)(1).
Accordingly, the claims recite the judicial exceptions of: mental processes; and
mathematical concepts.
Step 2A, Prong Two
These claims as a whole do not integrate the judicial exception into a practical application.
Although the claims additionally recite: a workspace, a grid, tracks, transport devices, containers, image sensors, cameras, control systems, service devices, robotic manipulators, these additional elements merely identify the environment in which the abstract idea is performed.
These claims merely: collect image information; analyze the information; determine the presence or absence of debris; generate annotation information; output data; output control signals.
These claims do not improve: image sensors, cameras, neural networks,
object detection models, warehouse grids, robotic transport devices, communication protocols, computer architecture, memory, processors, image acquisition hardware, machine learning technology itself.
Instead, these claims merely use conventional image processing and conventional machine-learning techniques as tools to automate recognition and decision making.
The claims therefore merely apply the judicial exception using generic computer technology.
These additional elements simply constitute insignificant extra-solution activity, including: obtaining image data, outputting annotation data, outputting updated images,
outputting exclusion-zone information, outputting deployment signals, outputting shutdown signals.
Likewise, reciting a CNN does not integrate the judicial exception into a practical application because the CNN merely performs the claimed mathematical analysis.
The recited transport devices, service devices, and robotic manipulators merely receive the result of the analysis and perform conventional actions in response.
Accordingly, the claims do not integrate the judicial exception into a practical application.
Step 2B
These claims do not include additional elements amounting to significantly more than the judicial exception.
These additional elements include: generic processors, image sensors, cameras, object detection models, convolutional neural networks, transport devices, warehouse tracks, robotic manipulators, service devices, control systems, computer-readable program code.
Viewed individually, these elements perform only their well-understood, routine, and conventional functions, such as: capturing images; executing software; processing image data; generating output data; transmitting signals; controlling conventional warehouse equipment.
Viewed as an ordered combination, the claims merely automate conventional warehouse inspection by: acquiring image information; analyzing the information; identifying debris; determining locations; generating annotations; producing output information; optionally issuing control instructions.
The ordered combination therefore merely automates what previously could be accomplished by a human warehouse operator visually inspecting the grid and directing cleanup or traffic control.
Automation of a manual process using generic computing technology does not amount to significantly more than the judicial exception.
See Alice Corp. v. CLS Bank Int'l, 573 U.S. 208 (2014).
These claims also do not recite: any unconventional neural-network architecture;
any improved object detection algorithm; any improvement to computer functionality;
any improvement to camera hardware; any improvement to image acquisition;
any improvement to communication technology; any new robotic control architecture.
Rather, these claims merely invoke generic machine-learning models as tools to analyze data and generate control outputs.
Accordingly, the claims do not amount to significantly more than the judicial exception.
Representative dependent claims 2–5 merely recite: generating annotation data; outputting annotated images; bounding boxes; convolutional neural networks.
These limitations merely further define the mathematical processing and therefore remain directed to the judicial exception.
Claims 6–13 merely add post-solution activity including: mapping target locations; generating exclusion zones; lifting exclusion zones; shutting down transport devices; dispatching cleaning devices; dispatching robotic manipulators; classifying debris; selecting different responses depending on debris class.
These limitations merely use the results of the abstract analysis to control conventional warehouse equipment and therefore constitute insignificant post-solution activity.
Claim 14 merely recites a generic data-processing apparatus configured to execute the abstract method and therefore does not add significantly more.
Claim 17 merely recites: an image sensor; an object detection model; determining debris; outputting annotation data.
Claims 18–19 merely specify: a wide-angle camera; and a convolutional neural network.
These limitations likewise merely implement the abstract idea using generic computer and imaging technology.
Accordingly, claims 1–19 are directed to judicial exceptions, namely mental processes and mathematical concepts, and the additional claim elements, individually and as an ordered combination, do not amount to significantly more than those judicial exceptions. Therefore, claims 1–19 are ineligible under 35 U.S.C. §101.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1–14, 17-19 are rejected under 35 U.S.C. §103 as being unpatentable over Shydo et al. (U.S. Patent No. 10,133,276 B1; hereinafter "Shydo") in view of Jikihara (U.S. Pub. No. 2011/0144850 A1; hereinafter "Jikihara"), and further in view of Sun et al. (U.S. Pub. No. 2017/0206431 A1; hereinafter "Sun").
With respect to claims 1 and 14, Shydo discloses a computer-implemented method of detecting debris in a workspace (Shydo, Abstract; Fig. 1; Fig. 4A inasmuch as the detected object constitutes an obstruction present within the robot workspace that is detected by onboard processing circuitry for subsequent operational control..)
obtaining image data representative of an image of at least part of the workspace (Shydo, Fig. 4A ("Scan Object with 3D Scanner"); Abstract; Fig. 1; col. 1, ll. 54–67; col. 5, ll. 20–67.)
However, Shydo does not expressly disclose that the obtained image data is processed using an object detection model trained to detect instances of debris on a grid, nor does Shydo expressly disclose generating annotation data identifying detected debris within an image using a trained deep-learning object detection model.
Sun remedies these deficiencies. Sun discloses processing input image data using an object detection and classification network that includes a Deep Convolutional Neural Network (Deep CNN), a Region Proposal Network (RPN), and a proposal classifier. Sun teaches receiving an input image, generating convolutional feature maps, identifying candidate object locations, producing rectangular bounding boxes corresponding to detected objects, classifying detected objects, and outputting object detection results. (Sun ¶¶ 43–49; ¶¶ 64–73; Figs. 3–5.)
Accordingly, Sun teaches: processing the image data with an object detection model trained to detect instances of debris on the grid (Sun ¶¶ 43–49; ¶¶ 64–73because the disclosed Deep CNN/RPN/Fast R-CNN architecture constitutes a trained object detection model configured to detect objects within image data and localize the detected objects using learned convolutional features.)
determining, based on the processing, whether the image includes annotation data indicative of the debris in the image (Sun ¶¶ 44–48; ¶¶ 70–73; Fig. 4; Fig. 5 teaches that the Region Proposal Network identifies candidate object locations and generates proposals represented by rectangular bounding boxes, while the proposal classifier classifies each detected object and outputs object classifications and confidence scores representative of the detected object within the image. These generated proposals constitute annotation data identifying the detected object within the image.), and
Jikihara further teaches employing obstacle detection within a moving apparatus operating along predetermined travel routes, detecting obstacles located on a travel route, determining whether alternate routing is possible, and when rerouting is unavailable, requesting obstacle removal. Jikihara therefore reinforces the use of obstacle detection results to control autonomous movement within a structured travel environment. (Jikihara ¶¶ 18–19; Figs. 2, 4A–4B.)
Shydo however, does not expressly teach that the workspace comprises a grid formed by a first set of tracks extending in a first direction and a second set of tracks extending in a second direction transverse to the first direction, as specifically recited.
Jikihara remedies this deficiency by disclosing a moving apparatus that travels along predetermined travel routes, prepares travel routes to destinations, detects obstacles located on those routes, determines whether alternate routes exist, and controls movement based upon the detected obstacle locations. Jikihara further teaches a structured travel environment in which a moving body travels along intersecting guide paths and movement is controlled according to the predetermined travel network. Such teachings correspond to the claimed workspace including intersecting tracks arranged in different directions for controlled movement of transport devices. (Jikihara, Abstract; Figs. 1–2, 4A–4B.)
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the structured guide-track transportation system of Jikihara into the warehouse object-detection system of Shydo in order to implement the disclosed autonomous transport operations within a predefined guide-path infrastructure, thereby providing predictable navigation, obstacle avoidance, and movement control for autonomous transport devices.
Furthermore, Shydo does not expressly disclose that the detected object is represented by annotation data indicative of the debris in the image, such as image-localization information generated by an image-based object detector.
Sun teaches this limitation by disclosing an object-detection framework in which an input image is processed through a trained deep convolutional neural network, a Region Proposal Network generates candidate object regions, and object proposals are represented by corresponding localization information, including rectangular bounding boxes associated with detected objects. Sun further teaches classifying the detected objects and outputting the detected object together with its associated localization information, thereby generating annotation information indicative of the detected object within the image. (Sun, Figs. 3–5.)
Therefore, Sun teaches the claimed limitation of: determining, based on the processing, whether the image includes annotation data indicative of the debris in the image. because the disclosed object-detection network determines whether detected objects are present within an image and outputs corresponding localization annotations identifying the detected objects.
It would have been obvious to one of ordinary skill in the art at the time the invention was made to modify the warehouse object-detection system of Shydo with the structured guide-path movement control of Jikihara and the trained image-based object-detection techniques of Sun. Shydo is directed to detecting objects encountered by autonomous warehouse robots and determining appropriate operational responses. Jikihara teaches controlling autonomous movement within a structured guide-path environment while detecting obstacles and modifying travel based upon obstacle locations. Sun teaches that deep convolutional neural-network-based object detection provides improved accuracy in locating and identifying objects within image data through the generation of object proposals and localization annotations.
A person of ordinary skill in the art would have recognized that employing Sun's trained image-based object detector within Shydo's warehouse object-detection system would have predictably improved the accuracy and reliability of detecting debris or obstructions encountered by autonomous warehouse robots. Likewise, incorporating Jikihara's structured guide-path navigation into Shydo's warehouse environment would have predictably enabled autonomous transport devices to navigate predetermined track networks while responding to detected debris or obstructions. The combination merely substitutes one known object-detection technique for another and applies known guide-path movement control to an autonomous warehouse environment to achieve the predictable result of more accurate debris detection and safer autonomous transport operations.
With respect to claim 2, Shydo discloses obtaining image information representative of objects encountered within a warehouse environment and processing the detected information to identify objects for subsequent operational control of autonomous warehouse robots. (Shydo, Abstract; Figs. 1, 4A; col. 3; col. 5.)
However, Shydo does not expressly disclose: wherein the method comprises generating the annotation data.
Sun remedies this deficiency by disclosing that an input image is processed using a trained object detection network, candidate object regions are generated by a Region Proposal Network, and localization information corresponding to detected objects is generated and output. The generated object localization information, including bounding boxes associated with detected objects, constitutes annotation data representative of the detected object. (Sun, Figs. 3–5.)
Accordingly, it would have been obvious to one of ordinary skill in the art to incorporate Sun's annotation-generation techniques into the warehouse object detection system of Shydo in order to improve localization and identification of detected debris while producing annotated object information suitable for subsequent warehouse control operations.
With respect to claim 3, Shydo discloses displaying and utilizing detected object information to determine operational responses for warehouse robots. (Shydo, Abstract; Figs. 1 and 4.)
Shydo, however, does not expressly teach: outputting an updated version of the image including the annotation data.
Sun teaches outputting object detection results in conjunction with the processed image, including object localization information identifying the detected object regions after processing by the object detection network. (Sun, Figs. 4–5.)
Therefore, it would have been obvious to modify Shydo's warehouse object detection system to output the processed image including the generated object annotations in order to provide visual confirmation of detected debris for warehouse operators and autonomous system controllers.
With respect to claim 4, Shydo detects objects encountered by autonomous warehouse robots but does not expressly disclose that the annotation data comprises a bounding box.
Sun expressly teaches generating rectangular object proposals identifying candidate object locations within an image, wherein each detected object is localized using a rectangular bounding box generated by the Region Proposal Network and subsequent classifier. (Sun, Figs. 3–5.)
Accordingly, it would have been obvious to utilize the bounding-box localization technique taught by Sun within the warehouse object detection system of Shydo because bounding boxes constitute a conventional and well-known technique for graphically identifying detected objects within digital images while improving visualization and subsequent processing of object locations.
With respect to claim 5, Shydo teaches processing detected sensor information to identify objects located within the warehouse environment but does not expressly disclose that the object detection model comprises a convolutional neural network.
Sun expressly teaches employing a Deep Convolutional Neural Network (Deep CNN) together with a Region Proposal Network and Fast R-CNN classifier for object detection and classification. (Sun, Figs. 3–5.)
It would have been obvious to substitute Sun's CNN-based object detection model for the object detection system of Shydo because Sun teaches that convolutional neural networks provide accurate and efficient object detection and localization from image data. The substitution merely applies one known object-detection technique for another to obtain the predictable benefit of improved object detection performance, consistent with KSR Int'l Co. v. Teleflex Inc., 550 U.S. 398 (2007).
With respect to claim 6, Shydo discloses wherein one or more transport devices are arranged to selectively move in at least one of the first or second direction on the tracks, and to handle a container stacked beneath the tracks within a footprint of a single grid cell"
inasmuch as autonomous warehouse transport robots travel throughout a structured warehouse storage system to transport inventory holders while navigating predetermined travel paths. (Shydo, Abstract; Fig. 1, 4A–4C.)
Shydo further teaches detecting an object within an image or scan of the workspace, classifying the detected object, and determining the object location relative to the robot path before taking corrective action. (Shydo, Abstract; Figs. 1 and 4A.)
However, Shydo does not expressly disclose: determining a target image portion of the image based on the annotation data.
Sun remedies this deficiency by teaching that an input image is processed using a Deep Convolutional Neural Network, a convolutional feature map is generated, a Region Proposal Network identifies candidate object regions, and a proposal classifier determines the location and classification of each detected object. The generated object proposals correspond to target image portions containing detected objects. (Sun, Abstract; Figs. 3–5.)
Accordingly, Sun teaches: determining a target image portion of the image based on the annotation data. Shydo further does not expressly teach: mapping the target image portion to a target location in the workspace.
Jikihara teaches obtaining obstacle position information, recording obstacle positions, determining whether the detected obstacle lies on the planned route, updating obstacle position information, and associating the detected obstacle with a corresponding physical location used for movement control. (Jikihara, Abstract; Figs. 2, 4A, and 4B.)
Thus, Jikihara teaches mapping a detected obstacle identified from sensing information to a corresponding location within the movement workspace.
Shydo also does not expressly disclose: determining, based on the mapping, an exclusion zone in the workspace, comprising the target location, in which the one or more transport devices are to be prohibited from entering.
Jikihara teaches that when an obstacle is detected on the travel route, the moving apparatus determines whether an alternate route exists and prevents the moving body from traversing the obstructed region until the obstacle is removed or an alternate route becomes available. The obstacle location therefore functions as a prohibited movement region surrounding the detected obstacle. (Jikihara, Abstract; Figs. 4A and 4B.)
Accordingly, Jikihara teaches establishing an exclusion region surrounding the detected obstacle to prevent transport devices from entering the obstructed area.
Shydo further does not expressly disclose: outputting, to a control system, exclusion-zone data representative of the exclusion zone for implementing the exclusion zone in the workspace.
Shydo teaches transmitting messages to a central control system after object detection and classification so that the central controller may control robot operation, including stopping, rerouting, or other corrective actions. (Shydo, Abstract; Figs. 1 and 4A.)
Jikihara likewise teaches providing obstacle information to route-preparing and movement-control functions to prevent movement through obstructed locations. (Jikihara, Figs. 2 and 4A.)
Therefore, the combined teachings suggest outputting exclusion-zone information to a warehouse control system for implementation by autonomous transport devices.
It would have been obvious to one of ordinary skill in the art to modify the warehouse object-detection system of Shydo by incorporating Sun's image-localization techniques together with Jikihara's obstacle-position mapping and route-restriction techniques in order to accurately associate detected debris with physical warehouse locations and prohibit autonomous transport devices from entering those locations, thereby improving operational safety and avoiding collisions with detected debris. Such a modification merely combines known object-detection techniques with known warehouse movement-control techniques to obtain the predictable result of safer autonomous warehouse operation.
With respect to claim 7, Shydo teaches rerouting autonomous robots around detected objects so that the robots avoid the obstructed portion of the warehouse travel path. (Shydo, Abstract; Fig. 1; Fig. 4A.)
Jikihara teaches preventing movement through an obstacle location and selecting alternate routes around the detected obstacle until removal of the obstacle has occurred. Such rerouting necessarily prevents movement not only through the obstacle position itself but also through the surrounding movement region required for safe navigation. (Jikihara, Abstract; Figs. 4A–4B.)
Accordingly, it would have been obvious to define the exclusion zone as comprising a plurality of adjacent grid cells surrounding the detected debris, because expanding the prohibited region around an obstacle represents a well-known safety measure for autonomous vehicle navigation to account for positioning uncertainty, robot dimensions, and collision avoidance while producing the predictable result of improved operational safety.
With respect to claim 8, Shydo teaches repeatedly scanning the robot pathway using onboard sensors while the robot continues warehouse operation, thereby obtaining additional sensor data representative of the environment after previous object detections. (Shydo, Figs. 1 and 4A–4C.)
Accordingly, Shydo teaches: obtaining further image data representative of a further image of at least part of the workspace.
Sun teaches processing newly received images using the same trained object detection network to determine whether objects continue to be present within subsequent images. (Sun, Abstract; Figs. 1 and 4.)
Accordingly, Sun teaches: processing the further image data with the object detection model; and determining, based on the processing, whether the further image includes debris on the grid.
Jikihara teaches that once the obstacle is removed or an alternate route becomes available, the moving apparatus resumes normal travel through the previously restricted region. (Jikihara, Figs. 4A and 4B.)
Therefore, Jikihara teaches: causing, in response to determining that the further image does not include debris on the grid, the exclusion zone to be lifted."
It would have been obvious to remove the movement restriction after confirming that the obstacle is no longer present because doing so restores normal warehouse traffic while avoiding unnecessary interruption of autonomous transport operations.
With respect to claim 9, Shydo teaches that after detecting and classifying an object obstructing the planned route, the robot communicates with the central control system and causes the robot to stop when the detected object cannot be safely traversed or classified. (Shydo, Abstract; Figs. 1 and 4A.)
Accordingly, Shydo teaches:
outputting, in response to determining that the image includes debris on the grid, a signal to a master controller of the one or more transport devices."
Shydo further teaches stopping robot operation pending further instructions or a manual reset after object detection. (Shydo, Abstract; Fig. 4A.)
Therefore, Shydo teaches: to cause the master controller to shut down the one or more transport devices.
Alternatively, Jikihara teaches stopping movement of the moving body when an obstacle cannot be safely removed or bypassed, thereby reinforcing the use of centralized movement control to halt transport devices in response to obstacle detection. (Jikihara, Figs. 4A–4B.)
It would have been obvious to communicate the detected obstacle condition to a master controller to suspend operation of autonomous transport devices until the detected debris has been removed, because centralized shutdown of autonomous transport devices is a predictable safety response that prevents collisions, protects transported inventory, and preserves reliable warehouse operation.
With respect to claim 10, Shydo teaches detecting an object within a warehouse environment, classifying the detected object, determining the object location relative to the robot travel path, and communicating the detected object information to a central controller for subsequent operational control of autonomous warehouse robots. (Shydo, Abstract; Figs. 1, 4A–4C.)
However, Shydo does not expressly disclose: determining a target image portion of the image based on the annotation data.
Sun remedies this deficiency by teaching processing an input image using a Deep Convolutional Neural Network, generating object proposals through a Region Proposal Network, and producing localized object regions corresponding to detected objects within the image. The object proposals correspond to target image portions determined from annotation information. (Sun, Abstract; Figs. 3–5.)
Shydo further does not expressly disclose: mapping the target image portion to a target location in the workspace.
Jikihara teaches recording obstacle position information, determining the location of detected obstacles within the movement environment, and using the obstacle position for movement control and route planning. (Jikihara, Abstract; Figs. 2, 4A–4B.)
Shydo additionally does not expressly disclose: outputting a signal for deploying a service device to the target location, the service device being arranged to selectively move in at least one of the first or second directions on the tracks and comprising a cleaning mechanism with means for removing debris present on the grid.
Jikihara teaches that, after determining the obstacle location, the system outputs a request for obstacle removal and identifies a person or entity to remove the obstacle so that normal movement may resume. The obstacle-removal request constitutes deployment of a service resource to the mapped obstacle location. (Jikihara, Abstract; Figs. 2 and 4B.)
It would have been obvious to substitute Jikihara's obstacle-removal operation with an autonomous cleaning robot operating on the same warehouse track system because automated debris removal eliminates manual intervention, improves warehouse throughput, and represents a predictable application of known autonomous service devices to known warehouse transport systems.
With respect to claim 11, Shydo teaches a warehouse including autonomous transport robots transporting inventory holders throughout storage locations under centralized warehouse control. (Shydo, Abstract; Fig. 1.)
Sun teaches determining a target image portion corresponding to a detected object through object localization generated by the object detection network. (Sun, Figs. 3–5.)
Jikihara teaches mapping the detected obstacle position to a corresponding location within the movement environment and determining whether the obstacle interferes with movement along a particular route. (Jikihara, Figs. 2, 4A.)
Accordingly, the combined references teach: determining a target image portion; mapping the target image portion to a target location in the workspace; and determining whether the detected debris is adjacent to a predetermined operational area associated with warehouse operations.
Shydo, however, does not expressly disclose: outputting a signal to cause a robotic manipulator of a picking station to remove debris from the tracks.
Jikihara teaches outputting obstacle-removal requests directed to an external removal mechanism or operator after determining the obstacle location and selecting the obstacle to be removed. (Jikihara, Abstract; Fig. 4B.)
It would have been obvious to direct an existing robotic manipulator located at a warehouse picking station to remove nearby debris instead of requesting manual removal because utilizing an already-installed robotic manipulator to perform debris removal merely represents the predictable use of an existing robotic actuator for another known manipulation task, thereby reducing response time and improving warehouse efficiency.
With respect to claim 12, Shydo teaches detecting objects within the warehouse environment and classifying the detected objects to determine appropriate operational responses. The detected objects may include inventory items, warehouse equipment, persons, or unidentified objects. (Shydo, Abstract; Fig. 1.)
However, Shydo does not expressly disclose: processing, in response to determining that the image includes debris on the grid, the image data with one or more object-classification models trained to classify debris.
Sun expressly teaches processing image data using trained object detection and object classification networks, including a Deep Convolutional Neural Network, Region Proposal Network, and proposal classifier that classify detected objects and generate corresponding confidence scores. (Sun, Abstract; Figs. 1, 4, and 5.)
Accordingly, Sun teaches:
processing the image data with one or more trained object-classification models.
Sun further teaches determining an output representative of the class assigned to each detected object. Accordingly, Sun teaches: determining classification data representative of a class of debris to which the detected debris belongs, based on the processing."
It would have been obvious to utilize Sun's trained object-classification network within Shydo's warehouse object-detection system because classifying detected debris enables more appropriate downstream responses depending upon the detected object type, thereby improving warehouse safety and operational efficiency.
With respect to claim 13, Sun teaches generating classification information representative of the class assigned to each detected object through the trained object-classification network. (Sun, Abstract; Figs. 3–5.)
However, neither Shydo nor Sun expressly teaches selecting different operational responses depending upon the determined debris class.
Jikihara teaches selecting different obstacle-removal actions depending upon the nature, removability, and characteristics of the detected obstacle, including determining whether an obstacle should be removed, selecting the obstacle to be removed, identifying an appropriate request target, and issuing corresponding obstacle-removal requests. (Jikihara, Abstract; Figs. 2 and 4B.)
Accordingly, it would have been obvious that: deploying, in response to the classification data being indicative of the detected debris belonging to a first class of debris, a service device arranged to move on the tracks and comprising a cleaning mechanism with means for removing debris present on the grid; is suggested by combining Sun's object classification with Jikihara's obstacle-removal operation. Likewise, it would have been obvious that: arranging, in response to the classification data being indicative of the detected debris belonging to a second class of debris, any transport devices on the grid, the transport devices being arranged to move on the tracks to transport containers between grid cells; is suggested by Shydo's centralized control of warehouse transport robots together with Jikihara's selective routing and movement control based upon obstacle characteristics. (Shydo, Abstract; Fig. 1; Jikihara, Figs. 2 and 4A–4B.)
It would have been obvious to one of ordinary skill in the art to vary the response to detected debris based upon its classified type because different categories of obstacles require different corrective actions. Employing Sun's trained classification network enables the warehouse control system of Shydo to distinguish debris types, while Jikihara teaches selecting different obstacle-removal operations based on obstacle characteristics. Combining these teachings merely applies known classification techniques to known warehouse movement-control systems to achieve the predictable result of selecting the most appropriate response for each detected obstacle, thereby improving safety, reducing unnecessary service operations, and increasing warehouse efficiency.
With respect to claim 17, Shydo discloses a detection system for detecting debris in a workspace comprising a grid formed by a first set of tracks extending in a first direction and a second set of tracks extending in a second direction transverse to the first direction.
The warehouse environment disclosed by Shydo includes autonomous transport robots travelling on predetermined travel paths throughout a structured storage workspace while detecting objects that obstruct the transport routes. (Shydo, Abstract; Fig. 1.)
Shydo further teaches: an image sensor to capture an image of at least part of the workspace. Specifically, Shydo discloses an object detection system including sensors such as a three-dimensional scanner configured to scan the robot operating environment for objects located within the travel path, thereby obtaining image or scan information representative of the workspace. (Shydo, Abstract; Fig. 4A.)
However, Shydo does not expressly disclose: an object detection model trained to detect instances of debris on the grid.
Sun remedies this deficiency by expressly teaching an object detection network including a Deep Convolutional Neural Network, a Region Proposal Network, and a proposal classifier trained to detect and classify objects within digital images. Sun further teaches receiving an input image, generating convolutional feature maps, identifying candidate object regions, classifying detected objects, and outputting object classification results with associated confidence scores. (Sun, Abstract; Figs. 1, 4, and 5.)
Accordingly, Sun teaches: an object detection model trained to detect instances of debris on the grid."
Shydo further teaches processing sensor information to determine whether an object exists within the robot pathway and thereafter classifying the detected object for subsequent operational control. (Shydo, Abstract; Fig. 1.)
However, Shydo does not expressly disclose processing image data using a trained deep-learning object detection model.
Sun teaches: to process the image data with the object detection model. by processing an input image through the Deep CNN, Region Proposal Network, and proposal classifier. (Sun, Abstract; Figs. 4 and 5.)
Sun further teaches: determine, based on the processing, whether the image includes debris on the grid, by determining whether one or more objects are present within the input image and assigning object classifications to the detected objects. (Sun, Abstract; Fig. 4.)
Sun additionally teaches: output, in response to determining that the image includes debris on the grid, annotation data indicative of the debris in the image. because the Region Proposal Network generates localized object proposals, and the proposal classifier outputs object classifications associated with localized object regions corresponding to detected objects within the image. (Sun, Figs. 3–5.)
Jikihara further reinforces the use of obstacle detection within an autonomous movement environment by teaching detection of obstacles located on predetermined travel routes and using obstacle position information for movement control and route management. (Jikihara, Abstract; Figs. 2 and 4A.)
It would have been obvious to one of ordinary skill in the art to incorporate the trained image-based object detection network of Sun into the warehouse object detection system of Shydo while employing the structured movement environment of Jikihara in order to improve the accuracy of detecting debris within a warehouse workspace and to provide localized annotation information corresponding to detected debris. Such modification merely substitutes one known object detection technique for another while applying it within a known autonomous warehouse control environment to obtain the predictable result of improved debris detection and warehouse safety.
With respect to claim 18, Shydo teaches an object detection system including sensors for detecting objects within the warehouse environment but does not expressly disclose that the image sensor comprises a wide-angle or ultra-wide-angle camera. (Shydo, Abstract.)
Sun teaches receiving an input image covering a scene that is processed by an object detection network for detecting multiple objects within the image. A person of ordinary skill in the art would have understood that wide-angle image sensors were conventional image acquisition devices for capturing larger portions of a monitored environment to facilitate object detection using convolutional neural networks. (Sun, Abstract; Figs. 3–5.)
Accordingly, it would have been obvious to employ a wide-angle or ultra-wide-angle camera as the image sensor of Shydo in order to increase the observable area of the warehouse workspace, thereby reducing the number of cameras required while improving debris detection coverage. The substitution merely involves the predictable use of one well-known image sensor for another to obtain improved field-of-view characteristics.
With respect to claim 19, Shydo teaches detecting and classifying objects encountered within the warehouse environment but does not expressly disclose that the object detection model comprises a convolutional neural network. (Shydo, Abstract; Fig. 1.)
Sun expressly teaches that the object detection network includes a Deep Convolutional Neural Network (Deep CNN) configured to process input images, generate convolutional feature maps, generate object proposals through a Region Proposal Network, and classify detected objects using a proposal classifier. (Sun, Abstract; Figs. 1, 4, and 5.)
Accordingly, Sun teaches: wherein the object detection model comprises a convolutional neural network. It would have been obvious to substitute Sun's Deep Convolutional Neural Network for the object detection system employed by Shydo because convolutional neural networks were well known for providing superior object recognition, localization, and classification accuracy when processing image data. Incorporating Sun's CNN-based detector into Shydo's warehouse object detection system would have predictably improved the detection and localization of debris within the warehouse environment while maintaining the same overall system functionality, consistent with the rationale set forth in KSR Int'l Co. v. Teleflex Inc., 550 U.S. 398 (2007).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROKIB MASUD whose telephone number is (571)270-5390. The examiner can normally be reached Mon-Fri 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Fahd Obeid can be reached at 571-270-3324. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROKIB MASUD/Primary Examiner, Art Unit 3627
Prosecution Insights