DETAILED ACTION
This Office action is in response to the Application filed on April 16, 2024, which is a national stage application under 35 U.S.C. §371 of International Application No. PCT/GB2022/052638, filed on October 17, 2022, which claims foreign priority to GB Application No. GB2114848.1, filed on October 18, 2021. Claims 1, 3-7, 9, 11-15, and 17 have been amended and entered via preliminary amendment. An action on the merits follows. Claims 1-17 are pending on the application.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
The following title is suggested: SAFETY STRUCTURE DAMAGE STATUS DETECTION SYSTEM.
Claim Objections
Claims 1 and 11 are is objected to because of the following informalities:
Claim 1 recites the limitation “recognise a safety structure in the acquired images … compare the recognised safety structure in the acquired image” in lines 5-7 of the claim. However, the claimed “the acquired image” in line 6 of the claim should recite “the acquired images” instead in order to avoid lack of antecedent basis issue.
Therefore, for examination purposes, the claimed “recognise a safety structure in the acquired images … compare the recognised safety structure in the acquired image” recited in lines 5-7 of the claim will be interpreted as “recognise a safety structure in the acquired images … compare the recognised safety structure in the acquired images”.
Claim 11 recites the limitation “determine the distance to the safety structure” in line 3 of the claim. However, the claimed “the distance” in line 6 of the claim should recite “a distance” instead in order to avoid lack of antecedent basis issue.
Therefore, for examination purposes, the claimed “determine the distance to the safety structure” recited in line 3 of the claim will be interpreted as “determine a distance to the safety structure”.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-8, 12-14, and 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Bistry et al. (US PG Publication No. 2021/0276842 A1), hereafter referred to as Bistry, Applicant cited prior art, in view of Boyle et al. (US PG Publication No. US 2018/0247137 A1), hereafter referred to as Boyle.
Regarding claim 1, Bistry discloses a damage detection system (Par. [0001]: a system for inspecting a warehouse, an industrial truck comprising such a system and a method for inspecting a warehouse using such a system; Par. [0020-42]: computing unit is further adapted to detect individual storage elements from the constructed three-dimensional data of the environment. Storage elements in the context of this invention refer to all possible logistics elements or units that are used or required either for transport requests or for the storage of goods in a warehouse. They include… storage aids such as pallets or containers, storage locations or bins, racks or racking elements such as vertical supports, horizontal supports… By comparing the three-dimensional data of the individual detected storage elements with the data stored in the warehouse management system or elsewhere, the computer unit can detect any storage irregularities and/or damage to individual storage elements… irregularities or damage to a high-bay warehouse are detected at an early stage, thus enhancing occupational safety and logistical efficiency… method preferably further includes detection of damage and irregularities of the storage condition; Par. [0059-64]: computing unit 60 constructs three-dimensional data of the environment in the form of a point cloud. Through suitable image processing, the computing unit 60 can recognize various storage elements from the constructed point cloud and, based on these, determine three-dimensional data of the individual recognized storage elements. Based on a comparison of the three-dimensional data of individual detected storage elements with corresponding data from the warehouse management system and/or with predefined reference data, the computing unit 60 is able to automatically detect damage or irregularities in a warehouse. FIG. 4 and FIG. 5 each illustrate an example of this… FIG. 4 shows part of a rack 84 comprising a plurality of horizontal supports 86 and vertical supports 88. It can be seen in FIG. 4 that rack damage has occurred in the form of a deformed horizontal support 86. When the industrial truck 1 with the system 50 passes the affected storage area, the computing unit 60 is able to recognize, from the constructed point cloud and corresponding image processing algorithms, that the horizontal support 86 is deformed. Based on this, the computing unit 60 can output an instruction to have the horizontal beam 86 repaired or replaced; a damage detection system (e.g. system and method for inspecting and automatically detecting damage or irregularities in a warehouse), for example), comprising:
a camera associated with a vehicle (Par. [0001]: a system for inspecting a warehouse, an industrial truck comprising such a system and a method for inspecting a warehouse using such a system; Par. [0031]: a truck which is adapted to perform a work function such as lifting or lowering a load; Par. [0059-71]: FIG. 3 contains a rough schematic diagram illustrating the structure of the system 50 for inspecting a warehouse that is used in the industrial truck 1 shown in FIG. 1. The system 50 includes the laser scanner 52 and the computing unit 60, the laser scanner 52 being disposed to scan the environment in order to generate scan data… for example, only one rack support… is scanned and recognized when the target storage location is reached (see scanning range 200). Based on this, the control system of the industrial truck, together with data concerning the dimensions of the target storage bin saved locally or externally (such as in the warehouse management system), can calculate the target destination position Ptarget… using additional sensors, such as a camera, which also detects the storage location and/or the environment near the storage location during the deposit operation; a camera associated with a vehicle (e.g. system and method for inspecting and automatically detecting damage or irregularities in a warehouse, including an industrial truck (i.e. a vehicle) and a camera, which is used to detect storage location(s) and/or the environment near the storage location(s) during operation of the industrial truck (i.e. a camera associated with a vehicle), as indicated above, for example), for example); and
a controller configured to process acquired [three-dimensional data] in order to:
recognise a safety structure in the [three-dimensional data] (Par. [0020-26]: computing unit is further adapted to detect individual storage elements from the constructed three-dimensional data of the environment. Storage elements in the context of this invention refer to all possible logistics elements or units that are used or required either for transport requests or for the storage of goods in a warehouse. They include… storage aids such as pallets or containers, storage locations or bins, racks or racking elements such as vertical supports, horizontal supports… The computing unit is preferably adapted to recognize different storage elements from a constructed point cloud through image processing and to determine three-dimensional data of the individual recognized storage elements based on these… computing unit is further preferably adapted to check a storage condition… computer unit can detect any storage irregularities and/or damage to individual storage elements; Par. [0059-71]: system 50 includes the laser scanner 52 and the computing unit 60, the laser scanner 52 being disposed to scan the environment in order to generate scan data, and the computing unit 60 being disposed to receive the scan data and construct three-dimensional data of the environment based on these… the computing unit 60 constructs three-dimensional data of the environment in the form of a point cloud. Through suitable image processing, the computing unit 60 can recognize various storage elements from the constructed point cloud and, based on these, determine three-dimensional data of the individual recognized storage elements… FIG. 4 shows part of a rack 84 comprising a plurality of horizontal supports 86 and vertical supports 88. It can be seen in FIG. 4 that rack damage has occurred in the form of a deformed horizontal support 86. When the industrial truck 1 with the system 50 passes the affected storage area, the computing unit 60 is able to recognize, from the constructed point cloud and corresponding image processing algorithms, that the horizontal support 86 is deformed. Based on this, the computing unit 60 can output an instruction to have the horizontal beam 86 repaired or replaced… for example, only one rack support, in this case the vertical support 88 shown in FIG. 7, is scanned and recognized; and
a controller configured to process acquired [three-dimensional data] in order to:
recognise a safety structure in the [three-dimensional data] (e.g. system and method for inspecting and automatically detecting damage or irregularities in a warehouse include a computer/computing unit adapted to (i.e. a controller configured to) perform functions/operations including generating and receiving scan data of an environment to construct three-dimensional point cloud data of the environment, for example, and performing image processing to recognize various storage elements from the constructed point cloud including storage aids including racks or racking elements, such as vertical supports, horizontal supports (i.e. a safety structure), as indicated above), for example);
compare the recognised safety structure in the acquired [three-dimensional data] with an other [three-dimensional data] of the same, or a corresponding, safety structure (Par. [0020-21]: computing unit is further adapted to detect individual storage elements from the constructed three-dimensional data of the environment. Storage elements in the context of this invention refer to all possible logistics elements or units that are used or required either for transport requests or for the storage of goods in a warehouse. They include… storage aids such as pallets or containers, storage locations or bins, racks or racking elements such as vertical supports, horizontal supports… The computing unit is preferably adapted to recognize different storage elements from a constructed point cloud through image processing and to determine three-dimensional data of the individual recognized storage elements based on these… the computing unit is further adapted to check a storage condition based on a comparison of the three-dimensional data of the individual detected storage elements with corresponding data from the warehouse management system… The computing unit is further preferably adapted to check a storage condition based on a comparison of the three-dimensional data of the individual detected storage elements with predefined reference data. The reference data can be, for example, specific regulatory data defined by standards; Par. [0059-63]: system 50 for inspecting a warehouse that is used in the industrial truck 1 shown in FIG. 1. The system 50 includes the laser scanner 52 and the computing unit 60, the laser scanner 52 being disposed to scan the environment in order to generate scan data, and the computing unit 60 being disposed to receive the scan data and construct three-dimensional data of the environment based on these… Based on the scan data… the computing unit 60 constructs three-dimensional data of the environment in the form of a point cloud. Through suitable image processing, the computing unit 60 can recognize various storage elements from the constructed point cloud and, based on these, determine three-dimensional data of the individual recognized storage elements. Based on a comparison of the three-dimensional data of individual detected storage elements with corresponding data from the warehouse management system and/or with predefined reference data, the computing unit 60 is able to automatically detect damage or irregularities in a warehouse; compare the recognised safety structure in the acquired [three-dimensional data] with an other [three-dimensional data] of the same, or a corresponding, safety structure (e.g. system and method for inspecting and automatically detecting damage or irregularities in a warehouse include a computer/computing unit adapted to perform functions/operations including performing image processing to recognize various storage elements from the constructed point cloud including storage aids including racks or racking elements, such as vertical supports, horizontal supports (i.e. a safety structure), for example, in order to check a storage condition based on a comparison of the three-dimensional data of the individual detected storage elements with corresponding data from the warehouse management system and/or predefined reference data, as indicated above), for example); and
based on the comparison, provide a damage status of the safety structure (Par. [0024-26]: the computing unit is further adapted to check a storage condition based on a comparison of the three-dimensional data of the individual detected storage elements with corresponding data from the warehouse management system… The computing unit is further preferably adapted to check a storage condition based on a comparison of the three-dimensional data of the individual detected storage elements with predefined reference data. The reference data can be, for example, specific regulatory data defined by standards… By comparing the three-dimensional data of the individual detected storage elements with the data stored in the warehouse management system or elsewhere, the computer unit can detect any storage irregularities and/or damage to individual storage elements; Par. [0059-63]: system 50 for inspecting a warehouse that is used in the industrial truck 1 shown in FIG. 1. The system 50 includes the laser scanner 52 and the computing unit 60, the laser scanner 52 being disposed to scan the environment in order to generate scan data, and the computing unit 60 being disposed to receive the scan data and construct three-dimensional data of the environment based on these… Based on the scan data… the computing unit 60 constructs three-dimensional data of the environment in the form of a point cloud. Through suitable image processing, the computing unit 60 can recognize various storage elements from the constructed point cloud and, based on these, determine three-dimensional data of the individual recognized storage elements. Based on a comparison of the three-dimensional data of individual detected storage elements with corresponding data from the warehouse management system and/or with predefined reference data, the computing unit 60 is able to automatically detect damage or irregularities in a warehouse; based on the comparison, provide a damage status of the safety structure (e.g. system and method for inspecting and automatically detecting damage or irregularities in a warehouse include a computer/computing unit adapted to perform functions/operations including performing image processing to recognize various storage elements from the constructed point cloud including storage aids including racks or racking elements, such as vertical supports, horizontal supports (i.e. a safety structure), for example, in order to check a storage condition based on a comparison of the three-dimensional data of the individual detected storage elements with corresponding data from the warehouse management system and/or predefined reference data, for example, and based on the comparison of the three-dimensional data of individual detected storage elements with corresponding data from the warehouse management system and/or with predefined reference data, the computer/computing automatically detects any storage irregularities and/or damage to individual storage elements, as indicated above), for example).
Bistry’s teachings above disclose the damage detection system that generates and receives scan data of an environment to construct three-dimensional data point cloud of the environment, for example, and performs image processing to recognize various storage elements from the constructed point cloud including storage aids including racks or racking elements, such as vertical supports, horizontal supports (i.e. a safety structure), as indicated above, and a camera, which is used to detect storage location(s) and/or the environment near the storage location(s) during operation of the industrial truck, as indicated above, but fails to disclose that the camera is configured to acquire images of a vicinity of the vehicle and that the image processing being performed above is being performed on the acquired images.
However, Boyle teaches wherein the camera is configured to acquire images of a vicinity of the vehicle (Par. [0020]: a method, system, server and data capture device for roadside asset tracking and maintenance monitoring, which will overcome or substantially ameliorate at least some of the deficiencies of the above prior art, or to at least provide an alternative. Specifically, in embodiments as will be described in further detail below, there is described an asset and maintenance tracking system which may be deployed on roadside maintenance vehicles, such as grass cutters, to record roadside assets, such as traffic management post, reflector posts and the like, subsequently monitor the previously recorded assets, and to monitor the maintenance conducted by the roadside maintenance vehicle; Par. [0048-49]: mobile unit may comprise at least one camera for capturing image data from the perspective of the mobile unit… The mobile unit may comprise forward and rearward facing cameras for respectively capturing forward and rearward facing image data from the perspective of the mobile unit and the map representation may be configured for displaying before and after maintenance comparison imagery utilising the image data received from the forward and rearward facing cameras; Par. [0071-74]: maintenance data may comprise the location of a maintenance machinery… maintenance data may comprise maintenance image data… The maintenance image data may comprise maintenance image data representing a rearward image from the maintenance vehicle… The maintenance image data may comprise maintenance image data representing a forward image from the maintenance vehicle; Par. [0187-192]: the machinery 300 may comprise cameras, especially the forward facing camera 320 to record information relating to an asset… camera 320 would be appropriately located for an ideal vantage for recording image data relating to various roadside assets… the data capture device 105 may be adapted to capture images from the cameras wherein the images are processed using an image processing technique; wherein the camera is configured to acquire images of a vicinity of the vehicle (e.g. method, system, server and data capture device for roadside asset tracking and maintenance monitoring include a mobile vehicle (i.e. a vehicle), such as a maintenance vehicle, and a camera for capturing image data from the perspective of the mobile unit (i.e. camera is configured to acquire images of a vicinity of the vehicle), as indicated above), for example);
process the acquired images in order to:
recognise a safety structure in the acquired images;
compare the recognised safety structure in the acquired image[s]; and
based on the comparison, provide a damage-status-signal that represents a damage status of the safety structure (Par. [0020-22]: a method, system, server and data capture device for roadside asset tracking and maintenance monitoring, which will overcome or substantially ameliorate at least some of the deficiencies of the above prior art, or to at least provide an alternative… an asset and maintenance tracking system which may be deployed on roadside maintenance vehicles, such as grass cutters, to record roadside assets, such as traffic management post, reflector posts and the like, subsequently monitor the previously recorded assets, and to monitor the maintenance conducted by the roadside maintenance vehicle… the system may perform asset identification and valuation wherein once data is entered, the system will provide a data analysis option where each asset image is analysed for type, condition and valuation at the time of entry. Any asset needing attention will be flagged and given a liability risk rating;
process the acquired images (Par. [0055-56]: mobile unit may be configured for sending the asset image data to the server and the service performs the asset type image recognition… there is provided a server for roadside asset tracking, the server may comprise a processor for processing digital data; Par. [0078]: there is provided a data capture device for roadside asset tracking, the data capture device may comprise a processor for processing digital data; Par. [0183-192]: the data capture device 105 is adapted to automate the asset data recording process… the data capture device 105 may be adapted to capture images from the cameras wherein the images are processed using an image processing technique) in order to:
recognise a safety structure in the acquired images (Par. [0001-2]: a system, server and data capture device for roadside asset tracking and maintenance monitoring… Roadside assets, such as public infrastructure assets, such as speed management posts, reflector posts, roadside barriers and the like are installed and subsequently repaired and maintained; Par. [0020-22]: a method, system, server and data capture device for roadside asset tracking and maintenance monitoring, which will overcome or substantially ameliorate at least some of the deficiencies of the above prior art, or to at least provide an alternative… an asset and maintenance tracking system which may be deployed on roadside maintenance vehicles, such as grass cutters, to record roadside assets, such as traffic management post, reflector posts and the like, subsequently monitor the previously recorded assets, and to monitor the maintenance conducted by the roadside maintenance vehicle… the system may perform asset identification and valuation wherein once data is entered, the system will provide a data analysis option where each asset image is analysed for type, condition and valuation at the time of entry. Any asset needing attention will be flagged and given a liability risk rating… utilising an asset type image recognition technique for automating the identification of the roadside assets… The asset type image recognition technique may comprise an image recognition stage comprising edge shape detection; Par. [0052-56]: method may further comprise using an asset condition estimate image recognition technique to estimate a condition of each of the roadside assets… mobile unit may be configured for sending the asset image data to the server and the service performs the asset type image recognition… there is provided a server for roadside asset tracking, the server may comprise a processor for processing digital data; Par. [0084]: processor may be further controlled by the computer program code to identify the asset in accordance with an image recognition technique; Par. [0120-166]: the asset data input device 115 may comprise an imaging device, such as a camera or the like adapted to capture images of various assets. These images of the assets maybe then manually classified by operators, or alternatively utilised for automated image recognition purposes for identifying assets… the system 100 may be adapted for the automated identification of assets using image recognition technique performed on the image data… the data capture device 105, especially by utilising image recognition technique may identify assets requiring maintenance; recognise a safety structure in the acquired images (e.g. method, system, server and data capture device for roadside asset tracking and maintenance monitoring include a mobile vehicle (i.e. a vehicle), such as a maintenance vehicle, and a camera for capturing image data from the perspective of the mobile unit, as indicated above, for example, including an image recognition technique for automating identification of roadside assets, such as public infrastructure assets, roadside barriers (i.e. a safety structure), etc., in which capture images from the camera are processed using an image processing technique, as indicated above), for example);
compare the recognised safety structure in the acquired image[s]; and
based on the comparison, provide a damage-status-signal that represents a damage status of the safety structure (Par. [0020-42]: a method, system, server and data capture device for roadside asset tracking and maintenance monitoring, which will overcome or substantially ameliorate at least some of the deficiencies of the above prior art, or to at least provide an alternative… an asset and maintenance tracking system which may be deployed on roadside maintenance vehicles, such as grass cutters, to record roadside assets, such as traffic management post, reflector posts and the like, subsequently monitor the previously recorded assets, and to monitor the maintenance conducted by the roadside maintenance vehicle… the system may perform asset identification and valuation wherein once data is entered, the system will provide a data analysis option where each asset image is analysed for type, condition and valuation at the time of entry. Any asset needing attention will be flagged… system may further allow for real-time incident reporting wherein any incidents encountered in the maintenance run will be recorded and flagged for council or a supervisor's attention immediately reducing the risk of escalation of any incidents… method may further comprise image comparison for detecting roadside asset damage or degradation to identify assets requiring maintenance; Par. [0191-193]: the system 100 is adapted for the automated identification of assets in accordance with the image data. In one particular embodiment, the system 100 may employ image recognition for recognising various assets and asset types. It should be noted that the image recognition may be performed by the data capture device 105… data capture device 105 may be adapted to capture images from the cameras wherein the images are processed using an image processing technique… the database 135 may comprise exemplary representative imagery of various roadside assets for comparison purposes during the image recognition technique; Par. [0254-255]: method may further comprise image comparison for detecting roadside asset damage or degradation to identify roadside assets requiring maintenance. For example, the method may compare asset image data against previously recorded asset image data and detect changes… For those assets that show a change in colour or shape exceeding a threshold may trigger the system to update the database 135 with a flag indicating that a particular asset requires maintenance. For example, should a roadside be damaged… the image comparison may detect the colour degradation and flag the asset for maintenance; compare the recognised safety structure in the acquired images; and
based on the comparison, provide a damage-status-signal that represents a damage status of the safety structure (e.g. method, system, server and data capture device for roadside asset tracking and maintenance monitoring include an image recognition technique for automating identification of roadside assets, such as public infrastructure assets, roadside barriers (i.e. a safety structure), etc., in which captured/recorded images from the camera are processed using an image processing technique, as indicated above, for example, including performing image comparison for detecting roadside asset damage or degradation (i.e. a damage status of the safety structure) to identify assets requiring maintenance by comparing captured asset image data against previously recorded asset image data, including exemplary representative imagery of various roadside assets for comparison purposes during the image recognition technique, to detect changes, for example, and for assets that show a change exceeding a threshold trigger, the system provides a flag (i.e. status-signal) indicating that a particular asset requires maintenance, as indicated above), for example).
Bistry and Boyle are considered to be analogous art because they pertain to image processing applications. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the apparatus and method for inspecting and automatically detecting damage or irregularities in a warehouse (as disclosed by Bistry) with wherein the camera is configured to acquire images of a vicinity of the vehicle; process the acquired images in order to: recognise a safety structure in the acquired images; compare the recognised safety structure in the acquired image[s]; and based on the comparison, provide a damage-status-signal that represents a damage status of the safety structure (as taught by Boyle, Abstract, Par. [0001-2, 20-42, 48-49, 52-56, 71-74, 84, 120-166, 187-193, 254-255) to identify roadside assets requiring maintenance by detecting roadside asset damage or degradation of the roadside assets, such as public infrastructure assets, roadside barriers (Boyle, Abstract, Par. [0001-2, 20, 42, 254]), for example.
Regarding claim 2, claim 1 is incorporated and the combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), wherein the vehicle is a forklift truck (Bistry, Par. [0053]: FIG. 1 shows a side view of an exemplary embodiment of an industrial truck 1 with a system for inspecting a warehouse according to the invention, in which exemplary embodiment the industrial truck is a high-bay stacker designed as a trilateral forklift).
Regarding claim 3, claim 1 and the combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), further comprising a plurality of cameras, each configured to acquire images of the vicinity of the vehicle (Boyle, Par. [0024-49]: may be further adapted to provide a photographic summary of each maintenance job wherein front and rear mounted cameras… mobile unit may comprise forward and rearward facing cameras for respectively capturing forward and rearward facing image data from the perspective of the mobile unit).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 4, claim 1 is incorporated and the combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), wherein the controller is configured to recognise the safety structure in the image by:
performing an object recognition operation on the image in order to recognise one or more predetermined safety structures in the image (Bistry, Par. [0021]: computing unit is preferably adapted to recognize different storage elements from a constructed point cloud through image processing and to determine three-dimensional data of the individual recognized storage elements based on these. Image processing can take place either by segmenting the point cloud and then matching the individual segmented parts with predefined patterns or rules, or it can rely on an artificial intelligence method with learning capability, where the computing unit can be continuously trained with data to adapt to new environments; Bistry, Par. [0059-64]: computing unit 60 constructs three-dimensional data of the environment in the form of a point cloud. Through suitable image processing, the computing unit 60 can recognize various storage elements from the constructed point cloud and, based on these, determine three-dimensional data of the individual recognized storage elements. Based on a comparison of the three-dimensional data of individual detected storage elements with corresponding data from the warehouse management system and/or with predefined reference data, the computing unit 60 is able to automatically detect damage or irregularities in a warehouse. FIG. 4 and FIG. 5 each illustrate an example of this… FIG. 4 shows part of a rack 84 comprising a plurality of horizontal supports 86 and vertical supports 88. It can be seen in FIG. 4 that rack damage has occurred in the form of a deformed horizontal support 86. When the industrial truck 1 with the system 50 passes the affected storage area, the computing unit 60 is able to recognize, from the constructed point cloud and corresponding image processing algorithms, that the horizontal support 86 is deformed. Based on this, the computing unit 60 can output an instruction to have the horizontal beam 86 repaired or replaced); or
applying a machine learning algorithm to the image in order to determine a classification of a safety structure as one that was visible in an image that was used as training data for training the machine learning algorithm (Bistry, Par. [0021]: computing unit is preferably adapted to recognize different storage elements from a constructed point cloud through image processing and to determine three-dimensional data of the individual recognized storage elements based on these. Image processing can take place either by segmenting the point cloud and then matching the individual segmented parts with predefined patterns or rules, or it can rely on an artificial intelligence method with learning capability, where the computing unit can be continuously trained with data to adapt to new environments).
Regarding claim 5, claim 1 is incorporated and the combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), wherein the controller is configured to compare the recognised safety structure in the image with the other image of the same, or a corresponding, safety structure by:
determining an identifier for a type of safety structure that is recognised in the image (Bistry, Par. [0020-37]: computing unit is further adapted to detect individual storage elements from the constructed three-dimensional data of the environment. Storage elements in the context of this invention refer to all possible logistics elements or units that are used or required either for transport requests or for the storage of goods in a warehouse. They include stored goods such as merchandise or load units consisting of bundled merchandise or packages, storage aids such as pallets or containers, storage locations or bins, racks or racking elements such as vertical supports, horizontal supports… The computing unit is preferably adapted to recognize different storage elements from a constructed point cloud through image processing and to determine three-dimensional data of the individual recognized storage elements based on these… computing unit is also preferably adapted to receive data from a warehouse management system... identification numbers of individual storage elements, such as racking equipment, storage locations or stored goods, can be saved in the warehouse management system so that, by combining it with other identification sensors, such as barcode or RFID readers, attached to the industrial truck, information relating to the detected storage elements can be retrieved by the warehouse management system and matched with the constructed data of the storage elements… the method further includes automatic activation of the system when a reference point is recognized. For example, the system for inspecting a narrow-aisle high-bay warehouse associated with a high-bay stacker can be activated automatically when the high-bay stacker enters a narrow aisle of the high-bay warehouse, provided that a reader attached to the high-bay stacker has detected a reference point at the entrance area of the narrow aisle, the reference point being identifiable, for example, by a barcode attached to a side of the rack… reference points, such as a barcode, RFID transponder or image marker, can also be set up in the environment that comprise or represent specific spatial information, the reference points being detected either by the laser scanner or by other sensors; Bistry, Par. [0063-64]: computing unit 60 constructs three-dimensional data of the environment in the form of a point cloud. Through suitable image processing, the computing unit 60 can recognize various storage elements from the constructed point cloud and, based on these, determine three-dimensional data of the individual recognized storage elements. Based on a comparison of the three-dimensional data of individual detected storage elements with corresponding data from the warehouse management system and/or with predefined reference data, the computing unit 60 is able to automatically detect damage or irregularities in a warehouse… FIG. 4 shows part of a rack 84 comprising a plurality of horizontal supports 86 and vertical supports 88. It can be seen in FIG. 4 that rack damage has occurred in the form of a deformed horizontal support 86. When the industrial truck 1 with the system 50 passes the affected storage area, the computing unit 60 is able to recognize, from the constructed point cloud and corresponding image processing algorithms, that the horizontal support 86 is deformed. Based on this, the computing unit 60 can output an instruction to have the horizontal beam 86 repaired or replaced; Boyle, Par. [0055-60]: mobile unit may be configured for sending the asset image data to the server and the service performs the asset type image recognition… a server for roadside asset tracking, the server may comprise a processor for processing digital data; a memory device for storing digital data including computer program code, the memory device being operably coupled to the processor; a network interface adapted for sending and receiving data across a data network, the network interface being operably coupled to the processor; and a database adapted for storing asset data representing a plurality of assets, the asset data may comprise at least unique asset identifier data and asset location data, the database being operably coupled to the processor, wherein, in use, the processor may be controlled by the computer program code to receive, via the network interface, from a remote data capture device, asset data relating to an asset, the asset data may comprise at least location data representing the location of the asset; and insert, into the database, the asset data in association with a unique asset identifier and the location data… The asset data further may comprise the asset type and wherein the processor may be further controlled by the computer program code to store, in the database, the asset type in association with the unique asset identifier… asset data further may comprise image data and wherein the processor may be further controlled by the computer program code to store, in the database, the image data in association with the unique asset identifier; Boyle, Par. [0145]: the server 140 comprises a processor 220 for processing digital data, a memory device 210 for storing digital data including computer program code, the memory device 210 been operably coupled to the processor 220. Furthermore, the server 120 comprises a network interface adapted for sending and receiving data across the data network 130, the network interface been operably coupled to the processor 220. Furthermore, the server 140 comprises a database interface for interfacing with the database 135. In this regard, the database is adapted for storing asset data representing a plurality of assets wherein the asset data comprises at least a unique asset identifier data and asset location data. The database interface is similarly operably coupled to the processor 220);
retrieving one or more images of the same type of safety structure from a memory (Bistry, Par. [0022-25]: computing unit is also preferably adapted to receive data from a warehouse management system. A warehouse management system is a software-based system that manages the control of warehouse processes. Depending on the scope, various data can be stored in a warehouse management system which can be retrieved by external systems when required… identification numbers of individual storage elements, such as racking equipment, storage locations or stored goods, can be saved in the warehouse management system so that, by combining it with other identification sensors, such as barcode or RFID readers, attached to the industrial truck, information relating to the detected storage elements can be retrieved by the warehouse management system and matched with the constructed data of the storage elements… computing unit is further preferably adapted to check a storage condition based on a comparison of the three-dimensional data of the individual detected storage elements with predefined reference data. The reference data can be, for example, specific regulatory data defined by standards, such as the permitted deflection of a rack support, the permitted distance between two adjacent pallets stored in a high-bay warehouse, the permitted dimensions of a stored item, etc. The reference data can be saved either as master data in the warehouse management system or locally in a memory of the computer unit. The data can also be saved in another external store or in a cloud; Boyle, Par. [0028-79]: receiving the roadside asset image data representing images taken of a plurality of roadside assets… mobile unit may be configured for sending the asset image data to the server and the service performs the asset type image recognition… a server for roadside asset tracking, the server may comprise a processor for processing digital data; a memory device for storing digital data including computer program code, the memory device being operably coupled to the processor; a network interface adapted for sending and receiving data across a data network, the network interface being operably coupled to the processor; and a database adapted for storing asset data representing a plurality of assets… asset data receiver may comprise an image capture device and wherein, in use, the asset data may comprise image data representing an image of the asset; Boyle, Par. [0113-154]: system 100 comprises at least one data capture device 105 operably coupled to a server computing device 140 across a data network 130. In this manner, the data capture device 105 is adapted to record information relating to various roadside assets and convey the asset data to the server 140 for recordal. Operably coupled to the server 140 is a database 135 adapted for storing asset and maintenance data… The asset data may comprises asset type and location data representing a plurality of assets and their respective locations. Over and above this, the database 135 may store additional asset data as will be described in further detail below, such as asset image data… the data capture device 105 comprises a local database adapted for storing the asset data. In this manner, upon completion of an asset tracking run or maintenance job, the data from the local database may be retrieved such as by way of USB memory stick or the like for uploading to the server 140 or other computing device… receive asset data relating to a plurality of assets. As can be appreciated, there are a number of manners in which asset data relating to a plurality of assets may be input into the data capture device. These different manners will be described in further detail below, but for brief introductory purposes, in embodiments, the asset data input device 115 may comprise an imaging device, such as a camera or the like adapted to capture images of various assets… the server 140 is adapted to receive, from the data capture device 105 image data relating to a particular asset for storage within the database 135); and
comparing the recognised safety structure in the image with the one or more images of the same type of safety structure retrieved from the memory (Bistry, Par. [0020-21]: computing unit is further adapted to detect individual storage elements from the constructed three-dimensional data of the environment. Storage elements in the context of this invention refer to all possible logistics elements or units that are used or required either for transport requests or for the storage of goods in a warehouse. They include… storage aids such as pallets or containers, storage locations or bins, racks or racking elements such as vertical supports, horizontal supports… The computing unit is preferably adapted to recognize different storage elements from a constructed point cloud through image processing and to determine three-dimensional data of the individual recognized storage elements based on these… the computing unit is further adapted to check a storage condition based on a comparison of the three-dimensional data of the individual detected storage elements with corresponding data from the warehouse management system… The computing unit is further preferably adapted to check a storage condition based on a comparison of the three-dimensional data of the individual detected storage elements with predefined reference data. The reference data can be, for example, specific regulatory data defined by standards; Bistry, Par. [0059-63]: system 50 for inspecting a warehouse that is used in the industrial truck 1 shown in FIG. 1. The system 50 includes the laser scanner 52 and the computing unit 60, the laser scanner 52 being disposed to scan the environment in order to generate scan data, and the computing unit 60 being disposed to receive the scan data and construct three-dimensional data of the environment based on these… Based on the scan data… the computing unit 60 constructs three-dimensional data of the environment in the form of a point cloud. Through suitable image processing, the computing unit 60 can recognize various storage elements from the constructed point cloud and, based on these, determine three-dimensional data of the individual recognized storage elements. Based on a comparison of the three-dimensional data of individual detected storage elements with corresponding data from the warehouse management system and/or with predefined reference data, the computing unit 60 is able to automatically detect damage or irregularities in a warehouse; Boyle, Par. [0020-42]: a method, system, server and data capture device for roadside asset tracking and maintenance monitoring, which will overcome or substantially ameliorate at least some of the deficiencies of the above prior art, or to at least provide an alternative… an asset and maintenance tracking system which may be deployed on roadside maintenance vehicles, such as grass cutters, to record roadside assets, such as traffic management post, reflector posts and the like, subsequently monitor the previously recorded assets, and to monitor the maintenance conducted by the roadside maintenance vehicle… the system may perform asset identification and valuation wherein once data is entered, the system will provide a data analysis option where each asset image is analysed for type, condition and valuation at the time of entry. Any asset needing attention will be flagged and given a liability risk rating… system may further allow for real-time incident reporting wherein any incidents encountered in the maintenance run will be recorded and flagged for council or a supervisor's attention immediately reducing the risk of escalation of any incidents… method may further comprise image comparison for detecting roadside asset damage or degradation to identify assets requiring maintenance; Par. [0191-193]: the system 100 is adapted for the automated identification of assets in accordance with the image data. In one particular embodiment, the system 100 may employ image recognition for recognising various assets and asset types. It should be noted that the image recognition may be performed by the data capture device 105… data capture device 105 may be adapted to capture images from the cameras wherein the images are processed using an image processing technique… the database 135 may comprise exemplary representative imagery of various roadside assets for comparison purposes during the image recognition technique; Par. [0254-255]: method may further comprise image comparison for detecting roadside asset damage or degradation to identify roadside assets requiring maintenance. For example, the method may compare asset image data against previously recorded asset image data and detect changes… For those assets that show a change in colour or shape exceeding a threshold may trigger the system to update the database 135 with a flag indicating that a particular asset requires maintenance. For example, should a roadside be damaged… the image comparison may detect the colour degradation and flag the asset for maintenance).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 6, claim 5 is incorporated and the combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), wherein the one or more images of the same type of safety structure retrieved from the memory comprise images of the safety structure in an undamaged state and/or one or more damaged states (Bistry, Par. [0024-26]: the computing unit is further adapted to check a storage condition based on a comparison of the three-dimensional data of the individual detected storage elements with corresponding data from the warehouse management system… The computing unit is further preferably adapted to check a storage condition based on a comparison of the three-dimensional data of the individual detected storage elements with predefined reference data. The reference data can be, for example, specific regulatory data defined by standards… By comparing the three-dimensional data of the individual detected storage elements with the data stored in the warehouse management system or elsewhere, the computer unit can detect any storage irregularities and/or damage to individual storage elements; Bistry, Par. [0059-63]: system 50 for inspecting a warehouse that is used in the industrial truck 1 shown in FIG. 1. The system 50 includes the laser scanner 52 and the computing unit 60, the laser scanner 52 being disposed to scan the environment in order to generate scan data, and the computing unit 60 being disposed to receive the scan data and construct three-dimensional data of the environment based on these… Based on the scan data… the computing unit 60 constructs three-dimensional data of the environment in the form of a point cloud. Through suitable image processing, the computing unit 60 can recognize various storage elements from the constructed point cloud and, based on these, determine three-dimensional data of the individual recognized storage elements. Based on a comparison of the three-dimensional data of individual detected storage elements with corresponding data from the warehouse management system and/or with predefined reference data, the computing unit 60 is able to automatically detect damage or irregularities in a warehouse; Boyle, Par. [0020-42]: a method, system, server and data capture device for roadside asset tracking and maintenance monitoring, which will overcome or substantially ameliorate at least some of the deficiencies of the above prior art, or to at least provide an alternative… an asset and maintenance tracking system which may be deployed on roadside maintenance vehicles, such as grass cutters, to record roadside assets, such as traffic management post, reflector posts and the like, subsequently monitor the previously recorded assets, and to monitor the maintenance conducted by the roadside maintenance vehicle… the system may perform asset identification and valuation wherein once data is entered, the system will provide a data analysis option where each asset image is analysed for type, condition and valuation at the time of entry. Any asset needing attention will be flagged and given a liability risk rating… system may further allow for real-time incident reporting wherein any incidents encountered in the maintenance run will be recorded and flagged for council or a supervisor's attention immediately reducing the risk of escalation of any incidents… method may further comprise image comparison for detecting roadside asset damage or degradation to identify assets requiring maintenance; Boyle, Par. [0254-255]: method may further comprise image comparison for detecting roadside asset damage or degradation to identify roadside assets requiring maintenance. For example, the method may compare asset image data against previously recorded asset image data and detect changes… For those assets that show a change in colour or shape exceeding a threshold may trigger the system to update the database 135 with a flag indicating that a particular asset requires maintenance. For example, should a roadside be damaged… the image comparison may detect the colour degradation and flag the asset for maintenance).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 7, claim 5 is incorporated and the combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), wherein:
comparing the recognised safety structure in the image with the one or more images of the same type of safety structure retrieved from the memory comprises determining a degree of similarity between the images (Bistry, Par. [0021-26]: computing unit is preferably adapted to recognize different storage elements from a constructed point cloud through image processing and to determine three-dimensional data of the individual recognized storage elements based on these. Image processing can take place either by segmenting the point cloud and then matching the individual segmented parts with predefined patterns or rules, or it can rely on an artificial intelligence method with learning capability, where the computing unit can be continuously trained with data to adapt to new environments… the computing unit is further adapted to check a storage condition based on a comparison of the three-dimensional data of the individual detected storage elements with corresponding data from the warehouse management system… computing unit is further preferably adapted to check a storage condition based on a comparison of the three-dimensional data of the individual detected storage elements with predefined reference data. The reference data can be, for example, specific regulatory data defined by standards, such as the permitted deflection of a rack support, the permitted distance between two adjacent pallets stored in a high-bay warehouse, the permitted dimensions of a stored item, etc. The reference data can be saved either as master data in the warehouse management system or locally in a memory of the computer unit. The data can also be saved in another external store or in a cloud… By comparing the three-dimensional data of the individual detected storage elements with the data stored in the warehouse management system or elsewhere, the computer unit can detect any storage irregularities and/or damage to individual storage elements; Bistry, Par. [0063]: computing unit 60 constructs three-dimensional data of the environment in the form of a point cloud. Through suitable image processing, the computing unit 60 can recognize various storage elements from the constructed point cloud and, based on these, determine three-dimensional data of the individual recognized storage elements. Based on a comparison of the three-dimensional data of individual detected storage elements with corresponding data from the warehouse management system and/or with predefined reference data, the computing unit 60 is able to automatically detect damage or irregularities in a warehouse; Boyle, Par. [0254-255]: method may further comprise image comparison for detecting roadside asset damage or degradation to identify roadside assets requiring maintenance. For example, the method may compare asset image data against previously recorded asset image data and detect changes… For those assets that show a change in colour or shape exceeding a threshold may trigger the system to update the database 135 with a flag indicating that a particular asset requires maintenance. For example, should a roadside be damaged… the image comparison may detect the colour degradation and flag the asset for maintenance); and
the controller is configured to provide the damage-status-signal, that represents the damage status of the safety structure, based on the determined degree of similarity (Bistry, Par. [0059-63]: system 50 for inspecting a warehouse that is used in the industrial truck 1 shown in FIG. 1. The system 50 includes the laser scanner 52 and the computing unit 60, the laser scanner 52 being disposed to scan the environment in order to generate scan data, and the computing unit 60 being disposed to receive the scan data and construct three-dimensional data of the environment based on these… Based on the scan data… the computing unit 60 constructs three-dimensional data of the environment in the form of a point cloud. Through suitable image processing, the computing unit 60 can recognize various storage elements from the constructed point cloud and, based on these, determine three-dimensional data of the individual recognized storage elements. Based on a comparison of the three-dimensional data of individual detected storage elements with corresponding data from the warehouse management system and/or with predefined reference data, the computing unit 60 is able to automatically detect damage or irregularities in a warehouse; Boyle, Par. [0191-193]: the system 100 is adapted for the automated identification of assets in accordance with the image data. In one particular embodiment, the system 100 may employ image recognition for recognising various assets and asset types. It should be noted that the image recognition may be performed by the data capture device 105… data capture device 105 may be adapted to capture images from the cameras wherein the images are processed using an image processing technique… the database 135 may comprise exemplary representative imagery of various roadside assets for comparison purposes during the image recognition technique; Boyle, Par. [0254-255]: method may further comprise image comparison for detecting roadside asset damage or degradation to identify roadside assets requiring maintenance. For example, the method may compare asset image data against previously recorded asset image data and detect changes… For those assets that show a change in colour or shape exceeding a threshold may trigger the system to update the database 135 with a flag indicating that a particular asset requires maintenance. For example, should a roadside be damaged… the image comparison may detect the colour degradation and flag the asset for maintenance).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 8, claim 5 is incorporated and the combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), wherein the controller is configured to determine the identifier for the type of safety structure that is recognised in the image by reading a machine-readable code that is visible in the acquired image (Bistry, Par. [0020-37]: computing unit is further adapted to detect individual storage elements from the constructed three-dimensional data of the environment. Storage elements in the context of this invention refer to all possible logistics elements or units that are used or required either for transport requests or for the storage of goods in a warehouse. They include stored goods such as merchandise or load units consisting of bundled merchandise or packages, storage aids such as pallets or containers, storage locations or bins, racks or racking elements such as vertical supports, horizontal supports… The computing unit is preferably adapted to recognize different storage elements from a constructed point cloud through image processing and to determine three-dimensional data of the individual recognized storage elements based on these… computing unit is also preferably adapted to receive data from a warehouse management system... identification numbers of individual storage elements, such as racking equipment, storage locations or stored goods, can be saved in the warehouse management system so that, by combining it with other identification sensors, such as barcode or RFID readers, attached to the industrial truck, information relating to the detected storage elements can be retrieved by the warehouse management system and matched with the constructed data of the storage elements… the method further includes automatic activation of the system when a reference point is recognized. For example, the system for inspecting a narrow-aisle high-bay warehouse associated with a high-bay stacker can be activated automatically when the high-bay stacker enters a narrow aisle of the high-bay warehouse, provided that a reader attached to the high-bay stacker has detected a reference point at the entrance area of the narrow aisle, the reference point being identifiable, for example, by a barcode attached to a side of the rack… reference points, such as a barcode, RFID transponder or image marker, can also be set up in the environment that comprise or represent specific spatial information, the reference points being detected either by the laser scanner or by other sensors; Bistry, Par. [0063-64]: computing unit 60 constructs three-dimensional data of the environment in the form of a point cloud. Through suitable image processing, the computing unit 60 can recognize various storage elements from the constructed point cloud and, based on these, determine three-dimensional data of the individual recognized storage elements. Based on a comparison of the three-dimensional data of individual detected storage elements with corresponding data from the warehouse management system and/or with predefined reference data, the computing unit 60 is able to automatically detect damage or irregularities in a warehouse… FIG. 4 shows part of a rack 84 comprising a plurality of horizontal supports 86 and vertical supports 88. It can be seen in FIG. 4 that rack damage has occurred in the form of a deformed horizontal support 86. When the industrial truck 1 with the system 50 passes the affected storage area, the computing unit 60 is able to recognize, from the constructed point cloud and corresponding image processing algorithms, that the horizontal support 86 is deformed. Based on this, the computing unit 60 can output an instruction to have the horizontal beam 86 repaired or replaced; Boyle, Par. [0055-94]: mobile unit may be configured for sending the asset image data to the server and the service performs the asset type image recognition… a server for roadside asset tracking, the server may comprise a processor for processing digital data; a memory device for storing digital data including computer program code, the memory device being operably coupled to the processor; a network interface adapted for sending and receiving data across a data network, the network interface being operably coupled to the processor; and a database adapted for storing asset data representing a plurality of assets, the asset data may comprise at least unique asset identifier data and asset location data, the database being operably coupled to the processor, wherein, in use, the processor may be controlled by the computer program code to receive, via the network interface, from a remote data capture device, asset data relating to an asset, the asset data may comprise at least location data representing the location of the asset; and insert, into the database, the asset data in association with a unique asset identifier and the location data… The asset data further may comprise the asset type and wherein the processor may be further controlled by the computer program code to store, in the database, the asset type in association with the unique asset identifier… asset data further may comprise image data and wherein the processor may be further controlled by the computer program code to store, in the database, the image data in association with the unique asset identifier… asset data input device may comprise a scanning device adapted to scan the asset data from a data bearing computer readable medium… The data bearing computer readable medium may comprise a barcode… The barcode may comprise at least one of a 2-D and 3-D barcode… the asset data input device 115 may comprise an imaging device, such as a camera or the like adapted to capture images of various assets… the asset data input device 150 may comprise a scanning device adapted to scan a computer readable media attached to the asset, such as a barcode).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 12, claim 1 is incorporated and the combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), further comprising an alert signal generator that is configured to selectively provide an alert based on the damage-status-signal (Boyle, Par. [0020-42]: a method, system, server and data capture device for roadside asset tracking and maintenance monitoring, which will overcome or substantially ameliorate at least some of the deficiencies of the above prior art, or to at least provide an alternative… an asset and maintenance tracking system which may be deployed on roadside maintenance vehicles, such as grass cutters, to record roadside assets, such as traffic management post, reflector posts and the like, subsequently monitor the previously recorded assets, and to monitor the maintenance conducted by the roadside maintenance vehicle… the system may perform asset identification and valuation wherein once data is entered, the system will provide a data analysis option where each asset image is analysed for type, condition and valuation at the time of entry. Any asset needing attention will be flagged and given a liability risk rating… system may further allow for real-time incident reporting wherein any incidents encountered in the maintenance run will be recorded and flagged for council or a supervisor's attention immediately reducing the risk of escalation of any incidents… method may further comprise image comparison for detecting roadside asset damage or degradation to identify assets requiring maintenance; Boyle, Par. [0254-255]: method may further comprise image comparison for detecting roadside asset damage or degradation to identify roadside assets requiring maintenance. For example, the method may compare asset image data against previously recorded asset image data and detect changes… For those assets that show a change in colour or shape exceeding a threshold may trigger the system to update the database 135 with a flag indicating that a particular asset requires maintenance. For example, should a roadside be damaged… the image comparison may detect the colour degradation and flag the asset for maintenance).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 13, claim 12 is incorporated and the combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), wherein the controller is configured to:
determine a location of the safety structure that is recognised in the image (Bistry, Par. [0020-71]: computing unit is further adapted to detect individual storage elements from the constructed three-dimensional data of the environment. Storage elements in the context of this invention refer to all possible logistics elements or units that are used or required either for transport requests or for the storage of goods in a warehouse. They include… storage aids such as pallets or containers, storage locations or bins, racks or racking elements such as vertical supports, horizontal supports… The computing unit is preferably adapted to recognize different storage elements from a constructed point cloud through image processing and to determine three-dimensional data of the individual recognized storage elements based on these… computing unit is further preferably adapted to check a storage condition… computer unit can detect any storage irregularities and/or damage to individual storage elements… the vertical support 88 shown in FIG. 7, is scanned and recognized when the target storage location is reached (see scanning range 200). Based on this, the control system of the industrial truck, together with data concerning the dimensions of the target storage bin saved locally or externally (such as in the warehouse management system), can calculate the target destination position Ptarget. Although the strategy shown in FIG. 7 has the disadvantage that part of the target storage location cannot be scanned, this disadvantage can be compensated, for example, by using additional sensors, such as a camera, which also detects the storage location and/or the environment near the storage location during the deposit operation; Boyle, Par. [0028-58]: receiving the roadside asset image data representing images taken of a plurality of roadside assets; receiving location data from the GPS receiver representing the respective locations the roadside assets; utilising an asset type image recognition technique for automating the identification of the roadside assets; comparing roadside asset data of the asset tracking register database to: record newly identified roadside assets and their respective locations in the asset tracking register database; and identify missing roadside assets… method may further comprise generating a map representation representing the roadside section and the locations of the identified roadside assets along the roadside section…The mobile unit may comprise at least one camera for capturing image data from the perspective of the mobile unit and the map representation may be configured for selectively displaying image data from the at least one camera at a plurality of locations along the roadside section… the asset data may comprise at least unique asset identifier data and asset location data, the database being operably coupled to the processor, wherein, in use, the processor may be controlled by the computer program code to receive, via the network interface, from a remote data capture device, asset data relating to an asset, the asset data may comprise at least location data representing the location of the asset; Boyle, Par. [0145-146]: server 140 comprises a database interface for interfacing with the database 135. In this regard, the database is adapted for storing asset data representing a plurality of assets wherein the asset data comprises at least a unique asset identifier data and asset location data… the server 140 is adapted to receive, from the data capture device 105 asset data relating to a roadside asset, the asset data comprising at least the location of the roadside asset. In this manner, the server 140 is adapted to store, in the database 135, asset type and location data representing the asset and the location thereof); and
provide an alert that is based on the determined location of the safety structure (Boyle, Par. [0022-58]: system may perform asset identification and valuation wherein once data is entered, the system will provide a data analysis option where each asset image is analysed for type, condition and valuation at the time of entry. Any asset needing attention will be flagged…receiving the roadside asset image data representing images taken of a plurality of roadside assets; receiving location data from the GPS receiver representing the respective locations the roadside assets; utilising an asset type image recognition technique for automating the identification of the roadside assets; comparing roadside asset data of the asset tracking register database to: record newly identified roadside assets and their respective locations in the asset tracking register database; and identify missing roadside assets… method may further comprise generating a map representation representing the roadside section and the locations of the identified roadside assets along the roadside section…The mobile unit may comprise at least one camera for capturing image data from the perspective of the mobile unit and the map representation may be configured for selectively displaying image data from the at least one camera at a plurality of locations along the roadside section… the asset data may comprise at least unique asset identifier data and asset location data, the database being operably coupled to the processor, wherein, in use, the processor may be controlled by the computer program code to receive, via the network interface, from a remote data capture device, asset data relating to an asset, the asset data may comprise at least location data representing the location of the asset; Boyle, Par. [0236-255]: , along the roadside section the camera may capture periodic images, such as in front of the mobile unit 300, or that which is engaged by the grass cutting device 310 and wherein the method comprises employing the asset type image recognition to identify assets from those images… the data capture device 105 may transmit the image data and the location data from the location sensor 110 to the server 140 for processing by the server 140… method may further comprise image comparison for detecting roadside asset damage or degradation to identify roadside assets requiring maintenance. For example, the method may compare asset image data against previously recorded asset image data and detect changes… For those assets that show a change in colour or shape exceeding a threshold may trigger the system to update the database 135 with a flag indicating that a particular asset requires maintenance. For example, should a roadside be damaged… the image comparison may detect the colour degradation and flag the asset for maintenance).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 14, claim 1 is incorporated and the combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), wherein the controller is configured to trigger the camera to acquire the image:
periodically (Boyle, Par. [0086]: processor may be further controlled by the computer program code to capture the image data upon receipt of a trigger signal; Boyle, Par. [0175]: maintenance machinery 300 may comprise a forward facing camera 320 and a rearward facing camera 315. In this manner, at periodic intervals, the server 140 may be adapted to receive, from the data capture device 105 image data from these cameras; Par. [0190-236]: timing for the taking of the second rearward facing image may be timed in accordance with a predetermined time interval… along the roadside section the camera may capture periodic images, such as in front of the mobile unit 300);
in response to the vehicle having a predetermined location (Par. [0022-86]: system may perform asset identification and valuation wherein once data is entered, the system will provide a data analysis option where each asset image is analysed for type, condition and valuation at the time of entry. Any asset needing attention will be flagged…receiving the roadside asset image data representing images taken of a plurality of roadside assets; receiving location data from the GPS receiver representing the respective locations the roadside assets; utilising an asset type image recognition technique for automating the identification of the roadside assets; comparing roadside asset data of the asset tracking register database to: record newly identified roadside assets and their respective locations in the asset tracking register database; and identify missing roadside assets… method may further comprise generating a map representation representing the roadside section and the locations of the identified roadside assets along the roadside section…The mobile unit may comprise at least one camera for capturing image data from the perspective of the mobile unit and the map representation may be configured for selectively displaying image data from the at least one camera at a plurality of locations along the roadside section… the asset data may comprise at least unique asset identifier data and asset location data, the database being operably coupled to the processor, wherein, in use, the processor may be controlled by the computer program code to receive, via the network interface, from a remote data capture device, asset data relating to an asset, the asset data may comprise at least location data representing the location of the asset… processor may be further controlled by the computer program code to capture the image data upon receipt of a trigger signal; Boyle, Par. [0190]: the data capture device 105 captures the second rearward facing image in accordance with the speed of the maintenance vehicle 305, or the location of the maintenance vehicle 310 wherein the data capture device 105 captures the second rearward facing image in accordance with the location of the vehicle);
in response to receiving a proximity signal from a safety structure (Boyle, Par. [0039-86]: mobile unit may further comprise an asset proximity detector for detecting the proximity of roadside assets and the asset data capture device may be triggered by the asset proximity detector… processor may be further controlled by the computer program code to capture the image data upon receipt of a trigger signal; Boyle, Par. [0249]: mobile unit asset data capture device 115 comprises a detector in the form of an asset proximity detector to detect the proximity of the roadside asset. The detector may trigger the asset data capture device 115 to capture an image of the roadside asset; ); and/or
in response to an on-demand command provided by a user (Par. [0120]: the asset data input device 115 may comprise an imaging device, such as a camera or the like adapted to capture images of various assets. These images of the assets maybe then manually classified by operators, or alternatively utilised for automated image recognition purposes for identifying assets; Boyle, Par. [0217-218]: portion comprises an image of the asset for recordal. As mentioned above, the image of the asset may be taken manually by the operator, or in an automated manner such as wherein the data capture device 105 utilises image recognition technique, proximity detection or the like…Once the image of the asset has been taken by the operator, or the asset identified by the data capture device 105, the interface 600 may allow the operator to confirm the asset including in inputting additional information relating to the asset).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 16, Bistry discloses a controller for a damage detection system (Par. [0020-26]: computing unit is further adapted to detect individual storage elements from the constructed three-dimensional data of the environment… the computer unit can detect any storage irregularities and/or damage to individual storage elements; Par. [0059-64]: computing unit 60 constructs three-dimensional data of the environment in the form of a point cloud. Through suitable image processing, the computing unit 60 can recognize various storage elements from the constructed point cloud and, based on these, determine three-dimensional data of the individual recognized storage elements. Based on a comparison of the three-dimensional data of individual detected storage elements with corresponding data from the warehouse management system and/or with predefined reference data, the computing unit 60 is able to automatically detect damage or irregularities in a warehouse. FIG. 4 and FIG. 5 each illustrate an example of this… FIG. 4 shows part of a rack 84 comprising a plurality of horizontal supports 86 and vertical supports 88. It can be seen in FIG. 4 that rack damage has occurred in the form of a deformed horizontal support 86. When the industrial truck 1 with the system 50 passes the affected storage area, the computing unit 60 is able to recognize, from the constructed point cloud and corresponding image processing algorithms, that the horizontal support 86 is deformed. Based on this, the computing unit 60 can output an instruction to have the horizontal beam 86 repaired or replaced). The steps further recited in claim 16 are rejected as applied to the apparatus claim 1 above.
Regarding claim 17, is a corresponding method claim rejected as applied to the apparatus claim 1 above.
Claims 9 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Bistry, in view of Boyle, as applied to claim 1, in further view of Shalev-Shwartz et al. (US PG Publication No. US 2019/0291728 A1), hereafter referred to as Shalev-Shwartz.
Regarding claim 9, claim 1 is incorporated and the combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), but fails to teach the following as further recited in claim 9.
However, Shalev-Shwartz teaches wherein the controller is configured to:
combine a plurality of images of the same safety structure into a combined-image (Par. [0154-198]: a first processing device may receive images from both the main camera and the narrow field of view camera, and perform vision processing of the narrow FOV camera to, for example, detect other vehicles, pedestrians, lane marks, traffic signs, traffic lights, and other road objects. Further, the first processing device may calculate a disparity of pixels between the images from the main camera and the narrow camera and create a 3D reconstruction of the environment of vehicle 200. The first processing device may then combine the 3D reconstruction with 3D map data or with 3D information calculated based on information from another camera… processing unit 110 may select information derived from two of the first, second, and third plurality of images by determining the extent to which information derived from one image source is consistent with information derived from other image sources. For example, processing unit 110 may combine the processed information derived from each of image capture devices 122, 124, and 126 (whether by monocular analysis, stereo analysis, or any combination of the two) and determine visual indicators (e.g., lane markings, a detected vehicle and its location and/or path, a detected traffic light, etc.) that are consistent across the images captured from each of image capture devices); and
compare the combined-image with the other image of the same, or a corresponding, safety structure (Par. [0154-198]: a first processing device may receive images from both the main camera and the narrow field of view camera, and perform vision processing of the narrow FOV camera to, for example, detect other vehicles, pedestrians, lane marks, traffic signs, traffic lights, and other road objects. Further, the first processing device may calculate a disparity of pixels between the images from the main camera and the narrow camera and create a 3D reconstruction of the environment of vehicle 200. The first processing device may then combine the 3D reconstruction with 3D map data or with 3D information calculated based on information from another camera… Processing unit 110 may execute monocular image analysis module 402 to implement process 500B. At step 540, processing unit 110 may determine a set of candidate objects representing possible vehicles and/or pedestrians. For example, processing unit 110 may scan one or more images, compare the images to one or more predetermined patterns, and identify within each image possible locations that may contain objects of interest (e.g., vehicles, pedestrians, or portions thereof)… processing unit 110 may select information derived from two of the first, second, and third plurality of images by determining the extent to which information derived from one image source is consistent with information derived from other image sources. For example, processing unit 110 may combine the processed information derived from each of image capture devices 122, 124, and 126 (whether by monocular analysis, stereo analysis, or any combination of the two) and determine visual indicators (e.g., lane markings, a detected vehicle and its location and/or path, a detected traffic light, etc.) that are consistent across the images captured from each of image capture devices).
Bistry, Boyle, and Shalev-Shwartz are considered to be analogous art because they pertain to image processing applications. Therefore, the combined teachings of Bistry, Boyle, and Shalev-Shwartz, as a whole, would have rendered obvious the invention recited in claim 9 with a reasonable expectation of success in order to modify the apparatus and method for inspecting and automatically detecting damage or irregularities in a warehouse (as disclosed by Bistry) with wherein the controller is configured to: combine a plurality of images of the same safety structure into a combined-image and compare the combined-image with the other image of the same, or a corresponding, safety structure (as taught by Shalev-Shwartz, Abstract, Par. [0154-198]) to detect a target vehicle in the environment of a host vehicle, to detect other vehicles, pedestrians, lane marks, traffic signs, traffic lights, and other road objects, and to detect various road hazards (Shalev-Shwartz, Abstract, Par. [0002-10, 154-198]).
Regarding claim 10, claim 9 and the combination of Bistry, Boyle, and Shalev-Shwartz, as a whole, teaches the system (Bistry, Par. [0001]), wherein the controller is configured to:
combine the plurality of images of the same safety structure into a 3-dimensional combined-image (Shalev-Shwartz, Par. [0154-198]: a first processing device may receive images from both the main camera and the narrow field of view camera, and perform vision processing of the narrow FOV camera to, for example, detect other vehicles, pedestrians, lane marks, traffic signs, traffic lights, and other road objects. Further, the first processing device may calculate a disparity of pixels between the images from the main camera and the narrow camera and create a 3D reconstruction of the environment of vehicle 200. The first processing device may then combine the 3D reconstruction with 3D map data or with 3D information calculated based on information from another camera… processing unit 110 may select information derived from two of the first, second, and third plurality of images by determining the extent to which information derived from one image source is consistent with information derived from other image sources. For example, processing unit 110 may combine the processed information derived from each of image capture devices 122, 124, and 126 (whether by monocular analysis, stereo analysis, or any combination of the two) and determine visual indicators (e.g., lane markings, a detected vehicle and its location and/or path, a detected traffic light, etc.) that are consistent across the images captured from each of image capture devices).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 9.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Bistry, in view of Boyle, as applied to claim 1, in further view of Kumar et al. (US PG Publication No. US 2019/0176862 A1), hereafter referred to as Kumar.
Regarding claim 15, claim 1 and the combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), wherein the controller is configured to:
compare a colour of the recognised safety structure in the acquired image with a colour of the safety structure in the other image of the same, or a corresponding, safety structure (Boyle, Par. [0254-268]: method may further comprise image comparison for detecting roadside asset damage or degradation to identify roadside assets requiring maintenance. For example, the method may compare asset image data against previously recorded asset image data and detect changes… For those assets that show a change in colour or shape exceeding a threshold may trigger the system to update the database 135 with a flag… the method may utilise an asset condition estimate image recognition technique for estimating a condition of each of the roadside assets. For example, referring to the exemplary interface 700 as can be seen, asset A4262, the condition of the asset has been deemed to be excellent. The asset condition estimate image recognition technique to take into account various factors in determining the condition, such as colour, shape and the like); and
based on the comparison, provide a damage-status-signal (Boyle, Par. [0254-255]: method may further comprise image comparison for detecting roadside asset damage or degradation to identify roadside assets requiring maintenance. For example, the method may compare asset image data against previously recorded asset image data and detect changes… For those assets that show a change in colour or shape exceeding a threshold may trigger the system to update the database 135 with a flag indicating that a particular asset requires maintenance. For example, should a roadside be damaged… the image comparison may detect the colour degradation and flag the asset for maintenance).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
The combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), as indicated above, bot fails to teach the following as further recited in claim 15.
However, Kumar teaches signal that represents whether or not the safety structure is rusted (Par. [0345]: controller 2902 can direct the output device 2912 to display the current state of a route… depiction may be of the inspected object, whether a point on a road, a section of track, or a bridge or other infrastructure. That depiction may be a two-dimensional image, or may be three-dimensional rendered representation based in part on images taken from the mobile platform or from another data acquisition platform. Areas of interest (such as failure points, wear, fatigue, rust, cracks, and the like) may be highlighted on the depiction of the object (e.g., the three-dimensional rendering). These highlights may be coded by color or another symbol. For example, cracks may be flashing red while rust spots have arrows pointing to them).
Bistry, Boyle, and Kumar are considered to be analogous art because they pertain to image processing applications. Therefore, the combined teachings of Bistry, Boyle, and Kumar, as a whole, would have rendered obvious the invention recited in claim 15 with a reasonable expectation of success in order to modify the apparatus and method for inspecting and automatically detecting damage or irregularities in a warehouse (as disclosed by Bistry) with signal that represents whether or not the safety structure is rusted (as taught by Kumar, Abstract, Par. [0345]) to detect hazards, to detect obstacles on the route, problems with the route, or other hazards, to detect damage to the route, to detect hazards ahead of the vehicle, such as obstacles in front of the vehicle along the route, detect damaged segments of the route, and to highlight a depiction of areas of interest of an object, such as failure points, wear, fatigue, rust, cracks, etc. (Kumar, Abstract, Par. [0015-18, 140, 145, 184, 188-189, 345]), for example).
Allowable Subject Matter
Claim 11 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The following is a statement of reasons for the indication of allowable subject matter: The prior art of record fails to anticipate or render obvious the following limitations as claimed:
In view of claim 1 in its entirety, the further limitations of “… wherein the controller is configured to:
determine the [a] distance to the safety structure that is recognised in the image;
in response to the acquisition of subsequent images by the camera:
recognise the safety structure in the subsequent image;
determine a distance to the safety structure in the subsequent image;
calculate if the distance to the safety structure is increasing or decreasing, and:
if the determined distance is reducing, then identify the subsequent image as an approaching-image; and
if the determined distance is increasing, then identify the subsequent image as a retreating-image;
compare the recognised safety structure in a retreating-image with the recognised safety structure in an approaching-image; and
based on the comparison, provide the damage-status-signal that represents the damage-status of the safety structure” as recited in claim 11.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GUILLERMO M RIVERA-MARTINEZ whose telephone number is (571) 272-4979. The examiner can normally be reached on 9 am to 5 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached on 571-270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GUILLERMO M RIVERA-MARTINEZ/ Primary Examiner, Art Unit 2677
Prosecution Insights