DETAILED ACTION
Applicant's amendment of June 26, 2026 overcomes the following:
Specification objections
Objections to claims 1 and 11
Applicant has amended claims 1, 6-8, 11, and 16-17. Claim 5 has been canceled. Claims 1-4 and 6-17 are pending.
Response to Arguments
Regarding Applicant’s arguments filed on June 26, 2026 with respect to canceled claim 5, Applicant asserts that the examiner “has only raised an obviousness objection against claim 5” (Remarks, Pg. 11).
Examiner respectfully disagrees.
There was no “obviousness objection” raised against canceled claim 5 in the Non-Final Office action (OA) of March 27, 2026, as Applicant asserts above. Canceled claim 5 was instead rejected under 35 USC 103, as previously set forth in the Non-Final Office action (OA) of March 27, 2026, Pg. 22-31.
Applicant’s arguments with respect to amended independent claims 1 and 16-17 have been considered but are moot in view of revised ground(s) of rejection. Amended independent claims 1 and 16-17 now recite feature limitations of previously rejected claim 5, which is now canceled. The amended resulted in changes to the scope and contents of independent claims 1 and 16-17; therefore, the grounds of rejection are revised accordingly. It is noted that previously applied prior arts remain in effect.
Claim Objections
Upon further review of the claims, claims 8 and 15-17 are objected to because of the following informalities:
Claim 8 recites the limitation “the acquired image” in line 3 of the claim. However, the claimed “the acquired image” in line 3 of the claim should recite “the acquired images” instead to clarify that the claim is referring to the previously recited “acquired images”.
Therefore, for examination purposes, the claimed “the acquired image” recited in line 3 of the claim will be interpreted as “the acquired images”.
Claim 15 recites the limitation “the acquired image” in line 2 of the claim. However, the claimed “the acquired image” in line 2 of the claim should recite “the acquired images” instead to clarify that the claim is referring to the previously recited “acquired images”.
Therefore, for examination purposes, the claimed “the acquired image” recited in line 2 of the claim will be interpreted as “the acquired images”.
Claim 16 recites the limitation “recognise a safety structure in the acquired images … compare the recognised safety structure in the acquired image” in lines 4-5 of the claim. However, the claimed “the acquired image” in line 5 of the claim should recite “the acquired images” instead to clarify that the claim is referring to the previously recited “acquired images”.
Therefore, for examination purposes, the claimed “recognise a safety structure in the acquired images … compare the recognised safety structure in the acquired image” recited in lines 4-5 of the claim will be interpreted as “recognise a safety structure in the acquired images … compare the recognised safety structure in the acquired images”.
Claim 17 recites the limitation “recognising a safety structure in the acquired images … comparing the recognised safety structure in the acquired image” in lines 4-5 of the claim. However, the claimed “the acquired image” in line 5 of the claim should recite “the acquired images” instead to clarify that the claim is referring to the previously recited “acquired images”.
Therefore, for examination purposes, the claimed “recognise a safety structure in the acquired images … compare the recognised safety structure in the acquired image” recited in lines 4-5 of the claim will be interpreted as “recognise a safety structure in the acquired images … compare the recognised safety structure in the acquired images”.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6-8, 12-14 and 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Bistry et al. (US PG Publication No. 2021/0276842 A1), hereafter referred to as Bistry, Applicant cited prior art, in view of Boyle et al. (US PG Publication No. US 2018/0247137 A1), hereafter referred to as Boyle.
Regarding claim 1, Bistry discloses a damage detection system (Par. [0001]: system for inspecting a warehouse, an industrial truck comprising such a system and a method for inspecting a warehouse using such a system; Par. [0020-42]: computing unit is further adapted to detect individual storage elements from the constructed three-dimensional data of the environment. Storage elements in the context of this invention refer to all possible logistics elements or units that are used or required either for transport requests or for the storage of goods in a warehouse. They include… storage aids such as pallets or containers, storage locations or bins, racks or racking elements such as vertical supports, horizontal supports… computer unit can detect any storage irregularities and/or damage to individual storage elements… irregularities or damage to a high-bay warehouse are detected at an early stage, thus enhancing occupational safety and logistical efficiency… method preferably further includes detection of damage and irregularities of the storage condition; Par. [0059-64]: computing unit 60 is able to automatically detect damage or irregularities in a warehouse. FIG. 4 and FIG. 5 each illustrate an example of this… FIG. 4 shows part of a rack 84 comprising a plurality of horizontal supports 86 and vertical supports 88. It can be seen in FIG. 4 that rack damage has occurred in the form of a deformed horizontal support 86. When the industrial truck 1 with the system 50 passes the affected storage area, the computing unit 60 is able to recognize, from the constructed point cloud and corresponding image processing algorithms, that the horizontal support 86 is deformed. Based on this, the computing unit 60 can output an instruction to have the horizontal beam 86 repaired or replaced; a damage detection system (e.g. system and method for inspecting and automatically detecting damage or irregularities in a warehouse), for example), comprising:
a camera associated with a vehicle (Par. [0001]: a system for inspecting a warehouse, an industrial truck comprising such a system and a method for inspecting a warehouse using such a system; Par. [0031]: a truck which is adapted to perform a work function such as lifting or lowering a load; Par. [0059-71]: system 50 for inspecting a warehouse that is used in the industrial truck 1 shown in FIG. 1. The system 50 includes… system 50 includes the laser scanner 52 and the computing unit 60, the laser scanner 52 being disposed to scan the environment in order to generate scan data, and the computing unit 60 being disposed to receive the scan data and construct three-dimensional data of the environment based on these… industrial truck 1 preferably receives a transport order from a warehouse management system, which specifies the destination storage location where a load is to be deposited… target storage location can be scanned when the industrial truck passes the target storage location in direction G, and three-dimensional data of the target storage location can be constructed based on this… additional sensors, such as a camera, which also detects the storage location and/or the environment near the storage location during the deposit operation; a camera associated with a vehicle (e.g. system and method for inspecting and automatically detecting damage or irregularities in a warehouse, including an industrial truck (i.e. a vehicle) and a computing unit being disposed to receive scan data from laser scanner and construct three-dimensional data of an environment of the industrial truck, including three-dimensional data of a target storage location where a load is to be deposited by the industrial truck, for example, and including additional sensors, such as a camera (i.e. a camera associated with a vehicle), which also detects the storage location and/or the environment near the storage location during a deposit operation of the industrial truck, as indicated above, for example), for example); and
a controller configured to process acquired [three-dimensional data] (Par. [0019-21]: laser scanner has scanning directions that include the two horizontal directions and the vertical upward direction. The laser scanner thus has a scanning angle of at least 180°, so that racking equipment on both sides of the industrial truck can be captured… The laser scanner is also preferably adapted to scan the floor of the racking equipment on both sides. In this way, the entire rack front of a rack can be captured… computing unit is preferably adapted to recognize different storage elements from a constructed point cloud through image processing and to determine three-dimensional data of the individual recognized storage elements based on these. Image processing can take place either by segmenting the point cloud and then matching the individual segmented parts with predefined patterns or rules, or it can rely on an artificial intelligence method with learning capability, where the computing unit can be continuously trained with data to adapt to new environments; Par. [0059-63]: system 50 for inspecting a warehouse that is used in the industrial truck 1 shown in FIG. 1. The system 50 includes… system 50 includes the laser scanner 52 and the computing unit 60, the laser scanner 52 being disposed to scan the environment in order to generate scan data, and the computing unit 60 being disposed to receive the scan…computing unit 60 constructs three-dimensional data of the environment in the form of a point cloud. Through suitable image processing, the computing unit 60 can recognize various storage elements from the constructed point cloud and, based on these, determine three-dimensional data of the individual recognized storage elements; a controller configured to process acquired three-dimensional data (e.g. system and method for inspecting and automatically detecting damage or irregularities in a warehouse, including an industrial truck (i.e. a vehicle) and a computing unit being disposed to (i.e. a controller configured to) receive scan data from laser scanner and to construct three-dimensional data of an environment of the industrial truck (i.e. a controller configured to process acquired three-dimensional data), for example, including a computing unit adapted to recognize different storage elements from a constructed point cloud through image processing and to determine three-dimensional data of the individual recognized storage elements (i.e. a controller configured to process acquired three-dimensional data), as indicated above), for example) in order to:
recognise a safety structure in the [three-dimensional data] (Par. [0019-21]: laser scanner has scanning directions that include the two horizontal directions and the vertical upward direction. The laser scanner thus has a scanning angle of at least 180°, so that racking equipment on both sides of the industrial truck can be captured… The laser scanner is also preferably adapted to scan the floor of the racking equipment on both sides. In this way, the entire rack front of a rack can be captured… the computing unit is further adapted to detect individual storage elements from the constructed three-dimensional data of the environment. Storage elements in the context of this invention refer to all possible logistics elements or units that are used or required either for transport requests or for the storage of goods in a warehouse. They include stored goods such as merchandise or load units consisting of bundled merchandise or packages, storage aids such as pallets or containers, storage locations or bins, racks or racking elements such as vertical supports, horizontal supports… computing unit is preferably adapted to recognize different storage elements from a constructed point cloud through image processing and to determine three-dimensional data of the individual recognized storage elements based on these. Image processing can take place either by segmenting the point cloud and then matching the individual segmented parts with predefined patterns or rules, or it can rely on an artificial intelligence method with learning capability, where the computing unit can be continuously trained with data to adapt to new environments; Par. [0059-71]: system 50 includes the laser scanner 52 and the computing unit 60, the laser scanner 52 being disposed to scan the environment in order to generate scan data, and the computing unit 60 being disposed to receive the scan data and construct three-dimensional data of the environment based on these… the computing unit 60 constructs three-dimensional data of the environment in the form of a point cloud. Through suitable image processing, the computing unit 60 can recognize various storage elements from the constructed point cloud and, based on these, determine three-dimensional data of the individual recognized storage elements… FIG. 4 shows part of a rack 84 comprising a plurality of horizontal supports 86 and vertical supports 88. It can be seen in FIG. 4 that rack damage has occurred in the form of a deformed horizontal support 86. When the industrial truck 1 with the system 50 passes the affected storage area, the computing unit 60 is able to recognize, from the constructed point cloud and corresponding image processing algorithms, that the horizontal support 86 is deformed. Based on this, the computing unit 60 can output an instruction to have the horizontal beam 86 repaired or replaced… for example, only one rack support, in this case the vertical support 88 shown in FIG. 7, is scanned and recognized; in order to: recognise a safety structure in the three-dimensional data (e.g. system and method for inspecting and automatically detecting damage or irregularities in a warehouse, including an industrial truck (i.e. a vehicle) and a computing unit being disposed to (i.e. a controller configured to) receive scan data from laser scanner and to construct three-dimensional data of an environment of the industrial truck (i.e. a controller configured to process acquired three-dimensional data), for example, including a computing unit adapted to recognize different storage elements from a constructed point cloud through image processing and to determine three-dimensional data of the individual recognized storage elements, including storage aids, such as pallets or containers, storage locations or bins, racks or racking elements (i.e. safety structures), such as vertical supports, horizontal supports (i.e. in order to: recognise a safety structure in the three-dimensional data), as indicated above), for example);
compare the recognised safety structure in the acquired [three-dimensional data] with an other [three-dimensional data] of the same, or a corresponding, safety structure (Par. [0020-25]: computing unit is further adapted to detect individual storage elements from the constructed three-dimensional data of the environment. Storage elements in the context of this invention refer to all possible logistics elements or units that are used or required either for transport requests or for the storage of goods in a warehouse. They include… storage aids such as pallets or containers, storage locations or bins, racks or racking elements such as vertical supports, horizontal supports… The computing unit is preferably adapted to recognize different storage elements from a constructed point cloud through image processing and to determine three-dimensional data of the individual recognized storage elements based on these… the computing unit is further adapted to check a storage condition based on a comparison of the three-dimensional data of the individual detected storage elements with corresponding data from the warehouse management system… The computing unit is further preferably adapted to check a storage condition based on a comparison of the three-dimensional data of the individual detected storage elements with predefined reference data. The reference data can be, for example, specific regulatory data defined by standards, such as the permitted deflection of a rack support; Par. [0063]: a comparison of the three-dimensional data of individual detected storage elements with corresponding data from the warehouse management system and/or with predefined reference data; compare the recognised safety structure in the acquired three-dimensional data with an other three-dimensional data of the same, or a corresponding, safety structure (e.g. system and method for inspecting and automatically detecting damage or irregularities in a warehouse, including an industrial truck (i.e. a vehicle) and a computing unit being disposed to receive scan data from laser scanner and to construct three-dimensional data of an environment of the industrial truck, for example, including a computing unit adapted to recognize different storage elements from a constructed point cloud through image processing and to determine three-dimensional data of the individual recognized storage elements, including storage aids, such as pallets or containers, storage locations or bins, racks or racking elements (i.e. safety structures), such as vertical supports, horizontal supports, for example, and check a storage condition based on a comparison of the three-dimensional data of individual detected storage elements with corresponding data (i.e. or corresponding, safety structures) from a warehouse management system and/or with predefined reference data (i.e. compare the recognised safety structure in the acquired three-dimensional data with an other three-dimensional data of the same, or a corresponding, safety structure), as indicated above), for example); and
based on the comparison, provide a damage status of the safety structure (Par. [0024-26]: computing unit is further adapted to check a storage condition based on a comparison of the three-dimensional data of the individual detected storage elements with corresponding data from the warehouse management system… The computing unit is further preferably adapted to check a storage condition based on a comparison of the three-dimensional data of the individual detected storage elements with predefined reference data. The reference data can be, for example, specific regulatory data defined by standards… By comparing the three-dimensional data of the individual detected storage elements with the data stored in the warehouse management system or elsewhere, the computer unit can detect any storage irregularities and/or damage to individual storage elements; Par. [0059-64]: system 50 for inspecting a warehouse that is used in the industrial truck 1 shown in FIG. 1. The system 50 includes the laser scanner 52 and the computing unit 60, the laser scanner 52 being disposed to scan the environment in order to generate scan data, and the computing unit 60 being disposed to receive the scan data and construct three-dimensional data of the environment based on these… Based on the scan data… the computing unit 60 constructs three-dimensional data of the environment in the form of a point cloud. Through suitable image processing, the computing unit 60 can recognize various storage elements from the constructed point cloud and, based on these, determine three-dimensional data of the individual recognized storage elements. Based on a comparison of the three-dimensional data of individual detected storage elements with corresponding data from the warehouse management system and/or with predefined reference data, the computing unit 60 is able to automatically detect damage or irregularities in a warehouse… FIG. 4 shows part of a rack 84 comprising a plurality of horizontal supports 86 and vertical supports 88. It can be seen in FIG. 4 that rack damage has occurred in the form of a deformed horizontal support 86. When the industrial truck 1 with the system 50 passes the affected storage area, the computing unit 60 is able to recognize, from the constructed point cloud and corresponding image processing algorithms, that the horizontal support 86 is deformed. Based on this, the computing unit 60 can output an instruction to have the horizontal beam 86 repaired or replaced; and based on the comparison, provide a damage status of the safety structure (e.g. system and method for inspecting and automatically detecting damage or irregularities in a warehouse, including an industrial truck (i.e. a vehicle) and a computing unit being disposed to receive scan data from laser scanner and to construct three-dimensional data of an environment of the industrial truck, for example, including a computing unit adapted to recognize different storage elements from a constructed point cloud through image processing and to determine three-dimensional data of the individual recognized storage elements, including storage aids, such as pallets or containers, storage locations or bins, racks or racking elements (i.e. safety structures), such as vertical supports, horizontal supports, for example, and check a storage condition based on a comparison of the three-dimensional data of individual detected storage elements with corresponding data (i.e. or corresponding, safety structures) from a warehouse management system and/or with predefined reference data, for example, and based on the comparison of the three-dimensional data of individual detected storage elements with corresponding data from the warehouse management system and/or with predefined reference data, the computer/computing automatically detects any storage irregularities and/or damage to individual storage elements, as indicated above), for example).
Bistry’s teachings above disclose the system and method for inspecting and automatically detecting damage or irregularities in a warehouse, including an industrial truck (i.e. a vehicle) and a computing unit being disposed to receive scan data from laser scanner and construct three-dimensional data of an environment of the industrial truck, including three-dimensional data of a target storage location where a load is to be deposited by the industrial truck, for example, and including additional sensors, such as a camera (i.e. a camera associated with a vehicle), which also detects the storage location and/or the environment near the storage location during a deposit operation of the industrial truck, as indicated above, but fails to disclose that the camera is configured to acquire images of a vicinity of the vehicle and that the image processing performed above is being performed on the acquired images.
However, Boyle teaches wherein the camera is configured to acquire images of a vicinity of the vehicle (Par. [0020]: a method, system, server and data capture device for roadside asset tracking and maintenance monitoring, which will overcome or substantially ameliorate at least some of the deficiencies of the above prior art, or to at least provide an alternative. Specifically, in embodiments as will be described in further detail below, there is described an asset and maintenance tracking system which may be deployed on roadside maintenance vehicles, such as grass cutters, to record roadside assets, such as traffic management post, reflector posts and the like, subsequently monitor the previously recorded assets, and to monitor the maintenance conducted by the roadside maintenance vehicle; Par. [0048-49]: mobile unit may comprise at least one camera for capturing image data from the perspective of the mobile unit… The mobile unit may comprise forward and rearward facing cameras for respectively capturing forward and rearward facing image data from the perspective of the mobile unit and the map representation may be configured for displaying before and after maintenance comparison imagery utilising the image data received from the forward and rearward facing cameras; Par. [0071-74]: maintenance data may comprise the location of a maintenance machinery… maintenance data may comprise maintenance image data… The maintenance image data may comprise maintenance image data representing a rearward image from the maintenance vehicle… The maintenance image data may comprise maintenance image data representing a forward image from the maintenance vehicle; Par. [0187-192]: the machinery 300 may comprise cameras, especially the forward facing camera 320 to record information relating to an asset… camera 320 would be appropriately located for an ideal vantage for recording image data relating to various roadside assets… the data capture device 105 may be adapted to capture images from the cameras wherein the images are processed using an image processing technique; wherein the camera is configured to acquire images of a vicinity of the vehicle (e.g. method, system, server and data capture device for roadside asset tracking and maintenance monitoring include a mobile vehicle (i.e. a vehicle), such as a maintenance vehicle, and a camera for capturing image data from the perspective of the mobile unit (i.e. camera is configured to acquire images of a vicinity of the vehicle), as indicated above), for example);
process the acquired images (Par. [0055-56]: mobile unit may be configured for sending the asset image data to the server and the service performs the asset type image recognition… there is provided a server for roadside asset tracking, the server may comprise a processor for processing digital data; Par. [0078]: there is provided a data capture device for roadside asset tracking, the data capture device may comprise a processor for processing digital data; Par. [0183-192]: the data capture device 105 is adapted to automate the asset data recording process… the data capture device 105 may be adapted to capture images from the cameras wherein the images are processed using an image processing technique) in order to:
recognise a safety structure in the acquired images (Par. [0001-2]: a system, server and data capture device for roadside asset tracking and maintenance monitoring… Roadside assets, such as public infrastructure assets, such as speed management posts, reflector posts, roadside barriers and the like are installed and subsequently repaired and maintained; Par. [0020-22]: a method, system, server and data capture device for roadside asset tracking and maintenance monitoring… an asset and maintenance tracking system which may be deployed on roadside maintenance vehicles, such as grass cutters, to record roadside assets, such as traffic management post, reflector posts and the like, subsequently monitor the previously recorded assets, and to monitor the maintenance conducted by the roadside maintenance vehicle… the system may perform asset identification and valuation wherein once data is entered, the system will provide a data analysis option where each asset image is analysed for type, condition and valuation at the time of entry. Any asset needing attention will be flagged and given a liability risk rating… utilising an asset type image recognition technique for automating the identification of the roadside assets… The asset type image recognition technique may comprise an image recognition stage comprising edge shape detection; Par. [0052-56]: method may further comprise using an asset condition estimate image recognition technique to estimate a condition of each of the roadside assets… mobile unit may be configured for sending the asset image data to the server and the service performs the asset type image recognition… there is provided a server for roadside asset tracking, the server may comprise a processor for processing digital data; Par. [0084]: processor may be further controlled by the computer program code to identify the asset in accordance with an image recognition technique; Par. [0120-166]: the asset data input device 115 may comprise an imaging device, such as a camera or the like adapted to capture images of various assets. These images of the assets maybe then manually classified by operators, or alternatively utilised for automated image recognition purposes for identifying assets… the system 100 may be adapted for the automated identification of assets using image recognition technique performed on the image data… the data capture device 105, especially by utilising image recognition technique may identify assets requiring maintenance; recognise a safety structure in the acquired images (e.g. method, system, server and data capture device for roadside asset tracking and maintenance monitoring include a mobile vehicle (i.e. a vehicle), such as a maintenance vehicle, and a camera for capturing image data from the perspective of the mobile unit, as indicated above, for example, including an image recognition technique for automating identification of roadside assets, such as public infrastructure assets, roadside barriers (i.e. safety structures), for example, in which captured images from the camera are processed using an image processing technique, as indicated above), for example);
compare the recognised safety structure in the acquired image with an other image of the same, or a corresponding, safety structure; and
based on the comparison, provide a damage-status-signal that represents a damage status of the safety structure (Par. [0020-42]: a method, system, server and data capture device for roadside asset tracking and maintenance monitoring, which will overcome or substantially ameliorate at least some of the deficiencies of the above prior art, or to at least provide an alternative… an asset and maintenance tracking system which may be deployed on roadside maintenance vehicles, such as grass cutters, to record roadside assets, such as traffic management post, reflector posts and the like, subsequently monitor the previously recorded assets, and to monitor the maintenance conducted by the roadside maintenance vehicle… the system may perform asset identification and valuation wherein once data is entered, the system will provide a data analysis option where each asset image is analysed for type, condition and valuation at the time of entry. Any asset needing attention will be flagged… system may further allow for real-time incident reporting wherein any incidents encountered in the maintenance run will be recorded and flagged for council or a supervisor's attention immediately reducing the risk of escalation of any incidents… method may further comprise image comparison for detecting roadside asset damage or degradation to identify assets requiring maintenance; Par. [0066-68]: processor may be further controlled by the computer program code to select, from the database, previously recorded asset data representing a plurality of previously recorded assets; compare the asset data against the previously recorded asset data… The previously recorded asset data may comprise previously recorded image data representing a previous image of the asset and the asset data may comprise present image data representing a previous image of the asset and wherein, in identifying the asset as requiring maintenance, the processor may be further controlled by the computer program code to compare the previously recorded image data and the present image data; Par. [0084-85]: processor may be further controlled by the computer program code to identify the asset in accordance with an image recognition technique… The image recognition technique may comprise an image comparison technique with a plurality of asset type images; Par. [0191-193]: the system 100 is adapted for the automated identification of assets in accordance with the image data. In one particular embodiment, the system 100 may employ image recognition for recognising various assets and asset types. It should be noted that the image recognition may be performed by the data capture device 105… data capture device 105 may be adapted to capture images from the cameras wherein the images are processed using an image processing technique… the database 135 may comprise exemplary representative imagery of various roadside assets for comparison purposes during the image recognition technique; Par. [0254-255]: method may further comprise image comparison for detecting roadside asset damage or degradation to identify roadside assets requiring maintenance. For example, the method may compare asset image data against previously recorded asset image data and detect changes… For those assets that show a change in colour or shape exceeding a threshold may trigger the system to update the database 135 with a flag indicating that a particular asset requires maintenance. For example, should a roadside be damaged… the image comparison may detect the colour degradation and flag the asset for maintenance; compare the recognised safety structure in the acquired image with an other image of the same, or a corresponding, safety structure; and based on the comparison, provide a damage-status-signal that represents a damage status of the safety structure (e.g. method, system, server and data capture device for roadside asset tracking and maintenance monitoring include a processor controlled by computer program code to (i.e. the controller is configured to) perform processing functions including an image recognition technique for automating identification of roadside assets, such as public infrastructure assets, roadside barriers (i.e. recognized safety structures), for example, in which captured/recorded images from the camera are processed using the image processing technique (i.e. recognized safety structures in acquired images), including comparing captured asset image data (i.e. compare the recognised safety structure in the image) against previously recorded asset image data (i.e. with an other image of the same, or a corresponding, safety structure), including exemplary representative imagery of various roadside assets for comparison purposes during the image recognition technique, to detect changes, for example, including performing image comparison for detecting roadside asset damage or degradation (i.e. a damage status of the safety structure) to identify assets requiring maintenance, for example, and for assets that show a change exceeding a threshold trigger, the system provides a flag (i.e. status-signal) indicating that a particular asset requires maintenance, as indicated above), for example);
wherein the controller is configured to compare the recognised safety structure in the image with the other image of the same, or a corresponding, safety structure (Par. [0066-68]: processor may be further controlled by the computer program code to select, from the database, previously recorded asset data representing a plurality of previously recorded assets; compare the asset data against the previously recorded asset data… The previously recorded asset data may comprise previously recorded image data representing a previous image of the asset and the asset data may comprise present image data representing a previous image of the asset and wherein, in identifying the asset as requiring maintenance, the processor may be further controlled by the computer program code to compare the previously recorded image data and the present image data; Par. [0084-85]: processor may be further controlled by the computer program code to identify the asset in accordance with an image recognition technique… The image recognition technique may comprise an image comparison technique with a plurality of asset type images; Par. [0193]: database 135 may comprise exemplary representative imagery of various roadside assets for comparison purposes during the image recognition technique; wherein the controller is configured to compare the recognised safety structure in the image with the other image of the same, or a corresponding, safety structure (e.g. method, system, server and data capture device for roadside asset tracking and maintenance monitoring include a processor controlled by computer program code to (i.e. the controller is configured to) perform processing functions including an image recognition technique for automating identification of roadside assets, such as public infrastructure assets, roadside barriers (i.e. recognized safety structures), for example, in which captured/recorded images from the camera are processed using the image processing technique (i.e. recognized safety structures in acquired images), including comparing captured asset image data (i.e. compare the recognised safety structure in the image) against previously recorded asset image data (i.e. other image of the same, or a corresponding, safety structure), including exemplary representative imagery of various roadside assets for comparison purposes during the image recognition technique, to detect changes, as indicated above), for example) by:
determining an identifier for a type of safety structure that is recognised in the image (Par. [0022-29]: system may perform asset identification… the system will provide a data analysis option where each asset image is analysed for type… utilising an asset type image recognition technique for automating the identification of the roadside assets… The asset type image recognition technique may comprise an image recognition stage comprising edge shape detection; Par. [0055-57]: mobile unit may be configured for sending the asset image data to the server and the service performs the asset type image recognition… asset data representing a plurality of assets, the asset data may comprise at least unique asset identifier data and asset location data… The asset data further may comprise the asset type and wherein the processor may be further controlled by the computer program code to store, in the database, the asset type in association with the unique asset identifier; Par. [0083-85]: processor may be further controlled by the computer program code to identify the asset in accordance with the image data… processor may be further controlled by the computer program code to identify the asset in accordance with an image recognition technique… The image recognition technique may comprise an image comparison technique with a plurality of asset type images; Par. [0113-120]: system 100 comprises at least one data capture device 105 operably coupled to a server computing device 140 across a data network 130. In this manner, the data capture device 105 is adapted to record information relating to various roadside assets and convey the asset data to the server 140 for recordal… The asset data may comprises asset type and location data representing a plurality of assets and their respective locations… operably coupled to the computing device 200 is an asset data input device 115 adapted to receive asset data relating to a plurality of assets… the asset data input device 115 may comprise an imaging device, such as a camera or the like adapted to capture images of various assets. These images of the assets maybe… utilised for automated image recognition purposes for identifying assets; Par. [0145-156]: server 140 comprises a database interface for interfacing with the database 135… the database is adapted for storing asset data representing a plurality of assets wherein the asset data comprises at least a unique asset identifier data and asset location data… the server 140 is adapted to store, in the database 135, asset type and location data representing the asset and the location thereof… the server 140 may further be adapted to receive, from the data capture device 105, the asset type… system 100 may be adapted for the automated identification of assets using image recognition technique performed on the image data; Par. [0191]: system 100 is adapted for the automated identification of assets in accordance with the image data… the system 100 may employ image recognition for recognising various assets and asset types; Par. [0229-236]: method comprises utilising an asset type image recognition technique for automating the identification of the roadside assets. For example, the roadside asset types may be divided into any number of roadside asset type categories such as posts, barriers, roadsigns… or subsequent passes, the method further comprises identifying missing roadside assets by comparing the previously identified asset types at the respective locations… asset data capture device 115 comprises a camera (or is operably couples to a camera) for capturing roadside asset image data. In this manner, the method comprises utilising image recognition for identifying the roadside asset types… the method comprises employing the asset type image recognition to identify assets from those images. For example, the method may comprise identifying roadside posts, roadside signage and the like from the image is captured by the camera; determining an identifier for a type of safety structure that is recognised in the image (e.g. method, system, server and data capture device for roadside asset tracking and maintenance monitoring include a processor controlled by computer program code to perform processing functions including an image recognition technique for automating identification of roadside assets, such as public infrastructure assets, roadside barriers (i.e. recognized safety structures), for example, in which captured/recorded images from the camera are processed using the image processing technique (i.e. recognized safety structures in images), including identifying roadside assets (i.e. determining identifiers) in accordance with the image data, for example, in which each asset image is analyzed for type by using an asset type image recognition technique for automating the identification of the roadside assets by determining asset type and location data representing a plurality of assets and their respective locations (i.e. determining an identifier for a type of safety structure that is recognised in the image), as indicated above), for example);
retrieving one or more images of the same type of safety structure from a memory (Par. [0028]: receiving the roadside asset image data representing images taken of a plurality of roadside assets; Par. [0066-68]: processor may be further controlled by the computer program code to select, from the database, previously recorded asset data representing a plurality of previously recorded assets; compare the asset data against the previously recorded asset data… The previously recorded asset data may comprise previously recorded image data representing a previous image of the asset and the asset data may comprise present image data representing a previous image of the asset and wherein, in identifying the asset as requiring maintenance, the processor may be further controlled by the computer program code to compare the previously recorded image data and the present image data; Par. [0079-85]: asset data receiver may comprise an image capture device and wherein, in use, the asset data may comprise image data representing an image of the asset… identify the asset in accordance with an image recognition technique…The image recognition technique may comprise an image comparison technique with a plurality of asset type images; Par. [0119-120]: computing device 200 is adapted to receive asset data and send the asset data to the server 140 across the network 130… computing device 200 is an asset data input device 115 adapted to receive asset data relating to a plurality of assets; Par. [0154-155]: server 140 is adapted to receive, from the data capture device 105 image data relating to a particular asset for storage within the database 135… image data may be stored for each asset recorded within the database 135, for subsequent visual inspection and other purposes; Par. [0184-193]: as the machinery 300 encounters various roadside assets, the data capture device 105 is adapted to receive from the asset data input, asset data relating to the asset and send, via the wireless network interface 125 and the network 130, the asset data to the server 140 for recordal within the database 135… the data capture device 105 may be adapted to receive information from the server 140 also, including information relating to previously recorded assets… database 135 may comprise exemplary representative imagery of various roadside assets for comparison purposes during the image recognition technique; Par. [0254]: method may further comprise image comparison for detecting roadside asset damage or degradation to identify roadside assets requiring maintenance. For example, the method may compare asset image data against previously recorded asset image data and detect changes; retrieving one or more images of the same type of safety structure from a memory (e.g. method, system, server and data capture device for roadside asset tracking and maintenance monitoring include a processor controlled by computer program code to perform processing functions including an image recognition technique for automating identification of roadside assets, such as public infrastructure assets, roadside barriers (i.e. safety structures), for example, and to perform further processing functions including receiving (i.e. retrieving, obtaining, etc.) roadside asset image data representing images taken of a plurality of roadside assets, including previously recorded (i.e. stored in memory) asset image data that includes a plurality of asset type images, for example, including previously recorded asset data comprising previously recorded image data representing a previous image of an asset (i.e. retrieving one or more images of the same type of safety structure from a memory), as indicated above), for example); and
comparing the recognised safety structure in the image with the one or more images of the same type of safety structure retrieved from the memory (Par. [0066-68]: processor may be further controlled by the computer program code to select, from the database, previously recorded asset data representing a plurality of previously recorded assets; compare the asset data against the previously recorded asset data… The previously recorded asset data may comprise previously recorded image data representing a previous image of the asset and the asset data may comprise present image data representing a previous image of the asset and wherein, in identifying the asset as requiring maintenance, the processor may be further controlled by the computer program code to compare the previously recorded image data and the present image data; Par. [0084-85]: processor may be further controlled by the computer program code to identify the asset in accordance with an image recognition technique… The image recognition technique may comprise an image comparison technique with a plurality of asset type images; Par. [0193]: database 135 may comprise exemplary representative imagery of various roadside assets for comparison purposes during the image recognition technique; comparing the recognised safety structure in the image with the one or more images of the same type of safety structure retrieved from the memory (e.g. method, system, server and data capture device for roadside asset tracking and maintenance monitoring include a processor controlled by computer program code to perform processing functions including an image recognition technique for automating identification of roadside assets, such as public infrastructure assets, roadside barriers (i.e. recognized safety structures), for example, in which captured/recorded images from the camera are processed using the image processing technique (i.e. recognized safety structures in acquired images), including comparing captured asset image data (i.e. compare the recognised safety structure in the image) against previously recorded (i.e. stored in memory) asset image data that includes a plurality of asset type images, for example, including previously recorded asset data comprising previously recorded image data representing a previous image of an asset (i.e. comparing the recognised safety structure in the image with the one or more images of the same type of safety structure retrieved from the memory), for example, to detect changes, as indicated above), for example).
Bistry and Boyle are considered to be analogous art because they pertain to image processing applications. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the apparatus and method for inspecting and automatically detecting damage or irregularities in a warehouse (as disclosed by Bistry) with wherein the camera is configured to acquire images of a vicinity of the vehicle; process the acquired images in order to: recognise a safety structure in the acquired images; compare the recognised safety structure in the acquired images; and based on the comparison, provide a damage-status-signal that represents a damage status of the safety structure, wherein the controller is configured to compare the recognised safety structure in the image with the other image of the same, or a corresponding, safety structure by: determining an identifier for a type of safety structure that is recognised in the image; retrieving one or more images of the same type of safety structure from a memory; and comparing the recognised safety structure in the image with the one or more images of the same type of safety structure retrieved from the memory. (as taught by Boyle, Abstract, Par. [0001-2, 20-42, 48-49, 52-57, 66-68, 71-74, 79-85, 113-166, 184-193, 229-236, 254-255) to identify roadside assets requiring maintenance by detecting roadside asset damage or degradation of the roadside assets, such as public infrastructure assets, roadside barriers (Boyle, Abstract, Par. [0001-2, 20, 42, 254]), for example.
Regarding claim 2, claim 1 is incorporated and the combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), wherein the vehicle is a forklift truck (Bistry, Par. [0053]: FIG. 1 shows a side view of an exemplary embodiment of an industrial truck 1 with a system for inspecting a warehouse according to the invention, in which exemplary embodiment the industrial truck is a high-bay stacker designed as a trilateral forklift).
Regarding claim 3, claim 1 and the combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), further comprising a plurality of cameras, each configured to acquire images of the vicinity of the vehicle (Boyle, Par. [0024-49]: may be further adapted to provide a photographic summary of each maintenance job wherein front and rear mounted cameras… mobile unit may comprise forward and rearward facing cameras for respectively capturing forward and rearward facing image data from the perspective of the mobile unit).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 4, claim 1 is incorporated and the combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), wherein the controller is configured to recognise the safety structure in the image by:
performing an object recognition operation on the image in order to recognise one or more predetermined safety structures in the image (Bistry, Par. [0021]: computing unit is preferably adapted to recognize different storage elements from a constructed point cloud through image processing and to determine three-dimensional data of the individual recognized storage elements based on these. Image processing can take place either by segmenting the point cloud and then matching the individual segmented parts with predefined patterns or rules, or it can rely on an artificial intelligence method with learning capability, where the computing unit can be continuously trained with data to adapt to new environments; Par. [0059-64]: computing unit 60 constructs three-dimensional data of the environment in the form of a point cloud. Through suitable image processing, the computing unit 60 can recognize various storage elements from the constructed point cloud and, based on these, determine three-dimensional data of the individual recognized storage elements. Based on a comparison of the three-dimensional data of individual detected storage elements with corresponding data from the warehouse management system and/or with predefined reference data, the computing unit 60 is able to automatically detect damage or irregularities in a warehouse. FIG. 4 and FIG. 5 each illustrate an example of this… FIG. 4 shows part of a rack 84 comprising a plurality of horizontal supports 86 and vertical supports 88. It can be seen in FIG. 4 that rack damage has occurred in the form of a deformed horizontal support 86. When the industrial truck 1 with the system 50 passes the affected storage area, the computing unit 60 is able to recognize, from the constructed point cloud and corresponding image processing algorithms, that the horizontal support 86 is deformed. Based on this, the computing unit 60 can output an instruction to have the horizontal beam 86 repaired or replaced; Par. [0084]: processor may be further controlled by the computer program code to identify the asset in accordance with an image recognition technique); or
applying a machine learning algorithm to the image in order to determine a classification of a safety structure as one that was visible in an image that was used as training data for training the machine learning algorithm (Bistry, Par. [0021]: computing unit is preferably adapted to recognize different storage elements from a constructed point cloud through image processing and to determine three-dimensional data of the individual recognized storage elements based on these. Image processing can take place either by segmenting the point cloud and then matching the individual segmented parts with predefined patterns or rules, or it can rely on an artificial intelligence method with learning capability, where the computing unit can be continuously trained with data to adapt to new environments).
Regarding claim 6, claim 1 is incorporated and the combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), wherein the one or more images of the same type of safety structure retrieved from the memory comprise images of the safety structure in an undamaged state and/or one or more damaged states (Boyle, Par. [0020-22]: a method, system, server and data capture device for roadside asset tracking and maintenance monitoring, which will overcome or substantially ameliorate at least some of the deficiencies of the above prior art, or to at least provide an alternative… an asset and maintenance tracking system which may be deployed on roadside maintenance vehicles, such as grass cutters, to record roadside assets, such as traffic management post, reflector posts and the like, subsequently monitor the previously recorded assets, and to monitor the maintenance conducted by the roadside maintenance vehicle… the system may perform asset identification and valuation wherein once data is entered, the system will provide a data analysis option where each asset image is analysed for type, condition; Par. [0042]: method may further comprise image comparison for detecting roadside asset damage or degradation to identify assets requiring maintenance; Par. [0066-68]: processor may be further controlled by the computer program code to select, from the database, previously recorded asset data representing a plurality of previously recorded assets; compare the asset data against the previously recorded asset data… The previously recorded asset data may comprise previously recorded image data representing a previous image of the asset and the asset data may comprise present image data representing a previous image of the asset and wherein, in identifying the asset as requiring maintenance, the processor may be further controlled by the computer program code to compare the previously recorded image data and the present image data; Par. [0184-193]: as the machinery 300 encounters various roadside assets, the data capture device 105 is adapted to receive from the asset data input, asset data relating to the asset and send, via the wireless network interface 125 and the network 130, the asset data to the server 140 for recordal within the database 135… the data capture device 105 may be adapted to receive information from the server 140 also, including information relating to previously recorded assets… database 135 may comprise exemplary representative imagery of various roadside assets for comparison purposes during the image recognition technique; Par. [0254-255]: method may further comprise image comparison for detecting roadside asset damage or degradation to identify roadside assets requiring maintenance. For example, the method may compare asset image data against previously recorded asset image data and detect changes… For those assets that show a change in colour or shape exceeding a threshold may trigger the system to update the database 135 with a flag indicating that a particular asset requires maintenance. For example, should a roadside be damaged… the image comparison may detect the colour degradation and flag the asset for maintenance; wherein the one or more images of the same type of safety structure retrieved from the memory comprise images of the safety structure in an undamaged state and/or one or more damaged states (e.g. method, system, server and data capture device for roadside asset tracking and maintenance monitoring include a processor controlled by computer program code to perform processing functions including an image recognition technique for automating identification of roadside assets, such as public infrastructure assets, roadside barriers (i.e. safety structures), for example, including performing image comparison for detecting roadside asset damage or degradation (i.e. a damage status of the safety structure) to identify assets requiring maintenance, for example, including previously recorded (i.e. stored in memory) asset image data that includes a plurality of asset type images, for example, including previously recorded asset data comprising previously recorded image data representing a previous image of an asset (i.e. wherein the one or more images of the same type of safety structure retrieved from the memory comprise images of the safety structure in an undamaged state and/or one or more damaged states), as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 7, claim 1 is incorporated and the combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), wherein:
comparing the recognised safety structure in the image with the one or more images of the same type of safety structure retrieved from the memory comprises determining a degree of similarity between the images (Bistry, Par. [0021-26]: computing unit is preferably adapted to recognize different storage elements from a constructed point cloud through image processing and to determine three-dimensional data of the individual recognized storage elements based on these. Image processing can take place either by segmenting the point cloud and then matching the individual segmented parts with predefined patterns or rules; Bistry, Par. [0063]: computing unit 60 constructs three-dimensional data of the environment in the form of a point cloud. Through suitable image processing, the computing unit 60 can recognize various storage elements from the constructed point cloud and, based on these, determine three-dimensional data of the individual recognized storage elements. Based on a comparison of the three-dimensional data of individual detected storage elements with corresponding data from the warehouse management system and/or with predefined reference data, the computing unit 60 is able to automatically detect damage or irregularities in a warehouse; Boyle, Par. [0254-255]: method may further comprise image comparison for detecting roadside asset damage or degradation to identify roadside assets requiring maintenance. For example, the method may compare asset image data against previously recorded asset image data and detect changes… For those assets that show a change in colour or shape exceeding a threshold may trigger the system to update the database 135 with a flag indicating that a particular asset requires maintenance. For example, should a roadside be damaged… the image comparison may detect the colour degradation and flag the asset for maintenance); and
the controller is configured to provide the damage-status-signal, that represents the damage status of the safety structure, based on the determined degree of similarity (Bistry, Par. [0059-63]: system 50 for inspecting a warehouse that is used in the industrial truck 1 shown in FIG. 1. The system 50 includes the laser scanner 52 and the computing unit 60, the laser scanner 52 being disposed to scan the environment in order to generate scan data, and the computing unit 60 being disposed to receive the scan data and construct three-dimensional data of the environment based on these… Based on the scan data… the computing unit 60 constructs three-dimensional data of the environment in the form of a point cloud. Through suitable image processing, the computing unit 60 can recognize various storage elements from the constructed point cloud and, based on these, determine three-dimensional data of the individual recognized storage elements. Based on a comparison of the three-dimensional data of individual detected storage elements with corresponding data from the warehouse management system and/or with predefined reference data, the computing unit 60 is able to automatically detect damage or irregularities in a warehouse; Boyle, Par. [0191-193]: the system 100 is adapted for the automated identification of assets in accordance with the image data. In one particular embodiment, the system 100 may employ image recognition for recognising various assets and asset types. It should be noted that the image recognition may be performed by the data capture device 105… data capture device 105 may be adapted to capture images from the cameras wherein the images are processed using an image processing technique… the database 135 may comprise exemplary representative imagery of various roadside assets for comparison purposes during the image recognition technique; Boyle, Par. [0254-255]: method may further comprise image comparison for detecting roadside asset damage or degradation to identify roadside assets requiring maintenance. For example, the method may compare asset image data against previously recorded asset image data and detect changes… For those assets that show a change in colour or shape exceeding a threshold may trigger the system to update the database 135 with a flag indicating that a particular asset requires maintenance. For example, should a roadside be damaged… the image comparison may detect the colour degradation and flag the asset for maintenance).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 8, claim 1 is incorporated and the combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), wherein the controller is configured to determine the identifier for the type of safety structure that is recognised in the image by reading a machine-readable code that is visible in the acquired image (Boyle, Par. [0055-94]: mobile unit may be configured for sending the asset image data to the server and the service performs the asset type image recognition… a server for roadside asset tracking, the server may comprise a processor for processing digital data; a memory device for storing digital data including computer program code, the memory device being operably coupled to the processor; a network interface adapted for sending and receiving data across a data network, the network interface being operably coupled to the processor; and a database adapted for storing asset data representing a plurality of assets, the asset data may comprise at least unique asset identifier data and asset location data, the database being operably coupled to the processor, wherein, in use, the processor may be controlled by the computer program code to receive, via the network interface, from a remote data capture device, asset data relating to an asset, the asset data may comprise at least location data representing the location of the asset; and insert, into the database, the asset data in association with a unique asset identifier and the location data… The asset data further may comprise the asset type and wherein the processor may be further controlled by the computer program code to store, in the database, the asset type in association with the unique asset identifier… asset data further may comprise image data and wherein the processor may be further controlled by the computer program code to store, in the database, the image data in association with the unique asset identifier… asset data input device may comprise a scanning device adapted to scan the asset data from a data bearing computer readable medium… The data bearing computer readable medium may comprise a barcode… The barcode may comprise at least one of a 2-D and 3-D barcode… the asset data input device 115 may comprise an imaging device, such as a camera or the like adapted to capture images of various assets… the asset data input device 150 may comprise a scanning device adapted to scan a computer readable media attached to the asset, such as a barcode).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 12, claim 1 is incorporated and the combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), further comprising an alert signal generator that is configured to selectively provide an alert based on the damage-status-signal (Boyle, Par. [0020-42]: a method, system, server and data capture device for roadside asset tracking and maintenance monitoring, which will overcome or substantially ameliorate at least some of the deficiencies of the above prior art, or to at least provide an alternative… an asset and maintenance tracking system which may be deployed on roadside maintenance vehicles, such as grass cutters, to record roadside assets, such as traffic management post, reflector posts and the like, subsequently monitor the previously recorded assets, and to monitor the maintenance conducted by the roadside maintenance vehicle… the system may perform asset identification and valuation wherein once data is entered, the system will provide a data analysis option where each asset image is analysed for type, condition and valuation at the time of entry. Any asset needing attention will be flagged and given a liability risk rating… system may further allow for real-time incident reporting wherein any incidents encountered in the maintenance run will be recorded and flagged for council or a supervisor's attention immediately reducing the risk of escalation of any incidents… method may further comprise image comparison for detecting roadside asset damage or degradation to identify assets requiring maintenance; Boyle, Par. [0254-255]: method may further comprise image comparison for detecting roadside asset damage or degradation to identify roadside assets requiring maintenance. For example, the method may compare asset image data against previously recorded asset image data and detect changes… For those assets that show a change in colour or shape exceeding a threshold may trigger the system to update the database 135 with a flag indicating that a particular asset requires maintenance. For example, should a roadside be damaged… the image comparison may detect the colour degradation and flag the asset for maintenance).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 13, claim 12 is incorporated and the combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), wherein the controller is configured to:
determine a location of the safety structure that is recognised in the image (Boyle, Par. [0028-58]: receiving the roadside asset image data representing images taken of a plurality of roadside assets; receiving location data from the GPS receiver representing the respective locations the roadside assets; utilising an asset type image recognition technique for automating the identification of the roadside assets; comparing roadside asset data of the asset tracking register database to: record newly identified roadside assets and their respective locations in the asset tracking register database; and identify missing roadside assets… method may further comprise generating a map representation representing the roadside section and the locations of the identified roadside assets along the roadside section…The mobile unit may comprise at least one camera for capturing image data from the perspective of the mobile unit and the map representation may be configured for selectively displaying image data from the at least one camera at a plurality of locations along the roadside section… the asset data may comprise at least unique asset identifier data and asset location data, the database being operably coupled to the processor, wherein, in use, the processor may be controlled by the computer program code to receive, via the network interface, from a remote data capture device, asset data relating to an asset, the asset data may comprise at least location data representing the location of the asset; Boyle, Par. [0145-146]: server 140 comprises a database interface for interfacing with the database 135. In this regard, the database is adapted for storing asset data representing a plurality of assets wherein the asset data comprises at least a unique asset identifier data and asset location data… the server 140 is adapted to receive, from the data capture device 105 asset data relating to a roadside asset, the asset data comprising at least the location of the roadside asset. In this manner, the server 140 is adapted to store, in the database 135, asset type and location data representing the asset and the location thereof); and
provide an alert that is based on the determined location of the safety structure (Boyle, Par. [0022-58]: system may perform asset identification and valuation wherein once data is entered, the system will provide a data analysis option where each asset image is analysed for type, condition and valuation at the time of entry. Any asset needing attention will be flagged…receiving the roadside asset image data representing images taken of a plurality of roadside assets; receiving location data from the GPS receiver representing the respective locations the roadside assets; utilising an asset type image recognition technique for automating the identification of the roadside assets; comparing roadside asset data of the asset tracking register database to: record newly identified roadside assets and their respective locations in the asset tracking register database; and identify missing roadside assets… method may further comprise generating a map representation representing the roadside section and the locations of the identified roadside assets along the roadside section…The mobile unit may comprise at least one camera for capturing image data from the perspective of the mobile unit and the map representation may be configured for selectively displaying image data from the at least one camera at a plurality of locations along the roadside section… the asset data may comprise at least unique asset identifier data and asset location data, the database being operably coupled to the processor, wherein, in use, the processor may be controlled by the computer program code to receive, via the network interface, from a remote data capture device, asset data relating to an asset, the asset data may comprise at least location data representing the location of the asset; Boyle, Par. [0236-255]: , along the roadside section the camera may capture periodic images, such as in front of the mobile unit 300, or that which is engaged by the grass cutting device 310 and wherein the method comprises employing the asset type image recognition to identify assets from those images… the data capture device 105 may transmit the image data and the location data from the location sensor 110 to the server 140 for processing by the server 140… method may further comprise image comparison for detecting roadside asset damage or degradation to identify roadside assets requiring maintenance. For example, the method may compare asset image data against previously recorded asset image data and detect changes… For those assets that show a change in colour or shape exceeding a threshold may trigger the system to update the database 135 with a flag indicating that a particular asset requires maintenance. For example, should a roadside be damaged… the image comparison may detect the colour degradation and flag the asset for maintenance).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 14, claim 1 is incorporated and the combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), wherein the controller is configured to trigger the camera to acquire the image:
periodically (Boyle, Par. [0086]: processor may be further controlled by the computer program code to capture the image data upon receipt of a trigger signal; Par. [0175]: maintenance machinery 300 may comprise a forward facing camera 320 and a rearward facing camera 315. In this manner, at periodic intervals, the server 140 may be adapted to receive, from the data capture device 105 image data from these cameras; Par. [0190-236]: timing for the taking of the second rearward facing image may be timed in accordance with a predetermined time interval… along the roadside section the camera may capture periodic images, such as in front of the mobile unit 300);
in response to the vehicle having a predetermined location (Boyle, Par. [0022-86]: system may perform asset identification and valuation wherein once data is entered, the system will provide a data analysis option where each asset image is analysed for type, condition and valuation at the time of entry. Any asset needing attention will be flagged…receiving the roadside asset image data representing images taken of a plurality of roadside assets; receiving location data from the GPS receiver representing the respective locations the roadside assets; utilising an asset type image recognition technique for automating the identification of the roadside assets; comparing roadside asset data of the asset tracking register database to: record newly identified roadside assets and their respective locations in the asset tracking register database; and identify missing roadside assets… method may further comprise generating a map representation representing the roadside section and the locations of the identified roadside assets along the roadside section…The mobile unit may comprise at least one camera for capturing image data from the perspective of the mobile unit and the map representation may be configured for selectively displaying image data from the at least one camera at a plurality of locations along the roadside section… the asset data may comprise at least unique asset identifier data and asset location data, the database being operably coupled to the processor, wherein, in use, the processor may be controlled by the computer program code to receive, via the network interface, from a remote data capture device, asset data relating to an asset, the asset data may comprise at least location data representing the location of the asset… processor may be further controlled by the computer program code to capture the image data upon receipt of a trigger signal; Par. [0190]: the data capture device 105 captures the second rearward facing image in accordance with the speed of the maintenance vehicle 305, or the location of the maintenance vehicle 310 wherein the data capture device 105 captures the second rearward facing image in accordance with the location of the vehicle);
in response to receiving a proximity signal from a safety structure (Boyle, Par. [0039-86]: mobile unit may further comprise an asset proximity detector for detecting the proximity of roadside assets and the asset data capture device may be triggered by the asset proximity detector… processor may be further controlled by the computer program code to capture the image data upon receipt of a trigger signal; Par. [0249]: mobile unit asset data capture device 115 comprises a detector in the form of an asset proximity detector to detect the proximity of the roadside asset. The detector may trigger the asset data capture device 115 to capture an image of the roadside asset; ); and/or
in response to an on-demand command provided by a user (Boyle, Par. [0217-218]: portion comprises an image of the asset for recordal. As mentioned above, the image of the asset may be taken manually by the operator, or in an automated manner such as wherein the data capture device 105 utilises image recognition technique, proximity detection or the like…Once the image of the asset has been taken by the operator, or the asset identified by the data capture device 105, the interface 600 may allow the operator to confirm the asset including in inputting additional information relating to the asset).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 16, Bistry discloses a controller for a damage detection system (Par. [0020-26]: computing unit is further adapted to detect individual storage elements from the constructed three-dimensional data of the environment… the computer unit can detect any storage irregularities and/or damage to individual storage elements; Par. [0059-64]: computing unit 60 constructs three-dimensional data of the environment in the form of a point cloud. Through suitable image processing, the computing unit 60 can recognize various storage elements from the constructed point cloud and, based on these, determine three-dimensional data of the individual recognized storage elements. Based on a comparison of the three-dimensional data of individual detected storage elements with corresponding data from the warehouse management system and/or with predefined reference data, the computing unit 60 is able to automatically detect damage or irregularities in a warehouse. FIG. 4 and FIG. 5 each illustrate an example of this… FIG. 4 shows part of a rack 84 comprising a plurality of horizontal supports 86 and vertical supports 88. It can be seen in FIG. 4 that rack damage has occurred in the form of a deformed horizontal support 86. When the industrial truck 1 with the system 50 passes the affected storage area, the computing unit 60 is able to recognize, from the constructed point cloud and corresponding image processing algorithms, that the horizontal support 86 is deformed. Based on this, the computing unit 60 can output an instruction to have the horizontal beam 86 repaired or replaced). The steps further recited in claim 16 are rejected as applied to the apparatus claim 1 above.
Regarding claim 17, is a corresponding method claim rejected as applied to the apparatus claim 1 above.
Claims 9 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Bistry, in view of Boyle, as applied to claim 1, in further view of Shalev-Shwartz et al. (US PG Publication No. US 2019/0291728 A1), hereafter referred to as Shalev-Shwartz.
Regarding claim 9, claim 1 is incorporated and the combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), but fails to teach the following as further recited in claim 9.
However, Shalev-Shwartz teaches wherein the controller is configured to:
combine a plurality of images of the same safety structure into a combined-image (Par. [0154-198]: a first processing device may receive images from both the main camera and the narrow field of view camera, and perform vision processing of the narrow FOV camera to, for example, detect other vehicles, pedestrians, lane marks, traffic signs, traffic lights, and other road objects. Further, the first processing device may calculate a disparity of pixels between the images from the main camera and the narrow camera and create a 3D reconstruction of the environment of vehicle 200. The first processing device may then combine the 3D reconstruction with 3D map data or with 3D information calculated based on information from another camera… processing unit 110 may select information derived from two of the first, second, and third plurality of images by determining the extent to which information derived from one image source is consistent with information derived from other image sources. For example, processing unit 110 may combine the processed information derived from each of image capture devices 122, 124, and 126 (whether by monocular analysis, stereo analysis, or any combination of the two) and determine visual indicators (e.g., lane markings, a detected vehicle and its location and/or path, a detected traffic light, etc.) that are consistent across the images captured from each of image capture devices); and
compare the combined-image with the other image of the same, or a corresponding, safety structure (Par. [0154-198]: a first processing device may receive images from both the main camera and the narrow field of view camera, and perform vision processing of the narrow FOV camera to, for example, detect other vehicles, pedestrians, lane marks, traffic signs, traffic lights, and other road objects. Further, the first processing device may calculate a disparity of pixels between the images from the main camera and the narrow camera and create a 3D reconstruction of the environment of vehicle 200. The first processing device may then combine the 3D reconstruction with 3D map data or with 3D information calculated based on information from another camera… Processing unit 110 may execute monocular image analysis module 402 to implement process 500B. At step 540, processing unit 110 may determine a set of candidate objects representing possible vehicles and/or pedestrians. For example, processing unit 110 may scan one or more images, compare the images to one or more predetermined patterns, and identify within each image possible locations that may contain objects of interest (e.g., vehicles, pedestrians, or portions thereof)… processing unit 110 may select information derived from two of the first, second, and third plurality of images by determining the extent to which information derived from one image source is consistent with information derived from other image sources. For example, processing unit 110 may combine the processed information derived from each of image capture devices 122, 124, and 126 (whether by monocular analysis, stereo analysis, or any combination of the two) and determine visual indicators (e.g., lane markings, a detected vehicle and its location and/or path, a detected traffic light, etc.) that are consistent across the images captured from each of image capture devices).
Bistry, Boyle, and Shalev-Shwartz are considered to be analogous art because they pertain to image processing applications. Therefore, the combined teachings of Bistry, Boyle, and Shalev-Shwartz, as a whole, would have rendered obvious the invention recited in claim 9 with a reasonable expectation of success in order to modify the apparatus and method for inspecting and automatically detecting damage or irregularities in a warehouse (as disclosed by Bistry) with wherein the controller is configured to: combine a plurality of images of the same safety structure into a combined-image and compare the combined-image with the other image of the same, or a corresponding, safety structure (as taught by Shalev-Shwartz, Abstract, Par. [0154-198]) to detect a target vehicle in the environment of a host vehicle, to detect other vehicles, pedestrians, lane marks, traffic signs, traffic lights, and other road objects, and to detect various road hazards (Shalev-Shwartz, Abstract, Par. [0002-10, 154-198]).
Regarding claim 10, claim 9 and the combination of Bistry, Boyle, and Shalev-Shwartz, as a whole, teaches the system (Bistry, Par. [0001]), wherein the controller is configured to:
combine the plurality of images of the same safety structure into a 3-dimensional combined-image (Shalev-Shwartz, Par. [0154-198]: a first processing device may receive images from both the main camera and the narrow field of view camera, and perform vision processing of the narrow FOV camera to, for example, detect other vehicles, pedestrians, lane marks, traffic signs, traffic lights, and other road objects. Further, the first processing device may calculate a disparity of pixels between the images from the main camera and the narrow camera and create a 3D reconstruction of the environment of vehicle 200. The first processing device may then combine the 3D reconstruction with 3D map data or with 3D information calculated based on information from another camera… processing unit 110 may select information derived from two of the first, second, and third plurality of images by determining the extent to which information derived from one image source is consistent with information derived from other image sources. For example, processing unit 110 may combine the processed information derived from each of image capture devices 122, 124, and 126 (whether by monocular analysis, stereo analysis, or any combination of the two) and determine visual indicators (e.g., lane markings, a detected vehicle and its location and/or path, a detected traffic light, etc.) that are consistent across the images captured from each of image capture devices).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 9.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Bistry, in view of Boyle, as applied to claim 1, in further view of Kumar et al. (US PG Publication No. US 2019/0176862 A1), hereafter referred to as Kumar.
Regarding claim 15, claim 1 and the combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), wherein the controller is configured to:
compare a colour of the recognised safety structure in the acquired image with a colour of the safety structure in the other image of the same, or a corresponding, safety structure (Boyle, Par. [0254-268]: method may further comprise image comparison for detecting roadside asset damage or degradation to identify roadside assets requiring maintenance. For example, the method may compare asset image data against previously recorded asset image data and detect changes… For those assets that show a change in colour or shape exceeding a threshold may trigger the system to update the database 135 with a flag… the method may utilise an asset condition estimate image recognition technique for estimating a condition of each of the roadside assets. For example, referring to the exemplary interface 700 as can be seen, asset A4262, the condition of the asset has been deemed to be excellent. The asset condition estimate image recognition technique to take into account various factors in determining the condition, such as colour, shape and the like); and
based on the comparison, provide a damage-status-signal (Boyle, Par. [0254-255]: method may further comprise image comparison for detecting roadside asset damage or degradation to identify roadside assets requiring maintenance. For example, the method may compare asset image data against previously recorded asset image data and detect changes… For those assets that show a change in colour or shape exceeding a threshold may trigger the system to update the database 135 with a flag indicating that a particular asset requires maintenance. For example, should a roadside be damaged… the image comparison may detect the colour degradation and flag the asset for maintenance).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
The combination of Bistry and Boyle, as a whole, teaches the system (Bistry, Par. [0001]), as indicated above, but fails to teach the following as further recited in claim 15.
However, Kumar teaches signal that represents whether or not the safety structure is rusted (Par. [0345]: controller 2902 can direct the output device 2912 to display the current state of a route… depiction may be of the inspected object, whether a point on a road, a section of track, or a bridge or other infrastructure. That depiction may be a two-dimensional image, or may be three-dimensional rendered representation based in part on images taken from the mobile platform or from another data acquisition platform. Areas of interest (such as failure points, wear, fatigue, rust, cracks, and the like) may be highlighted on the depiction of the object (e.g., the three-dimensional rendering). These highlights may be coded by color or another symbol. For example, cracks may be flashing red while rust spots have arrows pointing to them).
Bistry, Boyle, and Kumar are considered to be analogous art because they pertain to image processing applications. Therefore, the combined teachings of Bistry, Boyle, and Kumar, as a whole, would have rendered obvious the invention recited in claim 15 with a reasonable expectation of success in order to modify the apparatus and method for inspecting and automatically detecting damage or irregularities in a warehouse (as disclosed by Bistry) with signal that represents whether or not the safety structure is rusted (as taught by Kumar, Abstract, Par. [0345]) to detect hazards, to detect obstacles on the route, problems with the route, or other hazards, to detect damage to the route, to detect hazards ahead of the vehicle, such as obstacles in front of the vehicle along the route, detect damaged segments of the route, and to highlight a depiction of areas of interest of an object, such as failure points, wear, fatigue, rust, cracks, etc. (Kumar, Abstract, Par. [0015-18, 140, 145, 184, 188-189, 345]), for example).
Allowable Subject Matter
Claim 11 remains objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims, as previously indicated in Pgs. 56-57 of the Non-Final Office action (OA) of September 23, 2026.
Conclusion
THIS ACTION IS MADE FINAL.
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GUILLERMO RIVERA-MARTINEZ whose telephone number is 571-272-4979. The examiner can normally be reached on Monday-Friday (8am - 5pm Eastern Time). If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached on 571-270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GUILLERMO M RIVERA-MARTINEZ/ Primary Examiner, Art Unit 2677
Prosecution Insights