DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1,2,6,9,10,11,15,18, and 19 are rejected under 35 U.S.C. 103(a) as being unpatentable over Biancale (US 20220004761 A1) (hereinafter Biancale) in view of Tran (US 20230319140 A1) (hereinafter Tran) in further view of Haskin (WO 2022074643 A1) (hereinafter Haskin).
Regarding claim 1, Biancale teaches a method for automated navigational marker detection and association in maritime applications(Biancale discloses detecting navigational markers such as navigation buoys. Biancale, paragraph 1, method for identifying an object at least partially immerged in water such as e.g. an ocean, a sea or a lake. Biancale, paragraph 60, the system according to the invention allows to detect objects. For example, the system may be configured to detect pieces of debris, containers, wrecks, boats, navigation buoys), comprising:
providing a camera system equipped with one or more red-green-blue (RGB) cameras for capturing visual data of a surrounding maritime environment in real-time(Biancale, paragraph 22, the capturing module comprises one RGB camera configured to capture images on a field of view of at least 90°, advantageously 160°, preferably 185°. This type of capturing module is inexpensive and hence particularly adapted for cruise sailing boats. Biancale, paragraph 85, each camera of the capturing module 310 generates a sequence of images of and sends said sequence to the processing module 320, preferably in real time);
providing a computing unit comprising a neural network-based object detector(Biancale, paragraph 8, artificial neural network being configured to detect at least one object ), and a GPS mapping and (Biancale, paragraph 27, By combining this relative position with GPS data, for example available on a boat navigation bus, it is then possible to assign an absolute position to each pixel) ;
capturing visual data of the surrounding maritime environment using the camera system(Biancale, paragraph 22, the capturing module comprises one RGB camera configured to capture images on a field of view of at least 90°, advantageously 160°, preferably 185°. This type of capturing module is inexpensive and hence particularly adapted for cruise sailing boats);
processing said visual data by the computing unit using the neural network-based object detector to identify navigational markers(Biancale discloses object detection using neural network. The object can be a navigational marker such as a navigation buoy. Biancale, paragraph 8, processing module being configured to receive at least one sequence of images from said at least one camera and comprising at least one artificial neural network, said at least one artificial neural network being configured to detect at least one object in said at least one received sequence of images. Biancale, paragraph 60, the system may be configured to detect pieces of debris, containers, wrecks, boats, navigation buoys, marine animals, ice blocks, whales, orcas or any type of object which are at least partially immerged in water);
integrating the projected positions of the detected navigational markers into a navigational map(Biancale, paragraph 78, The processing module 320 is configured to generate a 2D map indicating the detected and/or identified objects 5, preferably showing the distance D between the boat 1 and each detected object 5.); and
While Biancale teaches about object detection such as navigational markers using neural network, it fails to disclose a projection mechanism module, chart data integration module, providing a database comprising pre-existing chart data of navigational markers, projecting pixel positions of detected navigational markers into a three-dimensional (3D) coordinate system, cross-referencing the projected positions of the detected navigational markers with pre-existing chart data of navigational markers for the same location to enhance navigational accuracy and reliability.
However, Tran, which is in the same analogous art and that teaches about smart car teaches a projection mechanism module(Tran discloses identifying and projecting 3d points that belong to different identified objects such as street signs, obstacles, or bike lanes. Tran, paragraph 121, Project image points onto the 3D plane. Tran, paragraph 131, The method further includes projecting the identified portion of the image corresponding to the bike lane onto the fitted plane in the 3D map), projecting pixel positions of detected objects into a three-dimensional (3D) coordinate system(Tran teaches detecting objects such as traffic sign in an image by identifying its vertices and projecting it onto the 3D plane. Tran, paragraph 68, The computer vision system 746 may be any system configured to process and analyze images captured by the camera 734 in order to identify objects. Tran, paragraph 118, These 3D points are used to fit a plane, wherein the HD map projects the sign’s image vertices onto that 3D plane to find the 3D coordinates of the sign’s vertices. Tran, paragraph 121, The overall process performed by the HD map for detecting sign features comprises the following steps: (1.) Receive as input one or more images with labelled sign vertices (2.) Identify 3D points in the scene (3.) Identify the 3D points that belong to the sign (4.) Fit a plane to the 3D sign points (5.) Project image points onto the 3D plane. ).
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Biancale with Tran to project detected object onto a three-dimensional coordinate system. By projecting detected objects/ navigation markers onto the surface, it is possible to provide users a better understanding of the size and shape of a detected object in comparison to the real world coordinates. Furthermore, it reduces the risk of collision between a vehicle and navigational markers (or different objects) by allowing the user to maintain a safe distance from the observable projection of the objects.
While the combination of Biancale and Tran teaches about detection and projection of navigational markers, it fails to disclose chart data integration module, providing a database comprising pre-existing chart data of navigational markers, cross-referencing the projected positions of the detected navigational markers with pre-existing chart data of navigational markers for the same location to enhance navigational accuracy and reliability.
However, Haskin, which is in the same analogous art and that teaches about improving geo-registration accuracy by using neural networks for identifying dynamic objects, discloses chart data integration module(Haskin discloses a database table that is used to associate an identified object with a location. Database table is similar to how chart data organizes information for visualization and parsing. Haskin, page 137 line 26, the database 68b includes a table 129 that associates an identified object 127a to the respective geographical location 127b of the object.), providing a database comprising pre-existing chart data of objects(Haskin discloses a location of an object from a database that it used by the neural network object detection system as a reference. And discussed above, Biancale discusses a mechanism to identify these objects as navigational markers. Haskin, page 104 line 19, a database that may associate a geographical location with each of the objects in the group ), cross-referencing the projected positions of the detected navigational markers with pre-existing chart data of navigational markers for the same location to enhance navigational accuracy and reliability( Haskin, page 117 line 11, Geo- synchronization is shown in FIG. 6. A video data stream is received as part of a step “Receive Video” step 61, such as from the video camera 34, which is part of the quadcopter 30a or the fixed wing UAV 30b. Since the analysis is on frame- by-frame basis, a single frame is extracted from the received video stream as part of an “Extract Frame” step 62. An object is identified in the image of the extracted frame as part of an “Identify Object”…. the image identification in the “Identify Object” step 63 is based on machine learning or neural network, such as ANN. As part of an “Associate Location” step 64, the physical geographical location of the identified object is determined, for example by using the location associated with the image best compared with the captured one. Haskin, page 104 line 19, geo-synchronization algorithm herein may use a database that may associate a geographical location with each of the objects in the group, and may comprises identifying, an object from the group in the image of the frame by comparing to the database images )
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Biancale and Tran with Haskin to associate object detected by neural network with prestored data in a database. By incorporating prestored database with object detected by the neural network, it is possible to increase the accuracy of a detection system by comparing detected markers with verified navigational marker characteristics such as specific size and shape found in a database. Additionally, new accurately detected objects and their characteristics can be stored in the database to expand the database with new reference, further increasing accuracy of marker detection system.
Regarding claim 2, the combination of Biancale, Tran and Haskin teaches the method of claim 1(Biancale, paragraph 60, the system may be configured to detect pieces of debris, containers, wrecks, boats, navigation buoys; Tran, paragraph 121, Project image points onto the 3D plane; Haskin, page 104 line 19, geo-synchronization algorithm herein may use a database that may associate a geographical location with each of the objects in the group), wherein the projection mechanism module uses an inertial measurement unit (IMU)-based orientation estimation techniques to facilitate the projection of pixel positions(Biancale, paragraph 27, Using the horizon and the position of the camera through the inertial measurement module, it is possible to correlate each pixel with a position in the environment. Biancale, paragraph 14, the system comprises an inertial measurement module configured to provide spatial data, said spatial data comprising at least the inclination angle, relatively to the terrestrial horizontal axis, of the support on which the capturing module is mounted ) into the 3D coordinate system(Tran, paragraph 68, The computer vision system 746 may be any system configured to process and analyze images captured by the camera 734 in order to identify objects. Tran, paragraph 118, These 3D points are used to fit a plane, wherein the HD map projects the sign’s image vertices onto that 3D plane to find the 3D coordinates of the sign’s vertices. Tran, paragraph 121, The overall process performed by the HD map for detecting sign features comprises the following steps: (1.) Receive as input one or more images with labelled sign vertices (2.) Identify 3D points in the scene (3.) Identify the 3D points that belong to the sign (4.) Fit a plane to the 3D sign points (5.) Project image points onto the 3D plane).
Regarding claim 6, the combination of Biancale and Haskin teaches the method of claim 1(Biancale, paragraph 60, the system may be configured to detect pieces of debris, containers, wrecks, boats, navigation buoys; Tran, paragraph 121, Project image points onto the 3D plane; Haskin, page 104 line 19, geo-synchronization algorithm herein may use a database that may associate a geographical location with each of the objects in the group), wherein the neural network-based object detector is trained to identify navigational markers based on characteristic shapes, colors, and patterns (Biancale discusses different classical visual algorithms applying few filters(color, shape, texture), and its system adding more filter. The addition of more filter implies its use of color, shape, texture to identify objects. Furthermore, texture of Biancale is similar to the pattern. Biancale, paragraph 77, Unlike the classical visual algorithms consisting in applying a few filters using mathematical convolution to the image, extracting minor features from the image (color, shape, texture) and analyzing the results, a CNN (convolutional neural network) stacks a big number of filters into a convolutional layer to detect as many features as possible, and applies more convolutional layers to these features in order to extract much deeper (high-level) features in the image ).
Regarding claim 9, the combination of Biancale, Tran, and Haskin teaches the method of claim 1(Biancale, paragraph 60, the system may be configured to detect pieces of debris, containers, wrecks, boats, navigation buoys; Tran, paragraph 121, Project image points onto the 3D plane; Haskin, page 104 line 19, geo-synchronization algorithm herein may use a database that may associate a geographical location with each of the objects in the group;),
wherein the GPS mapping and chart data integration module updates a navigational map in real-time to reflect the positions of detected navigational markers(Tran disclose detecting objects (as discussed above, Biancale discloses the determination of objects whether they are navigational markers), and updating the map based on the detected objects. Tran, paragraph 178, the new information is used to update a digital map that lacks the current information or that contains inaccuracies or may be incomplete. The digital map stored in the map database may be updated using the information processed by a map matching module, matched segment module, and unmatched segment module. The map matching module, once it has received obstacle location and GPS traces, processes obstacle locations and GPS traces by matching them to a road defined in the digital map. The map matching module matches the obstacles and the GPS traces with the most likely road positions corresponding to a viable route through the digital map by using the processor to execute a matching algorithm), and enhances the visual representation of said navigational map based on the association with the pre-existing chart data(Tran discloses updating inaccurate old map database with new information which is similar to enhancing of the chart data that is stored in the database. Tran, paragraph 178, The new information is used to update a digital map that lacks the current information or that contains inaccuracies or may be incomplete. The digital map stored in the map database may be updated using the information processed by a map matching module, matched segment module, and unmatched segment module.).
Regarding claim 10, Biancale teaches an automated navigational marker detection and association system for maritime applications(Biancale discloses detecting navigational markers such as navigation buoys. Biancale, paragraph 60, the system according to the invention allows to detect objects. For example, the system may be configured to detect pieces of debris, containers, wrecks, boats, navigation buoys), comprising:
a camera system equipped with one or more red-green-blue (RGB) cameras for capturing visual data of the surrounding maritime environment in real-time(Biancale, paragraph 22, the capturing module comprises one RGB camera configured to capture images on a field of view of at least 90°, advantageously 160°, preferably 185°. This type of capturing module is inexpensive and hence particularly adapted for cruise sailing boats. Biancale, paragraph 85, each camera of the capturing module 310 generates a sequence of images of and sends said sequence to the processing module 320, preferably in real time);
a computing unit comprising a neural network-based object detector(Biancale, paragraph 8, artificial neural network being configured to detect at least one object), a database comprising data of navigational markers(Biancale’s object can be navigation markers such as navigation buoys. Biancale, paragraph 87, the processing module 320 detects the objects 5 in each segmented image and identify the objects 5 when said objects are known from its database using the segmented compensated image. Biancale, paragraph 60, the system according to the invention allows to detect objects. For example, the system may be configured to detect pieces of debris, containers, wrecks, boats, navigation buoys.),and a GPS mapping and (Biancale, paragraph 27, By combining this relative position with GPS data, for example available on a boat navigation bus, it is then possible to assign an absolute position to each pixel)
wherein the neural network-based object detector is configured to process said visual data to identify navigational markers(Biancale discloses object detection using neural network. The object can be a navigational marker such as navigation buoy. Biancale, paragraph 8, processing module being configured to receive at least one sequence of images from said at least one camera and comprising at least one artificial neural network, said at least one artificial neural network being configured to detect at least one object in said at least one received sequence of images. Biancale, paragraph 60, the system may be configured to detect pieces of debris, containers, wrecks, boats, navigation buoys, marine animals, ice blocks, whales, orcas or any type of object which are at least partially immerged in water); and
wherein the GPS mapping is configured to integrate the projected positions of the detected navigational markers into a navigational map and(Biancale, paragraph 27, By combining this relative position with GPS data, for example available on a boat navigation bus, it is then possible to assign an absolute position to each pixel. Biancale, paragraph 78, The processing module 320 is configured to generate a 2D map indicating the detected and/or identified objects 5, preferably showing the distance D between the boat 1 and each detected object 5).
While Biancale teaches about object detection such as navigational markers using neural network, it fails to disclose a projection mechanism module, chart data integration module, wherein the projection mechanism module is configured to project pixel positions of detected objects into a three-dimensional (3D) coordinate system by extending rays from said pixel positions to a water surface thereby generating projected positions of detected objects; chart data integration module is configured to integrate the projected positions of the detected navigational markers into a navigational map, to cross-reference and compare said projected positions of the detected navigational markers with pre-existing chart data of navigational markers stored in the database for the same location
However, Tran, which is in the same analogous art and that teaches about smart car teaches a projection mechanism module(Tran discloses identifying and projecting 3d points that belong to different identified objects such as street signs, obstacles, or bike lanes. Tran, paragraph 121, Project image points onto the 3D plane. Tran, paragraph 131, The method further includes projecting the identified portion of the image corresponding to the bike lane onto the fitted plane in the 3D map), wherein the projection mechanism module is configured to project pixel positions of detected objects into a three-dimensional (3D) coordinate system(Tran teaches detecting objects such as traffic sign in an image by identifying its vertices and projecting it onto the 3D plane. Tran, paragraph 68, The computer vision system 746 may be any system configured to process and analyze images captured by the camera 734 in order to identify objects. Tran, paragraph 118, These 3D points are used to fit a plane, wherein the HD map projects the sign’s image vertices onto that 3D plane to find the 3D coordinates of the sign’s vertices. Tran, paragraph 121, The overall process performed by the HD map for detecting sign features comprises the following steps: (1.) Receive as input one or more images with labelled sign vertices (2.) Identify 3D points in the scene (3.) Identify the 3D points that belong to the sign (4.) Fit a plane to the 3D sign points (5.) Project image points onto the 3D plane) by extending rays from said pixel positions to a water surface thereby generating projected positions of detected objects(Tran discloses the projection of detected pixel position on a land rather than water. However, it would obvious to one of ordinary skill in the art to implement Tran’s projection of pixel position on water. Tran, paragraph 131, the portion of the image corresponding to the bike lane can be projected through drawing a ray from the detection and ranging sensor through each individual pixel to determine an intersection of the ray with the fitted plane. The intersection is the projected position of that pixel.);
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Biancale with Tran to project detected object onto a three-dimensional coordinate system by drawing a ray from the ranging sensor. By projecting detected objects/ navigation markers onto the surface, it is possible to provide users a better understanding of the size and shape of a detected object in comparison to the real world coordinates. Furthermore, it reduces the risk of collision between a vehicle and navigational markers (or different objects) by allowing the user to maintain a safe distance from the observable projection of the objects.
While the combination of Biancale and Tran teaches about detection and projection of navigational markers, it fails to disclose chart data integration module, chart data integration module is configured to integrate the projected positions of the detected navigational markers into a navigational map, to cross-reference and compare said projected positions of the detected navigational markers with pre-existing chart data of navigational markers stored in the database for the same location
However, Haskin, which is in the same analogous art and that teaches about improving geo-registration accuracy by using neural networks for identifying dynamic objects, discloses chart data integration module(Biancale teaches database with navigation markers, but fails to disclose chart data. Haskin discloses a database table that is used to associate an identified object with location. Database table is similar to how chart data organizes information for visualization and parsing. Haskin, page 137 line 26, the database 68b includes a table 129 that associates an identified object 127a to the respective geographical location 127b of the object ), chart data integration module is configured to integrate the projected positions of the detected navigational markers into a navigational map(Haskin, page 137 line 26, the database 68b includes a table 129 that associates an identified object 127a to the respective geographical location 127b of the object. Haskin, page 151 line 10, In addition to predicting the presence of an object within the region proposals, the algorithm also predicts four values which are offset values to increase the precision of the bounding box. The R-CNN may be a Fast R-CNN, where the input image is fed to the CNN to generate a convolutional feature map),and to cross-reference and compare said projected positions of the detected navigational markers with pre-existing chart data of navigational markers stored in the database for the same location(Haskin, page 117 line 11, Geo- synchronization is shown in FIG. 6. A video data stream is received as part of a step “Receive Video” step 61, such as from the video camera 34, which is part of the quadcopter 30a or the fixed wing UAV 30b. Since the analysis is on frame- by-frame basis, a single frame is extracted from the received video stream as part of an “Extract Frame” step 62. An object is identified in the image of the extracted frame as part of an “Identify Object”…. the image identification in the “Identify Object” step 63 is based on machine learning or neural network, such as ANN. As part of an “Associate Location” step 64, the physical geographical location of the identified object is determined, for example by using the location associated with the image best compared with the captured one. Haskin, page 104 line 19, geo-synchronization algorithm herein may use a database that may associate a geographical location with each of the objects in the group, and may comprises identifying, an object from the group in the image of the frame by comparing to the database images).
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Biancale and Tran with Haskin to associate object detected by neural network with prestored data in a database. By incorporating prestored database with object detected by the neural network, it is possible to increase the accuracy of a detection system by comparing detected markers with verified navigational marker characteristics such as specific size and shape found in a database. Additionally, new accurately detected objects and their characteristics can be stored in the database to expand the database with new reference, further increasing accuracy of marker detection system.
Regarding claim 11, the combination of Biancale, Tran and Haskin teaches the system of claim 10(Biancale, paragraph 60, the system may be configured to detect pieces of debris, containers, wrecks, boats, navigation buoys; Tran, paragraph 121, Project image points onto the 3D plane; Haskin, page 104 line 19, geo-synchronization algorithm herein may use a database that may associate a geographical location with each of the objects in the group), wherein the projection mechanism uses an Inertial Measurement Unit (IMU)-based orientation estimation techniques to facilitate the projection of pixel positions(Biancale, paragraph 27, Using the horizon and the position of the camera through the inertial measurement module, it is possible to correlate each pixel with a position in the environment ) into the 3D coordinate system(As discusses above, Biancale discloses displaying detected object in a 3D map, that indicates the system’s capability to integrate 3D coordinates system. Biancale, paragraph 95, the processing module 320 may display a 2D or a 3D map indicating the detected and/or identified objects 5).
Regarding claim 15, the combination of Biancale and Haskin teaches the system of claim 10(Biancale, paragraph 60, the system may be configured to detect pieces of debris, containers, wrecks, boats, navigation buoys; Tran, paragraph 121, Project image points onto the 3D plane; Haskin, page 104 line 19, geo-synchronization algorithm herein may use a database that may associate a geographical location with each of the objects in the group), wherein the neural network-based object detector is trained to identify navigational markers based on characteristic shapes, colors, and patterns(Biancale discusses different classical visual algorithms applying few filters(color, shape, texture), and its system adding more filter. The addition of more filter implies its use of color, shape, texture to identify objects. Furthermore, texture of Biancale is similar to the pattern. Biancale, paragraph 77, Unlike the classical visual algorithms consisting in applying a few filters using mathematical convolution to the image, extracting minor features from the image (color, shape, texture) and analyzing the results, a CNN (convolutional neural network) stacks a big number of filters into a convolutional layer to detect as many features as possible, and applies more convolutional layers to these features in order to extract much deeper (high-level) features in the image).
Regarding claim 18, the combination of Biancale, Tran, and Haskin teaches the system of claim 10(Biancale, paragraph 60, the system may be configured to detect pieces of debris, containers, wrecks, boats, navigation buoys; Tran, paragraph 121, Project image points onto the 3D plane; Haskin, page 104 line 19, geo-synchronization algorithm herein may use a database that may associate a geographical location with each of the objects in the group;), wherein the GPS mapping and chart data integration module updates a navigational map in real-time to reflect the positions of detected navigational markers(Tran disclose detecting objects (as discussed above, Biancale discloses the determination of objects whether they are navigational markers), and updating the map based on the detected objects. Tran, paragraph 178, the new information is used to update a digital map that lacks the current information or that contains inaccuracies or may be incomplete. The digital map stored in the map database may be updated using the information processed by a map matching module, matched segment module, and unmatched segment module. The map matching module, once it has received obstacle location and GPS traces, processes obstacle locations and GPS traces by matching them to a road defined in the digital map. The map matching module matches the obstacles and the GPS traces with the most likely road positions corresponding to a viable route through the digital map by using the processor to execute a matching algorithm), and enhances the visual representation of said navigational map based on the association with pre-existing chart data(Tran discloses updating inaccurate old map database with new information which is similar to enhancing of the chart data that is stored in the database. Tran, paragraph 178, The new information is used to update a digital map that lacks the current information or that contains inaccuracies or may be incomplete. The digital map stored in the map database may be updated using the information processed by a map matching module, matched segment module, and unmatched segment module.).
Regarding claim 19. The system of claim 10, wherein said system is configured to operate on commercial vessels, fishing boats, recreational boats, and sailing yachts(Biancale discloses its system can be implemented for sailing boats and Yachts, which implies it can also be configured for commercial vessels and fishing boats. Biancale, paragraph 43,this configuration is particularly adapted for boats that navigate fast and indistinctively both at day time and at night time, in particular for race sailing boats and yachts).
Claims 3 and 12 are rejected under 35 U.S.C. 103(a) as being unpatentable over Biancale (US 20220004761 A1) (hereinafter Biancale) in view of Tran (US 20230319140 A1) (hereinafter Tran) in further view of Haskin (WO 2022074643 A1) (hereinafter Haskin) in further view of Liu (CN 116264034 A) (hereinafter Liu).
Regarding claim 3, the combination of Biancale, Tran, and Haskin teaches the method of claim 1(Biancale, paragraph 60, the system may be configured to detect pieces of debris, containers, wrecks, boats, navigation buoys; Tran, paragraph 121, Project image points onto the 3D plane; Haskin, page 104 line 19, geo-synchronization algorithm herein may use a database that may associate a geographical location with each of the objects in the group),facilitate the projection of pixel positions into the 3D coordinate system(Tran teaches detecting objects such as traffic sign in an image by identifying its vertices and projecting it onto the 3D plane. Tran, paragraph 68, The computer vision system 746 may be any system configured to process and analyze images captured by the camera 734 in order to identify objects. Tran, paragraph 118, These 3D points are used to fit a plane, wherein the HD map projects the sign’s image vertices onto that 3D plane to find the 3D coordinates of the sign’s vertices. Tran, paragraph 121, The overall process performed by the HD map for detecting sign features comprises the following steps: (1.) Receive as input one or more images with labelled sign vertices (2.) Identify 3D points in the scene (3.) Identify the 3D points that belong to the sign (4.) Fit a plane to the 3D sign points (5.) Project image points onto the 3D plane).
While the combination of Biancale, Tran, and Haskin teaches about detection and projection of navigational markers, it fails to disclose a method wherein the projection mechanism module uses a Computer Vision (CV)-based orientation estimation techniques.
However, Liu, which is in the same analogous art and that teaches about a dangerous road identification method discloses a method wherein the projection mechanism module uses a Computer Vision (CV)-based orientation estimation techniques(Liu, paragraph 4, A target identification based on the computer vision technology is to identify what objects in the image and report the position and orientation of the object in the scene of the image representation)
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Biancale, Tran, and Haskin with Liu to estimate the orientation of an object using computer vision. By using computer vision, it is possible to determine orientation of plurality of composite shaped navigational markers that might be difficult for other orientation sensors such as gyroscope. Additionally, computer vision helps determine the orientation of a navigation marker from a distance without requiring a sensor being mounted on the navigation marker.
Regarding claim 12, the combination of Biancale, Tran and Haskin teaches the system of claim 10(Biancale, paragraph 60, the system may be configured to detect pieces of debris, containers, wrecks, boats, navigation buoys; Tran, paragraph 121, Project image points onto the 3D plane; Haskin, page 104 line 19, geo-synchronization algorithm herein may use a database that may associate a geographical location with each of the objects in the group), facilitate the projection of pixel positions into the 3D coordinate system(Tran teaches detecting objects such as traffic sign in an image by identifying its vertices and projecting it onto the 3D plane. Tran, paragraph 68, The computer vision system 746 may be any system configured to process and analyze images captured by the camera 734 in order to identify objects. Tran, paragraph 118, These 3D points are used to fit a plane, wherein the HD map projects the sign’s image vertices onto that 3D plane to find the 3D coordinates of the sign’s vertices. Tran, paragraph 121, The overall process performed by the HD map for detecting sign features comprises the following steps: (1.) Receive as input one or more images with labelled sign vertices (2.) Identify 3D points in the scene (3.) Identify the 3D points that belong to the sign (4.) Fit a plane to the 3D sign points (5.) Project image points onto the 3D plane).
While the combination of Biancale, Tran, and Haskin teaches about detection and projection of navigational markers, it fails to disclose a system wherein the projection mechanism uses a Computer Vision (CV)-based orientation estimation techniques.
However, Liu, which is in the same analogous art and that teaches about a dangerous road identification method discloses a system wherein the projection mechanism uses a Computer Vision (CV)-based orientation estimation techniques(Liu, paragraph 4, A target identification based on the computer vision technology is to identify what objects in the image and report the position and orientation of the object in the scene of the image representation).
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Biancale, Tran, and Haskin with Liu to estimate the orientation of an object using computer vision. By using computer vision, it is possible to determine orientation of plurality of composite shaped navigational markers that might be difficult for other orientation sensors such as gyroscope. Additionally, computer vision helps determine the orientation of a navigation marker from a distance without requiring a sensor being mounted on the navigation marker.
Claims 4 and 13 are rejected under 35 U.S.C. 103(a) as being unpatentable over Biancale (US 20220004761 A1) (hereinafter Biancale) in view of Tran (US 20230319140 A1) (hereinafter Tran) in further view of Haskin (WO 2022074643 A1) (hereinafter Haskin) in further view of Matsue (JP 2020071046 A) (hereinafter Matsue).
Regarding claim 4, the combination of Biancale, Tran, and Haskin teaches the method of claim 1(Biancale, paragraph 60, the system may be configured to detect pieces of debris, containers, wrecks, boats, navigation buoys; Tran, paragraph 121, Project image points onto the 3D plane; Haskin, page 104 line 19, geo-synchronization algorithm herein may use a database that may associate a geographical location with each of the objects in the group).
While the combination of Biancale, Tran, and Haskin teaches about detection and projection of navigational markers, it specifically fails to disclose a method further comprising using a local-greedy association strategy for aligning the detected navigational markers positions with the pre-existing chart data based on proximity.
However, Matsue which is in the same analogous art and that teaches about guidance terminal device that use position information discloses a method further comprising using a local-greedy association strategy for aligning the detected navigational markers positions with the pre-existing chart data based on proximity(Matsue discloses using nearest neighbor algorithm, which is type of a greedy algorithm, to match with the closest position of guidance object found in the database. Matsue, page 6 line 18, the target object detection unit 213 executes the nearest neighbor target object detection process (FIG. 17 described later) for identifying the guidance target object with reference to the target object database 240. Matsue, page 15 line 8, In step S609, the object detection unit 213 updates the nearest object to the object.).
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Biancale, Tran, and Haskin with Matsue to identify the closest guidance object, similar to a closest position of navigational marker, found in the reference database. By implementing the nearest neighbor algorithm/ greedy algorithm, it is possible to detect a navigation marker’s location or type faster because of the it selects the immediate locally optimal choice at each step. Furthermore, greedy algorithm helps with memory optimization as it takes a single route to make a selection/association instead of exploring different routes, taking less memory.
Regarding claim 13, the combination of Biancale, Tran, and Haskin teaches the system of claim 10(Biancale, paragraph 60, the system may be configured to detect pieces of debris, containers, wrecks, boats, navigation buoys; Tran, paragraph 121, Project image points onto the 3D plane; Haskin, page 104 line 19, geo-synchronization algorithm herein may use a database that may associate a geographical location with each of the objects in the group),
While the combination of Biancale, Tran, and Haskin teaches about detection and projection of navigational markers, it specifically fails to disclose a system further comprising a local-greedy association strategy for aligning the detected navigational markers positions with pre-existing chart data based on proximity.
However, Matsue which is in the same analogous art and that teaches about guidance terminal device that use position information discloses a system further comprising a local-greedy association strategy for aligning the detected navigational markers positions with pre-existing chart data based on proximity(Matsue discloses using nearest neighbor algorithm, which is type of a greedy algorithm, to match with the closest position of guidance object found in the database. Matsue, page 6 line 18, the target object detection unit 213 executes the nearest neighbor target object detection process (FIG. 17 described later) for identifying the guidance target object with reference to the target object database 240. Matsue, page 15 line 8, In step S609, the object detection unit 213 updates the nearest object to the object).
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Biancale, Tran, and Haskin with Matsue to identify the closest guidance object, similar to a closest position of navigational marker, found in the reference database. By implementing the nearest neighbor algorithm/ greedy algorithm, it is possible to detect a navigation marker’s location or type faster because of the it selects the immediate locally optimal choice at each step. Furthermore, greedy algorithm helps with memory optimization as it takes a single route to make a selection/association instead of exploring different routes, taking less memory.
Claims 5 and 14 are rejected under 35 U.S.C. 103(a) as being unpatentable over Biancale (US 20220004761 A1) (hereinafter Biancale) in view of Tran (US 20230319140 A1) (hereinafter Tran) in further view of Haskin (WO 2022074643 A1) (hereinafter Haskin) in further view of Zhang (CN 116380070 A) (hereinafter Zhang).
Regarding claim 5, the combination of Biancale, Tran, and Haskin teaches the method of claim 1(Biancale, paragraph 60, the system may be configured to detect pieces of debris, containers, wrecks, boats, navigation buoys; Tran, paragraph 121, Project image points onto the 3D plane; Haskin, page 104 line 19, geo-synchronization algorithm herein may use a database that may associate a geographical location with each of the objects in the group).
The combination of Biancale, Tran and Haskin fails to teach a method further comprising using a global-optimal association strategy that utilizes an optimization algorithm to minimize the summed distances between the detected navigational markers positions and the pre-existing chart data for the navigational markers.
However, Zhang, which is in the same analogous art and that teaches about visual inertia positioning method of a robot discloses further comprising using a global-optimal association strategy that utilizes an optimization algorithm to minimize the summed distances between the detected navigational markers positions and the pre-existing chart data for the navigational markers(Zhang discloses comparing current image frames with image frames in a database to find a candidate frame that matches the characteristics of the current image frames, and performing global optimization to reduce accumulated error in the matching process. Zhang, paragraph 49, after each sliding window optimization is finished, comparing the current frame with the nearest key frame as parallax, when the two frame parallax is greater than a certain threshold, judging the current frame as the key frame. after determining the key frame, comparing it with the key frame database to find candidate loop frame. if finding the candidate loop frame, matching the characteristic point, then removing the geometric abnormal value of the matched characteristic point, after eliminating the inner point is still satisfy threshold value, then judging as the correct loop frame. after finding the loop frame, it can establish loop constraint, performing global optimization, so as to reduce the accumulated error of the system).
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Biancale, Tran, and Haskin with Zhang to compare detected image frame with a frame from a database and utilize global optimization to reduce accumulated error of the system. By implementing global optimization, it is possible to select the best possible frame/object from all available database instead of the immediate local frame. This gives comprehensive prediction factoring in the total available data in the database. Global optimization provides more accurate navigation marker association than greedy local optimization.
Regarding claim 14, the combination of Biancale, Tran, and Haskin teaches the system of claim 10(Biancale, paragraph 60, the system may be configured to detect pieces of debris, containers, wrecks, boats, navigation buoys; Tran, paragraph 121, Project image points onto the 3D plane; Haskin, page 104 line 19, geo-synchronization algorithm herein may use a database that may associate a geographical location with each of the objects in the group),
The combination of Biancale, Tran and Haskin fails to teach a system further comprising a global-optimal association strategy that utilizes an optimization algorithm to minimize the summed distances between the detected navigational markers positions and pre-existing chart data for the navigational markers.
However, Zhang, which is in the same analogous art and that teaches about visual inertia positioning method of a robot discloses a system further comprising a global-optimal association strategy that utilizes an optimization algorithm to minimize the summed distances between the detected navigational markers positions and pre-existing chart data for the navigational markers(Zhang discloses comparing current image frames with image frames in a database to find a candidate frame that matches the characteristics of the current image frames, and performing global optimization to reduce accumulated error in the matching process. Zhang, paragraph 49, after each sliding window optimization is finished, comparing the current frame with the nearest key frame as parallax, when the two frame parallax is greater than a certain threshold, judging the current frame as the key frame. after determining the key frame, comparing it with the key frame database to find candidate loop frame. if finding the candidate loop frame, matching the characteristic point, then removing the geometric abnormal value of the matched characteristic point, after eliminating the inner point is still satisfy threshold value, then judging as the correct loop frame. after finding the loop frame, it can establish loop constraint, performing global optimization, so as to reduce the accumulated error of the system).
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Biancale, Tran, and Haskin with Zhang to compare detected image frame with a frame from a database and utilize global optimization to reduce accumulated error of the system. By implementing global optimization, it is possible to select the best possible frame/object from all available database instead of the immediate local frame. This gives comprehensive prediction factoring in the total available data in the database. Global optimization provides more accurate navigation marker association than greedy local optimization.
Claims 7, 8, 18, and 17 are rejected under 35 U.S.C. 103(a) as being unpatentable over Biancale (US 20220004761 A1) (hereinafter Biancale) in view of Tran (US 20230319140 A1) (hereinafter Tran) in further view of Haskin (WO 2022074643 A1) (hereinafter Haskin) in further view of Nowicka (US 20230023434 A1) (hereinafter Nowicka).
Regarding claim 7, the combination of Biancale, Tran, and Haskin teaches the method of claim 1(Biancale, paragraph 60, the system may be configured to detect pieces of debris, containers, wrecks, boats, navigation buoys; Tran, paragraph 121, Project image points onto the 3D plane; Haskin, page 104 line 19, geo-synchronization algorithm herein may use a database that may associate a geographical location with each of the objects in the group),
While the combination of Biancale, Tran, and Haskin teaches detection of objects from an image using neural network, and providing bounding box, it fails to disclose a method wherein the neural network-based object detector provides bounding boxes and confidence scores for each detected navigational marker.
However, Nowicka, which is in the same analogous art and that teaches about marine object classification disclose a method wherein the neural network-based object detector provides bounding boxes (Nowicka, paragraph 83, Object are identified by bounding boxes and labels such as: (i) bounding box 212a labeled as a nearby buoy 212b. Nowicka, paragraph 84, FIG. 3, which shows (i) an image 302 with a bounding box for object detection ), and confidence scores for each detected navigational marker(Nowicka, paragraph 10, The method may further comprise identifying a detected object using a convolutional neural network trained to identify the detected object and generated an associated confidence value. Nowicka, paragraph 49, the systems and methods of the present disclosure may be implemented to detect and identify a variety of objects including vessels (e.g., sailboats, powerboats), buoys, floating debris, person(s) in the water (e.g., person overboard), and other objects.).
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Biancale, Tran, and Haskin with Nowicka to provide bounding box, and determine the confidence score of a neural network based navigational marker detection system. By evaluating the confidence score of the navigational marker classifier, it is possible to determine if classification should be accepted or rejected based on the accuracy score of the classification. Furthermore, it is possible to do further classification using different neural network algorithm for classification with low accuracy score.
Regarding claim 8, the combination of Biancale, Tran, Haskin, and Nowicka teaches the method of claim 7(Biancale, paragraph 60, the system may be configured to detect pieces of debris, containers, wrecks, boats, navigation buoys; Tran, paragraph 121, Project image points onto the 3D plane; Haskin, page 104 line 19, geo-synchronization algorithm herein may use a database that may associate a geographical location with each of the objects in the group; Nowicka, paragraph 10, The method may further comprise identifying a detected object using a convolutional neural network trained to identify the detected object and generated an associated confidence value), wherein the neural network-based object detector further classifies the type of the detected navigational marker(Biancale, paragraph 9, An artificial neural network is a straight, easy and efficient way to process images generated by the at least one camera in order to detect, identify and track objects at least partially immerged in water. Predetermined set of features may be materialized into classes associated with known objects, for example boats, buoys, whales, containers. Nowicka, paragraph 97, A network may be pretrained on/using a marine dataset identifying boats and then fine-tuned on a more robust marine dataset to enable detection of various maritime objects such as buoys or navigation marks. Nowicka, paragraph 49, the systems and methods of the present disclosure may be implemented to detect and identify a variety of objects including vessels (e.g., sailboats, powerboats), buoys, floating debris, person(s) in the water (e.g., person overboard), and other objects).
Regarding claim 16, the combination of Biancale, Tran, and Haskin teaches the system of claim 10(Biancale, paragraph 60, the system may be configured to detect pieces of debris, containers, wrecks, boats, navigation buoys; Tran, paragraph 121, Project image points onto the 3D plane; Haskin, page 104 line 19, geo-synchronization algorithm herein may use a database that may associate a geographical location with each of the objects in the group),
While the combination of Biancale, Tran, and Haskin teaches detection of objects from an image using neural network, and providing bounding box, it fails to disclose a system wherein the neural network-based object detector provides bounding boxes and confidence scores for each detected navigational marker.
However, Nowicka, which is in the same analogous art and that teaches about marine object classification disclose a system wherein the neural network-based object detector provides bounding boxes(Nowicka, paragraph 83, Object are identified by bounding boxes and labels such as: (i) bounding box 212a labeled as a nearby buoy 212b. Nowicka, paragraph 84, FIG. 3, which shows (i) an image 302 with a bounding box for object detection) and confidence scores for each detected navigational marker(Nowicka, paragraph 10, The method may further comprise identifying a detected object using a convolutional neural network trained to identify the detected object and generated an associated confidence value. Nowicka, paragraph 49, the systems and methods of the present disclosure may be implemented to detect and identify a variety of objects including vessels (e.g., sailboats, powerboats), buoys, floating debris, person(s) in the water (e.g., person overboard), and other objects.).
Therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Biancale, Tran, and Haskin with Nowicka to provide bounding box, and determine the confidence score of a neural network based navigational marker detection system. By evaluating the confidence score of the navigational marker classifier, it is possible to determine if classification should be accepted or rejected based on the accuracy score of the classification. Furthermore, it is possible to do further classification using different neural network algorithm for classification with low accuracy score.
Regarding claim 17, the combination of Biancale, Tran, Haskin, and Nowicka teaches The system of claim 16(Biancale, paragraph 60, the system may be configured to detect pieces of debris, containers, wrecks, boats, navigation buoys; Tran, paragraph 121, Project image points onto the 3D plane; Haskin, page 104 line 19, geo-synchronization algorithm herein may use a database that may associate a geographical location with each of the objects in the group; Nowicka, paragraph 10, The method may further comprise identifying a detected object using a convolutional neural network trained to identify the detected object and generated an associated confidence value), wherein the neural network-based object detector further classifies the type of navigational marker(Biancale, paragraph 9, An artificial neural network is a straight, easy and efficient way to process images generated by the at least one camera in order to detect, identify and track objects at least partially immerged in water. Predetermined set of features may be materialized into classes associated with known objects, for example boats, buoys, whales, containers. Nowicka, paragraph 97, A network may be pretrained on/using a marine dataset identifying boats and then fine-tuned on a more robust marine dataset to enable detection of various maritime objects such as buoys or navigation marks. Nowicka, paragraph 49, the systems and methods of the present disclosure may be implemented to detect and identify a variety of objects including vessels (e.g., sailboats, powerboats), buoys, floating debris, person(s) in the water (e.g., person overboard), and other objects).
Prior Art of Record
The prior art made of record and not relied upon is considered pertinent to applicant’s
disclosure.
LaDuke (US 20130308064 A1) discloses a video processing of an infrared video camera output that can be used to generate a video mask to project with a video projector a video image (still and/or moving images) onto the water surface.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BESUFEKAD LEMMA TESSEMA whose telephone number is (571)272-6850. The examiner can normally be reached Monday - Friday 9:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hunter Lonsberry can be reached at 5712727298. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BESUFEKAD LEMMA TESSEMA/Examiner, Art Unit 3665
/HUNTER B LONSBERRY/Supervisory Patent Examiner, Art Unit 3665