Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendments and Remarks
Applicant's arguments filed 1/22/26 have been fully considered as follows:
Applicant argues:
Applicant notes that none of Johnson, Altamura, or Ambler disclose or teach "identify a location of interest or a representation for the location of interest that corresponds to the field of view, wherein the location of interest is underneath water and is associated in the memory with a stored underwater image at the location of interest" and "cause presentation of additional information about the location of interest to be overlayed onto the image at a position above the water at the location in the field of view, wherein the additional information includes at least the stored underwater image of the location of interest" as claimed. In this regard, the claimed invention first identifies a location within afield of view that has an associated stored underwater image of the location and displays it as an overlay onto the image with the overlay being at a position above the water at the location. None of Johnson, Altamura, or Ambler (nor their combination) teach or suggest such a feature, and it would not be obvious to one of ordinary skill in the art to adjust any of their teachings accordingly. Johnson and Altamura fail to teach presenting a stored underwater image above the water. Ambler does not cure that deficiency, as Ambler is focused on presenting images in the field of view (i.e., within the water from the perspective of the viewer). Indeed, for Ambler, presenting the image in that manner makes it hard for a user to make a decision as it requires a user to look all the way down (which could be difficult in deep water and loses context of where on the water surface the user would need to travel too). Indeed, the claimed invention provides a user with a stored, historical image of the underwater environment at the location of interest without the user having to travel to the location of interest (e.g., to use sonar or underwater cameras at the location to see what is under the water there) and without the user having to look all the way down. This helps the user know what is around them under the water, and enables them to determine whether to travel there or not.
(Remarks, Page 8).
In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Here, Johnson is primarily directed towards display of information above the water (See Fig. 19):
PNG
media_image1.png
370
647
media_image1.png
Greyscale
Ambler is cited to disclose additional information such as imagery of underwater objects, which could be readily incorporated into additional information display to maintain the contextual location of objects as POI markers.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Use of indicates a limitation is not explicitly disclosed by the reference alone.
Claim(s) 1-2, 6, 8-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Johnson (US 2021/0206459) in view of Altamura (US Patent 8,265,866) and Ambler (US 2020/0242848)
Claim 1
Johnson discloses a system for overlaying a representation of a location of interest on an image of an environment around a watercraft (Johnson, ¶ 179: “a low cost fixed mount camera is attached to a vessel…overlays rendered in display 226 and/or user interface 120 can be easily achieved.”), the system (e.g. Fig. 2A) comprising:
PNG
media_image2.png
482
611
media_image2.png
Greyscale
an electronic device having a display (226);
at least one processor (221);
a camera (Johnson, ¶ 95: “imaging modules 223 and/or 224 are oriented to capture a portion of scene 200 including waterline 205”);
a memory (Johnson, ¶ 71: “a machine readable medium may be provided for storing non-transitory instructions for loading into and execution by controller 130. In these and other embodiments, controller 130 may be implemented with other components where appropriate, such as volatile memory, non-volatile memory”) having stored thereon software instructions that, when executed by the at least one processor (Johnson, ¶ 66: “User interface 120 may also be implemented with one or more logic devices that may be adapted to execute instructions, such as software instructions, implementing any of the various processes and/or methods described herein”), cause the at least one processor to:
receive location information and orientation information for the electronic device or the watercraft (Johnson, ¶ 114: “In the embodiment shown in FIG. 2A, system 220 includes OPS 225. In some embodiments, controller 221 may be configured to receive the sonar data, the radar data, and/or image data based on a measured position and/or orientation provided by OPS 225. OPS 225 may be implemented as one or more orientation sensors, GNSS sensors, differential GNSS sensors, orientation/position reference transducers and/or optical sensors (e.g., for actuators), visible spectrum and/or infrared imaging modules, and/or other sensors configured to measure a relative and/or absolute orientation and/or position of system 220 and/or each of imaging modules 223 and 224 and display 226 and provide such measurements to controller 221”);
cause, based on camera data received from the camera, presentation of the image of the environment around the watercraft (Johnson, ¶ 113: “…configured to communicate with various modules of system 220 and be configured to receive imagery/image data of scene 200 from imaging modules 223 and/or 224, determine waterline 205 of a body of water 205a in scene 200 (e.g., from image data, position data, and/or orientation data provided by the device), render or display image data in any portion of an FOV of display 226 that extends above waterline 205…”);
determine a field of view for the camera that includes the environment around the watercraft within the image (Johnson, ¶ 113: “render or display image data in any portion of an FOV of display 226 that extends above waterline 205…”);
identify a location of interest or a representation for the location of interest that corresponds to the field of view, wherein the location of interest is underneath water and is (Johnson, ¶ 126: “Scene 200 includes features above waterline 205 illustrated in FIG. 2A and additionally includes mountains/land features 204, tree 202, vessel 213, floating object 211a, surface 205c of body of water 205a, and deck 106b (e.g., of mobile structure/boat 101 in FIG. 1B). Also shown in FIG. 3, and in particular in the FOV of display 226, are detected waterline 205b, portion 330 of the FOV that extends below waterline 205b, and portion 334 of the FOV that extends above waterline 205b. System 220 may in some embodiments be configured to render detected waterline 205b in display 226 to illustrate a detected location of waterline 205 relative to the FOV of display 226. Portion 330 may include imagery representing bottom feature 207, fish 208, submerged object 209, and the submerged portion of the floating object 211b similar to objects illustrated in FIG. 2A”);
cause the representation of the location of interest to be overlayed onto the image at a position above the water at the location in the field of view so as to generate an augmented reality image (Johnson, Fig. 19; ¶ 226: “In various embodiments, it can be beneficial to generate augmented reality (AR) display views (e.g., for rendering on a display of user interface 120 of system 100, display 226 of system 220, and/or displays 426 of portable imaging device/system 420) that include graphics or AR overlays designed to indicate hidden or metadata characteristics of a particular identified object or position, or chart-referenced object or position, intuitively so as to reduce risk of user confusion when viewing the AR display view. For example, FIG. 19 illustrates an AR display view 1900 including two different types of waypoint flags (e.g., waypoint flags 1910 and 1912, vs. waypoint flags 1920 and 1922) or waypoint graphics designed and positioned within display view 1900 to indicate whether a referenced position is submerged and not visible or is at least partially visible (e.g., detectable by imaging modules 223 and/or 224) above or on surface of water 205a, in accordance with an embodiment of the disclosure.”);
PNG
media_image3.png
381
658
media_image3.png
Greyscale
Johnson does not explicitly disclose, but Altamura discloses receive an input requesting further information for the location of interest (Altamura, Col. 3: “Through user selection, additional information 192 regarding an indicator may be obtained.”); and
cause presentation of additional information about the location of interest to be overlayed onto the image, (Altamura, Col. 3: “Through user selection, additional information 192 regarding an indicator may be obtained.”).
PNG
media_image4.png
410
570
media_image4.png
Greyscale
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to provide additional information based on user input.
One of ordinary skill in the art would have motivation to provide in context information (Altamura, Col. 1: “The augmentation provided by such systems is conventionally in real-time and in semantic context with environmental elements…By adding computer vision and/or object recognition, the information about the surrounding real world of the user becomes interactive and usable.”) One of ordinary skill in the art would have had a reasonable expectation of success because Johnson is also directed to providing in context overlay of POI information and could benefit from providing additional in context information.
Johnson in view of Altamura do not explicitly disclose, but Ambler discloses associated in the memory with a stored underwater image at the location of interest (Ambler, ¶ 26-27: “For example, in cases where the environmental sensor 108 determines the position of obstructed objects, the graphical image can include representations of objects that are obstructed from the user's current real-world view…correlating the data received from the environmental sensor 108 and the position sensor 122….the images presented to the user 102 can be based on prior data collected by the environmental sensor 108. In such instances, the prior environmental data is stored in a memory or other storage device located locally on the headset 116 or remotely (e.g., in the cloud)”)
wherein the additional information includes at least the stored underwater image of the location of interest (Ambler, ¶ 26-27: “For example, in cases where the environmental sensor 108 determines the position of obstructed objects, the graphical image can include representations of objects that are obstructed from the user's current real-world view…correlating the data received from the environmental sensor 108 and the position sensor 122….the images presented to the user 102 can be based on prior data collected by the environmental sensor 108. In such instances, the prior environmental data is stored in a memory or other storage device located locally on the headset 116 or remotely (e.g., in the cloud)”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to provide additional information such as additional images as claimed.
One of ordinary skill in the art would have motivation to provide in context information (Ambler, ¶ 4: “there is still a need for a system that detects information about real objects within a user's environment (e.g., hidden or obstructed objects) and uses an A/R graphical image to identify the object for the user.”). One of ordinary skill in the art would have had a reasonable expectation of success because Johnson is also directed to providing in context overlay of POI information and could benefit from providing additional in context information.
The combined teachings of Johnson and Ambler disclose the amended limitations of displaying both the image and additional information at a position above the water at the location in the field of view (e.g. Johnson considers above water indicators for subsurface POI; Fig. 19:
PNG
media_image5.png
374
653
media_image5.png
Greyscale
“relative position of a submerged object type (e.g., navigational hazard, or fish) within display view 1900 corresponding to a waypoint stored in memory and/or a detected object. For example, controller 130 and/or 221 may be configured to determine such range and/or relative position and/or orientation (e.g., based on an FOV of an imaging module providing underlying image data 1930, a current position/orientation of mobile structure 101, system 220, and/or device 420, and/or other data, as described herein) and render one or more such waypoint flags within display view 1900. Visible waypoint flags 1920 and 1922 each lack a flagpole base and instead include a flag lead 1921 and 1923, respectively, and be positioned and/or oriented above their corresponding object (e.g., a buoy, or an oil rig) to indicate a range to (e.g., 63 m, or 3.1 km) or relative position of a visible object or structure. General examples of visible objects include buoys, fuel spills, public toilets, restaurants, vessel deploy ramps, racemarks (all colors/directions), bottom marks, top marks, oil rigs, cans (all colors), nuns (all colors), preferred marks, posts, sailboats, sportfishers, trawlers, towers, hills/peaks, and/or other objects or structures visible on or above surface 205a, and each are indicated by a visible waypoint flag rendered within AR display view 1900, as shown. Other objects and corresponding symbols are defined as hidden or submerged waypoints and are correspondingly indicated by hidden waypoint flags rendered within AR display view 1900, as shown”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to provide additional information such as information above the water as claimed.
One of ordinary skill in the art would have motivation to provide in context information (Ambler, ¶ 4: “there is still a need for a system that detects information about real objects within a user's environment (e.g., hidden or obstructed objects) and uses an A/R graphical image to identify the object for the user.”). One of ordinary skill in the art would have had a reasonable expectation of success because Johnson is also directed to providing in context overlay of POI information and could benefit from providing additional in context information.
Claim 2
Johnson discloses wherein the representation comprises a symbol of the location of interest (Johnson Fig. 19; e.g. fish marker; X, oil rig, buoy graphics)
Claim 6
Johnson discloses wherein the location of interest is at least one of a second watercraft (Fig. 20, 2020), an underwater wreck, an underwater reef (1910 underwater hazard), a buoy (1920), a dock, a dumping ground, or an animal (1912, fish),
Claim 8
Johnson does not disclose, but Ambler makes obvious wherein the second image is a sonar image of the location of interest (Ambler, ¶ 27: “if the environmental sensor 108 is a sonar detector mounted under the hull of the boat 104, as the boat 104 moves, the terrain detected by the sensor 108 can also change. In such instances, if the user's real-world view includes an area that is not presently being detected by the environmental sensor 108 (e.g., behind the wake of the boat), then the images presented to the user 102…then the images presented to the user 102 can be based on prior data collected by the environmental sensor 108.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to provide additional information such as for improved sonar and radar display.
One of ordinary skill in the art would have motivation to provide in context information such as for improved subsurface display. One of ordinary skill in the art would have had a reasonable expectation of success because Johnson is also directed to providing in context overlay of POI information and could benefit from providing additional in context information.
Claim 9
Johnson discloses wherein the software instructions, when executed by the at least one processor, cause the at least one processor to: identify a second location of interest that is within the field of view; and cause a second representation of the second location of interest to be overlayed onto the image (Johnson, Fig. 19; multiple second locations of interests displayed in field of view)
Claim 10
Johnson discloses wherein the software instructions, when executed by the at least one processor, cause the at least one processor to: present distance information for the location of interest (Johnson, ¶ 227: “As shown in FIG. 19, hidden waypoint flags 1910 and 1912 each include a flagpole base 1911 and 1913, respectively, and are positioned and/or oriented on surface 205a to indicate a range to (e.g., 56 m, or 71 m) or relative position of a submerged object type (e.g., navigational hazard, or fish) within display view 1900 corresponding to a waypoint stored in memory and/or a detected object.”)
Claim 11
Johnson discloses wherein the software instructions, when executed by the at least one processor, cause the at least one processor to: present an indicator on the display, wherein the indicator provides navigational guidance as to how to reach the location of interest (e.g. navigation tracks; Johnson ¶ 232: “In some embodiments, controller 130 and/or 221 may be configured to render cross track error boundary line 2210 with indicators/arrows 2212 configured to indicate the appropriate track direction for the track being traversed by mobile structure 101, for example, to help guide a pilot whose perspective may be undefined, such as if mobile structure 101 has steered 180 degrees away from the intended track direction.”; e.g. POI markers are also located corresponding to position)
Claim 12
Johnson discloses wherein the software instructions, when executed by the at least one processor, cause the at least one processor to: present an environmental indication on the display, wherein the environmental indication is related to an environmental condition, wherein the environmental condition is at least one of a wind speed, current, water temperature, or water depth (Johnson, ¶ 128: “n such cases, the controller may then render the weather, graphics indicative of the weather, or a warning on display 226 to warn the user of the weather. In other embodiments, the controller may render the weather, graphics indicative of the weather, or messages to indicate the weather even if the weather data does not indicate that the weather conditions are representative of that of a storm or other bad weather.”; ¶ 89: “In such embodiment, the underwater track may be predetermined, for example, or may be determined based on criteria parameters, such as a minimum allowable depth, a maximum ensonified depth”)
Claim 13
Johnson discloses wherein the electronic device is a phone, a tablet, a computer, smart glasses, or a headset (Johnson, ¶ 69: “In one embodiment, user interface 120 may be integrated with one or more sensors (e.g., imaging modules, position and/or orientation sensors, other sensors) and/or be portable (e.g., such as a portable touch display or smart phone, for example, or a wearable user interface) to facilitate user interaction with various systems of mobile structure 101.”; ¶ 120: “In some embodiments, system 220 may be implemented as a wearable device, such as a pair of glasses”)
Claim 14
Johnson discloses wherein the camera is provided in the electronic device (Johnson, ¶ 109: “As shown, system 220 may include one or more controllers 221 (e.g., including memory 222), imaging modules (e.g., visible spectrum imaging module 223”).
Claim 15
The same teachings and rationales in claim 1 are applicable to claim 15.
Claim 16
The same teachings and rationales in claim 9 are applicable to claim 16.
Claim 17
The same teachings and rationales in claim 10 are applicable to claim 17.
Claim 18
The same teachings and rationales in claim 11 are applicable to claim 18.
Claim 19
The same teachings and rationales in claim 12 are applicable to claim 19.
Claim 20
Examiner’s Interpretation:
Machine readable media can encompass forms of signal transmission media that falls outside of the four statutory categories of invention. MPEP 2106; citing In re Nuijten, 500 F.3d 1346, 84 USPQ2d 1495 (Fed. Cir. 2007). A claim whose BRI covers both statutory and non-statutory embodiments embraces subject matter that is not eligible for patent protection and therefore is directed to non-statutory subject matter. MPEP 2106.
Claim 20 as drafted recites a non-transitory computer readable medium…
The broadest reasonable interpretation of the claimed medium in view of Applicant’s specification covers only eligible subject matter.
Claim Mapping:
The same teachings and rationales in claim 1 are applicable to claim 20.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN M GRAY whose telephone number is (571)272-4582. The examiner can normally be reached on Monday through Friday, 9:00am-5:30pm (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached on (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN M GRAY/Primary Examiner, Art Unit 2611