DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 2 recites the limitation "a timestamp associated with the image". It is unclear if “a timestamp” is the same or different as the instance recited in claim 1.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 2, 5 – 8, 15 – 18, and 20 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Russo et al. (US 2023/0194304).
Regarding independent claim 1, Russo teaches a method (Figures 3, 4) comprising:
identifying a landmark in image data (paragraph 24: the image data may include a new building, which can result in an update to the machine learning model such that the new building is considered in future updates, alerts, etc.) captured by a device (paragraph 20: The image data, for instance, can include images gathered by the first source, which can include an autonomous vehicle and/or an associated sensor (e.g., camera, heat sensor, etc.) in communication with the autonomous vehicle);
receiving pose data associated with the image data (paragraph 25: using metadata associated with the image data (e.g., heading information, location, time, etc.), a determination can be made that image data received includes the location and particular images that address the missing and/or outdated portion; paragraph 20: The image data can also include metadata that can include physical location data (e.g., address, GPS data, etc.), viewing direction data (e.g., a camera facing a particular direction), etc. to identify a location of the image ( e.g., the first location));
determining that the landmark is included in a map of an environment (paragraph 25: At 110, the processing resource 102 can execute instructions to identify the image data as being associated with a missing portion, an outdated portion, or both, of the map, and at 112, the processing resource 102 can execute instructions to update the missing portion, the outdated portion, or both, of the map with the image data); and
in response to determining that the landmark is included in the map,
determining that a timestamp associated with the image data meets a criterion (paragraph 19: An image may be considered outdated, for instance, if an updated image has not been added to the map within a threshold period of time (e.g., 10 minutes, 1 hour, 1 day, 1 month, etc.)), and
in response to determining the timestamp meets the criterion, modifying the map based on the landmark and the pose data (paragraph 26: if the map includes an image of a building taken more than a threshold time in the past (e.g., 10 minutes ago), this area of the map may be considered outdated, and it can be replaced with a new image having matching metadata).
Regarding dependent claim 2, Russo teaches capturing multiple images by the device (paragraph 33: The plurality of sources, for instance, can include autonomous vehicles and/or their associated sensors that gather images and other data as they travel), wherein the multiple images including the landmark and the image data includes data from the multiple images (paragraph 59: For instance, in the water example, a calculation for depth of water can be made based on different images from the same vehicle 570, from different vehicles, and/or from landmark comparisons on old and new images); and
discarding image data from an image of the multiple images as discarded image data based on a timestamp associated with the image (paragraph 34: it can be determined that portions of the map, including the second plurality of images, have not been updated within a threshold amount of time. For instance, a list could be updated that lists images that are missing from a recent specified period of time (e.g., 10 minutes). Examiner notes that any images over 10 minutes old are not used).
Regarding dependent claim 5, Russo teaches wherein the criterion is a time greater than a threshold (paragraph 19: An image may be considered outdated, for instance, if an updated image has not been added to the map within a threshold period of time (e.g., 10 minutes, 1 hour, 1 day, 1 month, etc.)).
Regarding dependent claim 6, Russo teaches wherein the map includes data associated with the environment, data associated with the landmark, and the pose data (paragraph 59: For instance, in the water example, a calculation for depth of water can be made based on different images from the same vehicle 570, from different vehicles, and/or from landmark comparisons on old and new images).
Regarding dependent claim 7, Russo teaches wherein in response to determining the landmark is not included in the map (paragraph 25: At 110, the processing resource 102 can execute instructions to identify the image data as being associated with a missing portion, an outdated portion, or both, of the map, and at 112, the processing resource 102 can execute instructions to update the missing portion, the outdated portion, or both, of the map with the image data), the method further comprising:
generating map data based on the environment, data associated with the landmark, and the pose data (paragraph 54: At 454, locations on the map that have not received images in a threshold period of time can be flagged, as well as areas of the maps with missing images or camera angles. At 456, the cloud storage and other V2X communications are checked for images that match the missing and/or desired images based on the metadata); and
including the map data in the map (paragraph 54: At 458, a matching, updated image found during the search is integrated into the map).
Regarding dependent claim 8, Russo teaches wherein the device is one of a wearable device (paragraph 26: A prompt, in some instances, can provided to computing devices such as nearby mobile devices; paragraph 2: Examples of IoT enabled devices include mobile phones, smartphones, tablets, phablets, computing devices, implantable devices, vehicles, home appliances, smart home devices, monitoring devices, wearable devices, devices enabling intelligent shopping systems, among other cyber-physical systems), a robot device (paragraph 20: An autonomous vehicle), or a drone device (paragraph 20: An autonomous vehicle).
Regarding independent claim 15, Russo teaches method comprising:
determining an environment lacks an associated map (paragraph 25: At 110, the processing resource 102 can execute instructions to identify the image data as being associated with a missing portion, an outdated portion, or both, of the map, and at 112, the processing resource 102 can execute instructions to update the missing portion, the outdated portion, or both, of the map with the image data);
traversing the environment by a device (paragraph 33: The plurality of sources, for instance, can include autonomous vehicles and/or their associated sensors that gather images and other data as they travel);
identifying a landmark in first image data captured by a device (paragraph 24: the image data may include a new building, which can result in an update to the machine learning model such that the new building is considered in future updates, alerts, etc.);
receiving pose data associated with the first image data (paragraph 25: using metadata associated with the image data (e.g., heading information, location, time, etc.), a determination can be made that image data received includes the location and particular images that address the missing and/or outdated portion; paragraph 20: The image data can also include metadata that can include physical location data (e.g., address, GPS data, etc.), viewing direction data (e.g., a camera facing a particular direction), etc. to identify a location of the image ( e.g., the first location));
determining the device has previously received second image data associated with a current location of the device within the environment (paragraph 37: A request may be an alert to sources in the area of the matching image, or it may be a search of a database or cloud storage for the matching image. A matching image may meet particular criteria, for instance, matching a portion or all of the metadata associated with the missing, outdated, or other image. For instance, matching location coordinates and a particular camera view within a threshold angle may be matching criteria); and
in response to determining the device has received the second image data, discard data associated with the landmark and the pose data (paragraph 34: it can be determined that portions of the map, including the second plurality of images, have not been updated within a threshold amount of time. For instance, a list could be updated that lists images that are missing from a recent specified period of time (e.g., 10 minutes). Examiner notes that any images over 10 minutes old are not used).
Regarding dependent claim 16, Russo teaches in response to determining the device has received the second image data,
generating map data based on the environment, data associated with the landmark, and the pose data (paragraph 54: At 454, locations on the map that have not received images in a threshold period of time can be flagged, as well as areas of the maps with missing images or camera angles. At 456, the cloud storage and other V2X communications are checked for images that match the missing and/or desired images based on the metadata), and generating the map based on the map data (paragraph 54: At 458, a matching, updated image found during the search is integrated into the map).
Regarding dependent claim 17, Russo teaches wherein the determining the device has received the second image data is based on the device traversing around the landmark and the method can further comprise generating the map in response to determining the device is traversing around the landmark (paragraph 53: At 450, the car is traveling down a roadway and capturing images as it travels. The time it is traveling, as well as location information, camera angle information, max image depth information, alert information, and other metadata are collected and kept with the captured images; paragraph 37: A request may be an alert to sources in the area of the matching image, or it may be a search of a database or cloud storage for the matching image. A matching image may meet particular criteria, for instance, matching a portion or all of the metadata associated with the missing, outdated, or other image. For instance, matching location coordinates and a particular camera view within a threshold angle may be matching criteria).
Regarding dependent claim 18, Russo teaches wherein the determining the device has received the second image data is based on the device traversing by the landmark in a same direction (paragraph 53: At 450, the car is traveling down a roadway and capturing images as it travels. The time it is traveling, as well as location information, camera angle information, max image depth information, alert information, and other metadata are collected and kept with the captured images; paragraph 37: A request may be an alert to sources in the area of the matching image, or it may be a search of a database or cloud storage for the matching image. A matching image may meet particular criteria, for instance, matching a portion or all of the metadata associated with the missing, outdated, or other image. For instance, matching location coordinates and a particular camera view within a threshold angle may be matching criteria).
Regarding dependent claim 20, Russo teaches wherein the device is one of a wearable device (paragraph 26: A prompt, in some instances, can provided to computing devices such as nearby mobile devices; paragraph 2: Examples of IoT enabled devices include mobile phones, smartphones, tablets, phablets, computing devices, implantable devices, vehicles, home appliances, smart home devices, monitoring devices, wearable devices, devices enabling intelligent shopping systems, among other cyber-physical systems), a robot device (paragraph 20: An autonomous vehicle), or a drone device (paragraph 20: An autonomous vehicle).
Claim(s) 9 – 11, 13, and 14 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Ogale (US 9,201,424)
Regarding independent claim 9, Ogale teaches a method (Figures 3, 4) comprising:
determining a device motion using a first plurality of landmarks and device poses (column 15, lines 50 – 67: An autonomous vehicle may utilize functions and processes 55 relating to structure from motion to determine an image-based pose from the images. . . The computing device may analyze the different images captured of objects by the vehicle's camera system to determine estimated 3D structures of the objects. The computing device may use a subset of the images provided by the camera system);
selecting a second plurality of landmarks, the second plurality of landmarks including a subset of the first plurality of landmarks, the second plurality of landmarks surrounding a current device location (column 16, line 45 – column 17, line 5: the computing device may track features, such as comers or line segments in a series of images based on a particular object or set of objects. In response to tracking features, the computing device may use the subset of images to further estimate motion and structure); and
updating map data of a map of an environment based on the second plurality of landmarks and the device poses (column 10, lines 24 – 34: the computer vision system 146 may use an object recognition algorithm, a Structure from Motion (SFM) algorithm, video tracking, or other computer vision techniques. In some examples, the computer vision system 146 may additionally be configured to map the environment, track objects, estimate speed of objects, etc.).
Regarding dependent claim 10, Ogale teaches obtaining image data from multiple images captured by the device, the image data including data corresponding to the first plurality of landmarks (column 15, lines 38 – 49: the camera system may provide different sets of images to a computing system or device of a vehicle that pertain to different objects);
selecting an image of the multiple images based on a timestamp associated with the image (column 24, lines 33 – 48: The computing device or system may factor the speed of the vehicle and/or the specific times the images are captured to link similar images together); and
using the selected image to determine the device motion (column 24, lines 49 – 59: In addition, a computing device or system of an autonomous vehicle may use the subset of images to perform bundle adjustment, which may include mapping 3D points within the images according to points as shown by the building. The computing device may further minimize error by performing bundle adjustment subsequent times. The computing device may bundle greater numbers of images to increase the accuracy of the image-pose determined for the object).
Regarding dependent claim 11, Ogale teaches obtaining image data from multiple images captured by the device, the image data including data corresponding to the first plurality of landmarks (column 15, lines 38 – 49: the camera system may provide different sets of images to a computing system or device of a vehicle that pertain to different objects); and
discarding image data of the multiple images based on a timestamp associated with the image (column 24, lines 33 – 48: The computing device or system may factor the speed of the vehicle and/or the specific times the images are captured to link similar images together. Thus images not withing the specific times are not used for the processing algorithm of the set of objects).
Regarding dependent claim 13, Ogale teaches refining an error correction associated with the determining of the device motion (column 17, lines 21 – 42: a computing device or system associated with an autonomous vehicle may be configured to perform a bundle adjustment process to determine an image-based pose. The computing device may use bundle adjustment to minimize the reprojection error that may occur based on the image location of observed and predicted image points, which may be expressed as the sum of squares of a large number of nonlinear, real-valued functions; column 17, lines 6 – 20: Through application of an image-matching process, a computing device may identify images from the numerous images received that may be useful for calibrating a camera or cameras of the camera system; column 16, line 45 – column 17, line 5: Tracking features may allow the computing device to determine which images to use and which images to disregard for automatically calibrating the extrinsic parameters of a camera or cameras).
Regarding dependent claim 14, Ogale teaches wherein the device is one of a wearable device, a robot device (column 15, lines 50 – 67: autonomous vehicle), or a drone device (column 15, lines 50 – 67: autonomous vehicle).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 3 and 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Russo et al. (US 2023/0194304) in view of Ogale (US 9,201,424).
Regarding dependent claim 3, Ogla discloses, which Russo does not expressly disclose, wherein identifying the landmark includes identifying a plurality of landmarks (column 15, lines 50 – 67: An autonomous vehicle may utilize functions and processes 55 relating to structure from motion to determine an image-based pose from the images. . . The computing device may analyze the different images captured of objects by the vehicle's camera system to determine estimated 3D structures of the objects. The computing device may use a subset of the images provided by the camera system), the method further comprising:
determining a motion of the device using the plurality of landmarks and the pose data (column 15, lines 50 – 67: An autonomous vehicle may utilize functions and processes 55 relating to structure from motion to determine an image-based pose from the images);
selecting a subset of the plurality of landmarks, the subset including landmarks surrounding a location of the device (column 16, line 45 – column 17, line 5: the computing device may track features, such as comers or line segments in a series of images based on a particular object or set of objects. In response to tracking features, the computing device may use the subset of images to further estimate motion and structure); and
in response to determining the timestamp meets the criterion (column 24, lines 33 – 48: The computing device or system may factor the speed of the vehicle and/or the specific times the images are captured to link similar images together. Thus images not withing the specific times are not used for the processing algorithm of the set of objects), modifying the map based on the subset of the plurality of landmarks and the pose data (column 10, lines 24 – 34: the computer vision system 146 may use an object recognition algorithm, a Structure from Motion (SFM) algorithm, video tracking, or other computer vision techniques. In some examples, the computer vision system 146 may additionally be configured to map the environment, track objects, estimate speed of objects, etc.).
Russo discloses an autonomous vehicle that captures a set of structures and their features in a set of images and is similar endeavor as Ogale that captures a set of objects and their features in a set of images. It would have been obvious for one of ordinary skill in the art at the time of the invention (pre-AIA ) or at the time of the effective filing date of the application (AIA ) to modify Russo's system to determine a subset of a plurality of structures and modifying map data based on the subset of the plurality of structures and pose data based on the timestamp criteria. One would be motivated to do so because this would help keep the operations of the autonomous vehicle safe (column 10, lines 35 – 49).
Regarding dependent claim 4, Russo does not expressly disclose in response to determining the timestamp meets the criterion, refining an error correction associated with determining a motion of the device. Ogale discloses “a computing device or system associated with an autonomous vehicle may be configured to perform a bundle adjustment process to determine an image-based pose. The computing device may use bundle adjustment to minimize the reprojection error that may occur based on the image location of observed and predicted image points, which may be expressed as the sum of squares of a large number of nonlinear, real-valued functions “ (column 17, lines 21 – 42) and “In addition, a computing device or system of an autonomous vehicle may use the subset of images to perform bundle adjustment, which may include mapping 3D points within the images according to points as shown by the building. The computing device may further minimize error by performing bundle adjustment subsequent times. The computing device may bundle greater numbers of images to increase the accuracy of the image-pose determined for the object” (column 24, lines 49 – 59). It would have been obvious for one of ordinary skill in the art at the time of the invention (pre-AIA ) or at the time of the effective filing date of the application (AIA ) to modify Russo's system to correct errors associated with a motion of the device based on set of images within a period of time. One would be motivated to do so because this would help calibrate cameras to be more accurate (column 17, lines 6 – 20).
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ogale (US 9,201,424) in view of Russo et al. (US 2023/0194304)
Regarding dependent claim 12, Russo discloses, which Ogale does not expressly disclose, determining at least one landmark of the second plurality of landmarks is not included in the map (paragraph 25: At 110, the processing resource 102 can execute instructions to identify the image data as being associated with a missing portion, an outdated portion, or both, of the map, and at 112, the processing resource 102 can execute instructions to update the missing portion, the outdated portion, or both, of the map with the image data);
generating the map data based on a location of the device, data associated with the at least one landmark surrounding the current device location is not included in the map, and the device poses (paragraph 54: At 454, locations on the map that have not received images in a threshold period of time can be flagged, as well as areas of the maps with missing images or camera angles. At 456, the cloud storage and other V2X communications are checked for images that match the missing and/or desired images based on the metadata); and
including the map data in the map (paragraph 54: At 458, a matching, updated image found during the search is integrated into the map).
Russo discloses an autonomous vehicle that captures a set of structures and their features in a set of images and is similar endeavor as Ogale that captures a set of objects and their features in a set of images. It would have been obvious for one of ordinary skill in the art at the time of the invention (pre-AIA ) or at the time of the effective filing date of the application (AIA ) to modify Ogale's system to determine missing data and to obtain images pertaining to the missing data to be updated onto a map. One would be motivated to do so because this would provide the most recent up-to-date and accurate map (paragraph 39).
Claim(s) 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Russo et al. (US 2023/0194304) in view of Official Notice.
Regarding dependent claim 19, Russo does not expressly disclose wherein the determining the device has received the second image data is based on the device traversing by the landmark in a different direction, however Russo does disclose a matching image may meet particular criteria, for instance, matching a portion or all of the metadata associated with the missing, outdated, or other image (paragraph 37). Examiner takes Official Notice that the concept of an autonomous vehicle turning around and obtaining images of objects in the environment is well known and expected in the arts. It would have been obvious for one of ordinary skill in the art at the time of the invention (pre-AIA ) or at the time of the effective filing date of the application (AIA ) to achieve a predictable result of an autonomous vehicle to turn around to capture images of objects in a different direction by modifying Russo's system to partially match a portion of the metadata between images in a different direction by trying a limited number of directions to capturing objects around the autonomous vehicle, and the result would have been predictable.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JEFFREY J CHOW whose telephone number is (571)272-8078. The examiner can normally be reached 11AM-7PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JEFFREY J CHOW/Primary Examiner, Art Unit 2618
Prosecution Insights